Does stubbing make your tests brittle?

Stubbing and mocking in unit tests is generally considered a Good Thing. It can make your tests run faster, it helps you not to cross boundaries. But it also makes your tests more brittle.

The reason for this is simple. If you want to stub out some method, you have to do it very precisely. You have to know the underlying implementation and choose a very concrete instance and method for stubbing. This makes your test very tightly coupled to the implementation details of tested class. And tightly coupled is brittle. Let’s look at some trivial example:

class LolCat < ActiveRecord::Base 
  def can_has_cheezburger? 
    0 == LolCat.count( 
           :conditions => [ 
             "id = ? and date(last_cheezburger_date) = ?", 
             self.id, Date.today]) 
  end 
end

This silly simple method checks if LolCat can has its cheezburger: only if it didn’t have one today. How do we test it? For example:

class LolCatTest < Test::Unit::TestCase 
  def test_can_has_cheezburger 
    LolCat.stubs(:count).returns(0) 
    assert @lolcat.can_has_cheezburger? 
  end  

  def test_can_has_no_cheezburger 
    LolCat.stubs(:count).returns(1) 
    assert !@lolcat.can_has_cheezburger? 
  end 
end

This is simple enough. We don’t want to touch the database, so we stub out the count method. Our test should run faster and it doesn’t cross boundaries. And we don’t have to setup any fixtures for the two cases tested thanks to this stubbing. All is good.

Until someone encounters our can_has_cheezburger? method and decides “we don’t need count, we should have used exists? instead”. They proceed with the refactoring, run the tests and discover the failure of our brittle tests.

How to prevent this? I don’t have any good solution. I just try to find a balance. Use stubbing and mocks when it makes sense. When the coupling induced by stubbing pays for itself. In the above example, we avoided setting up a LolCat with and without a cheezburger and touching the database. Was it worth it? Probably.


10 responses to “Does stubbing make your tests brittle?

  • Mutwin Kraus

    My solution is to use as littlte mocking/stubbing in Model specs (mostly when these models interact with others) and completely mock/stub the Controller (and view) specs. This makes changes relatively easy while encouraging to keep your controllers simple.

  • Anonymous

    Any time you start stubbing methods in the class under test, you will potentially have problems. In this case, the tested class’ functionality is so tightly coupled to the database, it’s probably a better strategy to go ahead and test it against the database. How do you currently test whether the query in “count” is correct or not? What if the database schema changes?

    Also, mocking and stubbing are quite different and have different risks. You mention mocking has having the same effects as stubbing and that’s often not true.

    In general, we write integration tests of classes with tight coupling to the external resources like databases. Most of the code doesn’t have this coupling so we test those with unit testing techniques. The result is a large set of very fast and stable unit tests and a smaller, slower set of integration tests.

  • szeryf

    Mutwin: but still your controller specs are coupled to the controller class and sometimes even to the model class that the controller operates on (if I understand you correctly). But keeping controllers as simple as possibe could be a good way to reduce consequences of this coupling.

  • szeryf

    Anonymous: you have a good point, but tests in the above example weren’t meant to be realistic :) It was just simplest what I could invent that showed what I was talking about. In reality I would probably try to setup fixtures, just as you say. But your questions (which are perfectly valid) also show that there are more problems waiting for somebody that uses stubbing without caution.

    I think I understand difference between stubing and mocks, but in this situation the risk is IMO similar. If I used mock object instead of stubbing, it would also probably answer only to count method, leading to the same problem. Is there any real difference?

    Maybe we (me and my team) are doing something wrong way (that’s quite likely, because it’s our first project in Rails as team), but in our project most of the code in model class is somehow associated with database: finders, updaters, checkers and so on. So, most of our models tests would be integration tests by your standards. Do you have any advice on how to avoid coupling models to DB?

  • Pat Maddox

    Hey szeryf, I wrote a response to this post. Hope it gives you some ideas.

  • David Lowenfels

    Firstly, I think it’s unrealistic to expect that code changes will never break a test. You can’t write code without understanding your system. And changing a few tests is really not a huge deal, it’s simply the cost of changing your code. We are ahead of the game knowing when our code is working or not, thanks to the test suite.

    However that being said, the problem you illustrate suggests to me that you could break down your methods to use smaller private methods that call the database stuff. This has the additional advantage of being self-commenting if you name them well. For instance:

    def todays_cheezburger
    self.find_by_last_cheezburger_date( Date.today )
    end

    now your method becomes like so:

    def can_has_cheezburger?
    todays_cheezburger.nil?
    end

    and the two methods can be tested independently of each other. Divide and conquer! :)

  • szeryf

    David, of course it’s natural that changes to code could break the tests. But that should be big, important changes, not tweaking some internal implementation details. I think good tests should be at least partly immune to implementation changes.

    But I agree, divide and conquer is a good method to minimize possible test failures. Dividing between code that changes and code that doesn’t or between DB-stuff and non-DB-stuff is a good thing.

  • Donnie

    Refactoring aside stubbing/mocking (which I refer to interchangeably as “stubbing”) should not be coupled to the implementation. Why? Simple: coupling to implementation is a break in encapsulation which violates a core tenet of OO. Now, there is of course times when stubbing makes perfect sense, like when “stubbing” out some functionality that is _external to the code being tested_. I recently moved from C++ to Ruby and I’m finding that oftentimes the dynamism of Ruby is taken as license by developers to throw away good OOP practices like encapsulation. It’s high time for the community to recognize that the principles of OO should be maintained even if your language allows you to ignore them. I have seen tests that stub out every single method call that the method being tested calls just to verify that the other methods (oftentimes internal private methods) are being called. This pattern is followed ad nauseum until the entire unit test is tightly coupled to the entire implementation of the class. The smallest bit of refactoring grossly renders all the tests useless. Why? Because encapsulation is broken!

    Clearly this is an extreme. However, I find that more often than not OO principles are ignored in favor of a quick and dirty attempt at reaching full test coverage. This is backwards. Breaking encapsulation should be entirely avoided aside from a few exceptional cases. Instead, we have encapsulation being observed in a few exceptional cases.

    So, Szeryf, I think we agree: stubbing can make your tests brittle. The reason this is so is because stubs tightly coupled to implementation break encapsulation.

  • Sorry, You’re Just Not My Type | site_title

    […] szeryf asks if stubbing makes your tests brittle. His widely-shared concern is that mocks and stubs couple your tests to the implementation. […]

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: