Comments on: Does stubbing make your tests brittle?

By: Donnie

Donnie — Tue, 11 Nov 2008 07:48:43 +0000

Refactoring aside stubbing/mocking (which I refer to interchangeably as “stubbing”) should not be coupled to the implementation. Why? Simple: coupling to implementation is a break in encapsulation which violates a core tenet of OO. Now, there is of course times when stubbing makes perfect sense, like when “stubbing” out some functionality that is _external to the code being tested_. I recently moved from C++ to Ruby and I’m finding that oftentimes the dynamism of Ruby is taken as license by developers to throw away good OOP practices like encapsulation. It’s high time for the community to recognize that the principles of OO should be maintained even if your language allows you to ignore them. I have seen tests that stub out every single method call that the method being tested calls just to verify that the other methods (oftentimes internal private methods) are being called. This pattern is followed ad nauseum until the entire unit test is tightly coupled to the entire implementation of the class. The smallest bit of refactoring grossly renders all the tests useless. Why? Because encapsulation is broken!

Clearly this is an extreme. However, I find that more often than not OO principles are ignored in favor of a quick and dirty attempt at reaching full test coverage. This is backwards. Breaking encapsulation should be entirely avoided aside from a few exceptional cases. Instead, we have encapsulation being observed in a few exceptional cases.

So, Szeryf, I think we agree: stubbing can make your tests brittle. The reason this is so is because stubs tightly coupled to implementation break encapsulation.

By: Jogue fora os testes quebradiços : Learning on Rails

Jogue fora os testes quebradiços : Learning on Rails — Fri, 24 Oct 2008 17:28:25 +0000

[...] Does stubbing make your tests brittle? [...]

By: szeryf

szeryf — Sun, 10 Feb 2008 09:40:14 +0000

David, of course it’s natural that changes to code could break the tests. But that should be big, important changes, not tweaking some internal implementation details. I think good tests should be at least partly immune to implementation changes.

But I agree, divide and conquer is a good method to minimize possible test failures. Dividing between code that changes and code that doesn’t or between DB-stuff and non-DB-stuff is a good thing.

By: David Lowenfels

David Lowenfels — Sun, 10 Feb 2008 05:40:58 +0000

Firstly, I think it’s unrealistic to expect that code changes will never break a test. You can’t write code without understanding your system. And changing a few tests is really not a huge deal, it’s simply the cost of changing your code. We are ahead of the game knowing when our code is working or not, thanks to the test suite.

However that being said, the problem you illustrate suggests to me that you could break down your methods to use smaller private methods that call the database stuff. This has the additional advantage of being self-commenting if you name them well. For instance:

def todays_cheezburger
self.find_by_last_cheezburger_date( Date.today )
end

now your method becomes like so:

def can_has_cheezburger?
todays_cheezburger.nil?
end

and the two methods can be tested independently of each other. Divide and conquer! :)

By: Pat Maddox

Pat Maddox — Mon, 04 Feb 2008 06:53:20 +0000

Hey szeryf, I wrote a response to this post. Hope it gives you some ideas.

By: szeryf

szeryf — Tue, 08 Jan 2008 12:55:27 +0000

Anonymous: you have a good point, but tests in the above example weren’t meant to be realistic :) It was just simplest what I could invent that showed what I was talking about. In reality I would probably try to setup fixtures, just as you say. But your questions (which are perfectly valid) also show that there are more problems waiting for somebody that uses stubbing without caution.

I think I understand difference between stubing and mocks, but in this situation the risk is IMO similar. If I used mock object instead of stubbing, it would also probably answer only to count method, leading to the same problem. Is there any real difference?

Maybe we (me and my team) are doing something wrong way (that’s quite likely, because it’s our first project in Rails as team), but in our project most of the code in model class is somehow associated with database: finders, updaters, checkers and so on. So, most of our models tests would be integration tests by your standards. Do you have any advice on how to avoid coupling models to DB?

By: szeryf

szeryf — Tue, 08 Jan 2008 12:25:49 +0000

Mutwin: but still your controller specs are coupled to the controller class and sometimes even to the model class that the controller operates on (if I understand you correctly). But keeping controllers as simple as possibe could be a good way to reduce consequences of this coupling.

By: Anonymous

Anonymous — Tue, 08 Jan 2008 12:12:04 +0000

Any time you start stubbing methods in the class under test, you will potentially have problems. In this case, the tested class’ functionality is so tightly coupled to the database, it’s probably a better strategy to go ahead and test it against the database. How do you currently test whether the query in “count” is correct or not? What if the database schema changes?

Also, mocking and stubbing are quite different and have different risks. You mention mocking has having the same effects as stubbing and that’s often not true.

In general, we write integration tests of classes with tight coupling to the external resources like databases. Most of the code doesn’t have this coupling so we test those with unit testing techniques. The result is a large set of very fast and stable unit tests and a smaller, slower set of integration tests.

By: Mutwin Kraus

Mutwin Kraus — Mon, 07 Jan 2008 17:12:08 +0000

My solution is to use as littlte mocking/stubbing in Model specs (mostly when these models interact with others) and completely mock/stub the Controller (and view) specs. This makes changes relatively easy while encouraging to keep your controllers simple.