Why do we automate our tests?

Sounds like a simple question, but I see many people who don't actually understand why we automate our tests.

Before TDD I used to write a lot of one-off programs to test my code. While they were not exhaustive,they were pretty good - my code had few defects. Even with TDD my code is not perfect. I'm not sure what part of my better code today is because of TDD and what is because as an older programmer I have learned to write better code. So does automation gain anything? 

Some people have claimed that 90% of automated tests never fail. I don't know if this number is correct, but my personal experience suggests it isn't too far off. So why not throw them away and forget about automation?

Some people claim that automation is done so you can measure coverage of your tests. Someday I will write about this in more detail. For now, if test coverage numbers are your goal, you can get those numbers without automation. So again automation hasn't proven itself. (I will admit here that getting coverage without automation probably requires a lot more effort than automation, but the point is you don't need automation).

Why automate? I'll go back to the earlier claim that 90% of tests never fail. That leaves 10% of tests that at some point in the future have your back. That is someday in the future you, will make a change thinking all is well, not realizing some obscure side effect that broke some other seemingly unrelated feature. Fortunately some test will save you by informing you of that side effect. 

If only we knew which 10%, we could throw the rest away and get all the value. However we don't know which tests those are. Instead we keep them all, maintain them all,and wait for them to run each build. All for the time we weren't perfect.

Those 10% are the real value of automated tests. My one-off programs worked great, but they were moments after they passed.  The next day if I had to make a change I had to re-create them.  Generally I created the ones I thought I needed, which was not the complete set, and I had no way to being sure that everything I skipped actually didn't matter.  Thus the value of automated tests: all your tests are there and working for you when you need them.

So in conclusion, the reason to automate tests is your next code change, and has nothing to do with the current days effort. If your code has no future than automated tests are not useful.

Now that we have defined the test doubles. what next?

I previously defined the four different test doubles, but definitions are often not much help in figuring out what to use in your current situation.   So let me look at the differences in more detail.

The first thing to notice is stubs and fakes are about what a particular function call returns, while mocks and spys are about what a function was called with. This fundamental difference is a minor point though in real use, how can you stub/fake know what to return without the function arguments it is getting anyway? Likewise your mocks need to return something when they are called. (I have never seen a spy used where it has to return something, but I suppose someone might come up with one) 

There is a continuum between stub and fake, I don't know any hard rules to say where the boundary is, but in practice you generally know. A stub generally can only return a couple different canned values from a function, and that is it.  A fake is more complex: it tracks state and previously set values to create the proper return values, while still taking some shortcuts.

A fake is generally complex enough that you have tests just for the fake. Either because you know the behavior you want to model and directly test drive it, or after the fact adding tests because changes to the fake keep causing hundreds of tests to fail in non-obvious ways. In the latter case the test are added out of defense programing, you made the mistake so you write tests to help the next person not make it.

A fake is also something you could ship to some customers - as a demo of the full product. Since a fake is a shortcut to something hard to get access to, the fake makes a great training simulator. I even know of one case where a fake of the database is what was shipped when they realized the fake actually did everything they needed without the overhead of a SQL database.

If you are still confused, don't worry. The difference between stub and fake isn't important. It wouldn't be hard to make an argument that there is no difference. I like a separation myself, but I'm not sure if I'm being silly.

Mock frameworks have a interesting side effect, once you have gone through the effort of creating the mock once, you can quickly create many stubs just by changing the return value of only the functions that are called. This is often the way mock frameworks are used.

Spys are most often used in an observer pattern, since there is no return value to worry about. This is too bad because mock breaks the arrange act assert pattern of good tests as you have to assert before you act.

Defining the Test doubles

There is a lot of confusion about test doubles, many programmers are not even aware of all the options.  This is a shame because each option has value.

Stub is a implementation that returns canned values. This implies that some of the results can make no sense in the real world, but for the purposes of the test they are good enough.

Fake is a full implementation, but it doesn't do everything the real world implementation needs. It might use an in memory database, it might pretend to be a serial port with a device attached, but not actually use a real serial port. 

Mock is a way to ensure functions are called in specific ways. Call a function with the wrong argument and the test fails.

Spy is a way to check what functions were called after the fact.

I hope that the above are obvious, but still it is good to have everything defined so we can have a conversation about test doubles.

Of course the next step is defining when to use each.  That is a large topic that I hope to explore over time.

Should tests drive code?

On a code review recently someone asked a question about one test: what change did this test drive.  The implication being that the test in question didn't do anything to drive the change under review. Is this even a valid question to ask?

I do not know what motivated the question, but if I can speculate. Test Driven Development is, for good reasons, all the rage these days. TDD is all about writing tests to drive code, which is a great way to get started. 

What few people realize TDD misses is "works by design". There are many points in TDD cycle where the simplest thing that can possibly work has a side effect also causes some other useful behavior that users will rely on. These side effects need to be tested, even though the test will go green instantly. Otherwise some future person may refactor and lose that useful side effect without any tests breaking.

Sounds easy when I put it that way. In the real world it isn't. You may not even realize when you make the decision what the side effect is.  Even if you know, there may be a lot of other code required before the test that would require the behavior can be written. In the former case you obviously wouldn't write it, while in the latter it tends to get shoved to the bottom of your every expanding todo list and forgotten.

Thus there is always need when working on code to ask the question "are the existing tests in this area actually sufficient?" If the answer is no, then you need to write the missing tests.

But wait, what if my premiss is wrong.  There is also the possibility that the question is rooted in the statement that while the test itself good, it is outside the scope of the current story and review. This is actually a valid point, I'm all for breaking work up.  One advantage of great version control systems is when you are in code and see an opportunity where you want more tests. When you need to write a test that isn't related to the current work, just check the current work in, update your source, make the change, and check in, and go back to your main work.