Are integration tests the answer?

I have been picking on unit tests a lot lately. The obvious question is do I think that integration testing the answer?

Before I can answer this, we need a definition of integration test. Just like unit test, the definition of integration test goes back to long before automated tests. A integration test is any test that combines two or more units in a single test, with the purpose of testing interaction between units. Many other authors have attempted to redefine integration tests to something that makes more sense in an automation test world.

Back to the question, what about more integration tests? The standard answer is no: when an integration test fails there are many possible reasons, which means you waste time trying to figure out where things broke. It is agreed that when a test fails you should know exactly what part of code broke. Since an integration test covers a large part of the code the failure could be anywhere.

I question that answer. Sure in theory code could break for many reasons. However in the real world there is exactly one reason a test failed: the code you have touched in the last minute broke something. The point of automated tests is we run them all the time - several times a minute is the goal, and once a minute is common. Even the fastest typists cannot write much code in a minute: this leaves a tiny number of places to look for the failure. If a large integration test breaks you already have the root cause isolated to a couple lines of code. As a bonus that area is in your short term memory! (sometimes the solution is changing code elsewhere which is hard, but where the problem was introduced is obvious)

Unfortunately there are other problems with integration tests that are also used as reasons not to write them. These reasons are valid, and you need to understand the tradeoffs in detail before you write any tests.

The first problem with integration tests is they tend to run long. If you cannot run your entire test suite in 10 seconds (or less!) you need to do something. I might write about this latter, but a short (probably incomplete) summary of things to do. Use your build system to only run tests that actually test the code you change. Profile your tests and make them run faster. Split them into suites that can run in parallel. Split them into parts with a scheme where some run all the time, some less often. Use these tricks to get your test times down. There is one more option that deserves discussion: test smaller areas of code, this can get your test times down - at the expense of all the problems of unit tests.

A second problem is integration tests are fragile because of time. Time is easy to isolate in units - most of which don't use time anyway, but the larger the integration the more likely it is that something will fail because of time issues. I outlined a potential solution in my testing timers post, but this may not work for you.

Requirements change over time. This change is more likely to hit integration tests because you are testing a large part of the system. When your tests are tiny, there is correspondingly only a few possible ways the test can break. Larger tests have a larger surface to break when something changes. Thus integration tests are more subject to change. This is not always bad: sometimes a new feature cannot work with some existing feature, and the failing test is the first time anyone realizes the subtle I reason why. Failing tests are often a sign you need a conversation with the business analysts to understand what should happen. 

An important variation of the above, the user interface is likely to change often. When you have a feature working you are unlikely to change the code behind it. However the UI for feature not only has to let you use the feature, it also needs to look nice. Look nice is subjective style which changes over time. If your tests all depend on looking for a particular shade of yellow then every time tastes change, a bunch of tests need to change. A partial solution to this is get a UI test framework that knows about the objects on the screen, instead of looking for pixel positions and colors. The objects will change much less often, but even this isn't enough, a widget might move to a whole new screen which again can break a lot of tests.

Fragile can also mean the integration test doesn't inject test doubles like a unit test would. It is very useful to write these tests: the only way to know if you are using an API correctly is to use it. However anytime an API is used instead of a test double you take the risk that something real might/might not be there that breaks the test. A test that needs a special environment can be useful to ensure that your code works in that environment, but it also means anyone without that environment cannot run that test this is a tradeoff you need to evaluate yourself.

Perhaps the biggest problem with integration tests is sometimes you know an error is possible, but not how to create it. For example, filling your disk with data just to test disk full errors isn't very friendly. Worse there may not be a way to ensure the disk fills up at the right time leaving some error paths not tested. This is only one of the many possible errors you can get in a real world system that you need to handle, but probably cannot simulate well without test doubles.

The above is probably not even a complete list.

So do I think you should write integration tests? No, but only because the definition is too tied to the manual testing/unit testing world. What we need is something like integration tests, but without the above problems. There is no silver bullet here: at some time, no matter which testing philosophy you use, you will encounter a limit where you need to examine trade offs and decide what price to pay.