Testing timers

In the testing world very little is said about how to test timers. The little advice given seems to be you isolate it, and code review those isolated areas instead of testing. That advice is okay for a lot of applications: waiting is generally a bad thing. Nobody finishes preparing a web page and then puts in a delay before sending it. As such timers are generally only in a couple (not very critical) places, and not a core part of any business logic.

What if you don't have that luxury, what if time is what your business logic is about. My team is on an embedded system. A large part of our business logic related to timers. For example, sending keep alive messages every 200ms, unless some other message has been sent. We have to deal with a number of different hardware devices with different rules, so getting all the logic correct is a significant source of potential bugs. To make it harder, we need to get certification from a third party on each release, mess up one of the rules and we can't sell anything. To just rely on code review isn't enough: we need tests that use timers.

A few years ago one of my teammates injected a timer into his class. It seemed like a good idea, now that it is injected he can inject a mock and test his time using class without waiting for real time to pass. Since his requirement was a 20 second timeout, and there were several border cases to test, this brought his test time down from over a minute to a few milliseconds. As a bonus, his tests could run on heavily loaded machine (ie the CI system) without random failures. What is not to love?

For several years we continued that way. It was great, our tests were fast, and deterministic despite testing time which is notorious for leading to slow, fragile tests. Somehow it always felt a little icky to me though. It took me several years before I figured out why, and then when reading someone else's code review out of context of what his program did. Timers doesn't feel like a dependency, they feels like an implementation detail. As such injecting timers leads to the wrong architecture.

Here is my surprising conclusion: timers are shared global state and should be used that way!

That still leaves the problem of how to test timers open, and those are real enough problems. I solved it by embracing global state. In C the linker will take the first function of a name it finds. We (my pair and I) took advantage of that fact to write a fake timer, with all the timer functions, but a different implementation. You set a timer callback and it goes to a list, but no real timer is created.  Then we wrote and an advance time function that went through the list and triggered all the timers that expired in order. Last, we told the linker to use our new timer not the real thing in our tests.

Testing with the fake timer feels more natural than with injected test doubles: when using a double you tell each timer to fire, but you have to actually know the how the implementation uses the timer. The fake though just has one interface: AdvanceTime(). Advance the time 10 seconds, and the right things happens.  A 1 second timer will fire 10 times, while a 7 second timer fires once - between the 1 second timer firing 7 and 8 times (or maybe 6 and 7 times - depending on ordering).  If the 7 second timer starts a 2 second timer, than that new timer will fire at the simulated 9 seconds. The total time for this entire test: a couple milliseconds!

One additional bonus, our timer simulates time well. If you have a 200ms timer, advance time 50ms, and then restart the timer it will handle that correctly. If you cancel a timer it is canceled. Those testing with a mock timer had a lot of work to handle those cases, while it is easy with the fake. When the implementation decides to change the way the timer is used a bunch of unrelated tests broke, while the fake just handles that refactoring. Our timer supports an isRunning function - mocks typically did not model this state (and those that did often got it wrong somehow), while the fake handles all the special cases to ensure that when you ask is your timer running you get the right result.

It seems cool, even several months later. Maybe it is? I'm a little scared that as the inventor I'm missing something. I offer it to you though as something to try.