Unit Testing Strategies

      1 Comment on Unit Testing Strategies

If you read any book or guide on Unit Testing, it normally starts with an exercise like this:

While this demonstrates what a Unit Test is, how useful is it? Not at all.

If we are starting from absolutely nothing, building tests as we go is a near-trivial process. Ideally, we can build our core components in a TDD (Test Driven Development) way but the worst case is that we write the tests immediately after the functions themselves. This is a reasonable strategy and can keep things functioning as we expect long into the future.

Unfortunately, most of us don’t have this situation. We walk into a project with documentation ranging from non-existent to seemingly-useful-but-really-out-of-date, and hundreds of thousands of lines of code, and the coding standards and practices of every developer that has come before us. In this situations, the above Unit Test example is almost mocking us. We can’t start that simply!

Or can we?

When Trevor Morse and I started chatting on Unit Tests for web2project, we faced the same problem. We had well over 150kloc, zero tests, and the system was not built for testing. After quite a bit of deliberation, we decided on a two-prong strategy:

First, we dug around our bug tracking system to find the most problematic pieces of code. Luckily, the data was well-organized, so it was easy to track down the top few modules which generated the most bug reports. This was code actively causing problems for our users and it was worthwhile to address it. But then we added a second aspect to the “problematic” qualification to include code that was changing quickly. Why is this important? Generally, when code changes more often, it means more people are editing it. With more people involved, you have a wider variety of coding practices, understandings, and interpretations. It’s also possible that definitions and plans change over time. All of these reasons make the code a good candidate for Unit Tests.

Unfortunately, while these tests are useful and powerful, they’re also incredibly complex and likely to cause the biggest headaches in setUp() and tearDown(). It’s even worse when the system isn’t built for testing in the first place. In quite a few places, we found it was better to refactor existing functions into a series of function calls which are individually testable. Not ideal, but a really good start.

Next, we dug around in the system until we found the common core functions which are used throughout the system by a variety of modules. The most important part here is that breaking one of these functions would cause errors and/or odd behavior in modules all over the system. This also makes it these functions the most dangerous. An inexperienced user or tester would see errors and may report them without noticing the common symptoms all over. Once you have a pile of (unintentionally) misleading bug reports, you’re guaranteed to have a bad day.

In our specific case, this included functionality like date parsing and formatting, currency formatting, various Date manipulations, and our translation layer. In this example, we perform tests to make sure the language-specific translation files are loaded:

Realistically, while this test looks complicated at first pass, it’s not too far beyond the simple example above. We set the initial language, make some assertions, change the language, check a few more, and then switch back and test the original assertions. Changing the language is one of the first things our international users do after installation, so we need it to just work.

Overall, our Code Coverage is low but there’s no particular percentage on our priority list. Our goal is to improve the stability and reliability of the system incrementally but tangibly with each and every step. By adding Unit Tests based on the above criteria, we’ve closed dozens of bugs long before release and usually before they were even committed to our repository. And that is the point of Unit Testing.

Disclosure: This post overlaps with my presentation on the upcoming CodeWorks tour. I get into most specifics there. Further, I use web2project as an example quite a bit. It works well because there are no NDA’s involved and I don’t have to worry about letters from annoyed lawyers.

About Keith Casey

This should be something about myself. I've had suggestions that it should be "useful information" but I'm not sure about that.