Zend Framework application’s PHPUnit suite 3x faster

The problem

As I’ve mentioned before, Zend Framework isn’t really encouraging developers to do TDD (on the other hand, this shouldn’t be a surprise in the PHP community). The application architecture is based on singletons all around (front controller, session, layout, helper broker, registry, etc.), the bootstrap doesn’t “separate the cacheable from the non-cacheable”, there’s no built-in support for dependency injection, the ORM is cumbersome to stub out.

Our approach used to involve calling Our_Db_Table_Abstract::truncate() on each table used by the given test case. Not only was this inelegant, mundane to maintain and easy to forget, but as it later turned out, it was the biggest performance bottleneck. We’d either put it at the end of the test case – which would not be executed if the assertion failed and we’d end up with a dirty db – or we’d put it in tearDown() – which meant it ran even if a particular test didn’t use the given table.

Another problem was controller tests. We needed to explicitly declare, which tables would be used by the test. Even with extra-simple syntax, it was still terribly inelegant. And if we’d forget about one table…

This is how it would look like in model tests:

And in the controller:

As you can see, there’s a lot of verbosity, duplication and noise in the code. We need to do plenty of “plumbing” for the tasks that should be automatic.

The solution

The solution I came up with was inspired by Rails, and other frameworks. It can be summarized as: wrap each test case in a transaction, rolling it back after the test case.

I implemented the solution using a PHPUnit listener, hooking up to startTest() and endTest() callbacks. Here’s the appropriate code fragment:

These callbacks wrap a single test case, so the cycle is:

  1. startTest()
  2. setUp()
  3. test*()
  4. tearDown()
  5. endTest()

And here’s the appropriate fragment of phpunit.xml (we’re using PHPUnit 3.5):

Excellent, that’s an elegant solution with a properly separated concern. I got rid of all the ugly truncate()s and sped up the tests significantly.

Caveats

Of course, life’s not that full of sunshine and bunnies.

Problem #1

First problem I encountered, was bootstrapping the application inside the cycle mentioned above (anywhere within steps 2-4). This happens in controller, view, view helper and bootstrap test cases. Let’s go through a controller test case. We:

  1. startTest(), which begins a transaction
  2. setUp(), which bootstraps the application, which in turn puts a new ‘database’ object in Zend_Registry
  3. endTest() calls rollBack(), which explodes with No active transaction

This basically means that we’re running beginTransaction() and rollBack() on a different database connection.

One solution for this problem is to use a persistent PDO connection. This will most likely cause both methods to run on two different PHP objects, but same connection resource. To do this, just add the following setting to your database config:

I wanted something more robust, though, so I rollBack() manually right before bootstrapping and beginTransaction() immediately afterwards. It’s plumbing again, but at least it’s abstracted in a base class. Here’s an example:

Problem #2

Second caveat was quite obvious. Some of our tables were using MyISAM (which doesn’t support transactions), not InnoDB. I solved it by creating an appropriate migration, like so:

Problem #3

Some tests were still failing after all these changes. The problem was that they were assuming primary key values for newly created rows, i.e. expected a new row to have an ID of 1. This was true when we were truncating, but is no longer true for rollbacks, since autoincrement counters are not reset. I refactored it in a following way:

Problem #4

Not really a problem for me, but I should probably mention this. We could observe the performance boost on all developers’ machines with Ubuntu. Some of our colleagues are using Windows, and they had a very small gain in speed (around 5 seconds). But their benchmarks are generally tilted, since sometimes the suite would finish under 2 minutes, and sometimes after 5+ minutes. YMMV.

Conclusion

There you go, with a conceptually simple and elegant solution you can achieve quite a dramatic improvement in performance of PHPUnit suite for a Zend Framework application. My next idea is to cache the bootstrap input (i.e. routing). What’s yours?

And here’s the evidence for sceptics:

Before:

After: