Zend Framework application’s PHPUnit suite 3x faster

October 27, 2010

Uncategorized

The problem

As I’ve mentioned before, Zend Framework isn’t really encouraging developers to do TDD (on the other hand, this shouldn’t be a surprise in the PHP community). The application architecture is based on singletons all around (front controller, session, layout, helper broker, registry, etc.), the bootstrap doesn’t “separate the cacheable from the non-cacheable”, there’s no built-in support for dependency injection, the ORM is cumbersome to stub out.

Our approach used to involve calling Our_Db_Table_Abstract::truncate() on each table used by the given test case. Not only was this inelegant, mundane to maintain and easy to forget, but as it later turned out, it was the biggest performance bottleneck. We’d either put it at the end of the test case – which would not be executed if the assertion failed and we’d end up with a dirty db – or we’d put it in tearDown() – which meant it ran even if a particular test didn’t use the given table.

Another problem was controller tests. We needed to explicitly declare, which tables would be used by the test. Even with extra-simple syntax, it was still terribly inelegant. And if we’d forget about one table…

This is how it would look like in model tests:

<?php

class Our_Model_Admin_Employee_TableTest extends PHPUnit_Framework_TestCase
{
	function setUp()
	{
		$this->table = new Our_Model_Admin_Employee_Table;
	}

	function tearDown()
	{
		$this->table->truncate();
	}

	function testFindActiveFirstFetchDeletedAsLast()
	{
		$this->table->insert(array('email' => 'w@w.com'));
		$this->table->insert(array('email' => 'w22@w.com', 'dateDeleted' => new Zend_Db_Expr('NOW()')));
		$this->table->insert(array('email' => 'w2@w.com'));

		$actual = $this->table->findActiveFirst();
		$this->assertEquals(3, $actual->count());
		$this->assertNotNull($actual[2]->dateDeleted);
	}
}

class Our_Model_QuestionTest extends PHPUnit_Framework_TestCase
{
	function testUpdateAttributesSaveOnValid()
	{
		$question = Our_Model_Question::create();
		$question->updateAttributes(array('question' => 'q', 'answer' => 'a'));

		$this->assertEquals(1, $question->getTable()->count());

		$question->getTable()->truncate();
	}
}
  

And in the controller:

<?php

class Admin_ContentTranslationsControllerTest extends Our_Test_ControllerTestCase
{
	function testIndexAssignTranslationsNewFirst()
	{
		$this->useTable($table = new Our_Model_Translation_Table);
		$table->insert(array('locale' => 'el', 'key' => 'foo'));
		$table->insert(array('locale' => 'el', 'key' => 'bar', 'translation' => 'baz'));
		$this->get('index');
		$this->assertNull($this->assigns->translations[0]->translation);
		$this->assertEquals('baz', $this->assigns->translations[1]->translation);
	}
}

abstract class Our_Test_ControllerTestCase extends Zend_Test_PHPUnit_ControllerTestCase
{
	public $bootstrap = array('Application', 'bootstrap');
	protected $_usedTables = array();

	function tearDown()
	{
		foreach ($this->_usedTables as $table) {
			$table->truncate();
		}
	}

	function useTable(Our_Db_Table_Abstract $table)
	{
		$this->_usedTables[get_class($table)] = $table;
	}
}
  

As you can see, there’s a lot of verbosity, duplication and noise in the code. We need to do plenty of “plumbing” for the tasks that should be automatic.

The solution

The solution I came up with was inspired by Rails, and other frameworks. It can be summarized as: wrap each test case in a transaction, rolling it back after the test case.

I implemented the solution using a PHPUnit listener, hooking up to startTest() and endTest() callbacks. Here’s the appropriate code fragment:

<?php

class Our_Test_Listener_DatabaseTransaction implements PHPUnit_Framework_TestListener
{
	public function startTest(PHPUnit_Framework_Test $test)
	{
		Zend_Registry::get('database')->beginTransaction();
	}

	public function endTest(PHPUnit_Framework_Test $test, $time)
	{
		Zend_Registry::get('database')->rollBack();
	}
}
  

These callbacks wrap a single test case, so the cycle is:

  1. startTest()
  2. setUp()
  3. test*()
  4. tearDown()
  5. endTest()

And here’s the appropriate fragment of phpunit.xml (we’re using PHPUnit 3.5):


	
	

  

Excellent, that’s an elegant solution with a properly separated concern. I got rid of all the ugly truncate()s and sped up the tests significantly.

Caveats

Of course, life’s not that full of sunshine and bunnies.

Problem #1

First problem I encountered, was bootstrapping the application inside the cycle mentioned above (anywhere within steps 2-4). This happens in controller, view, view helper and bootstrap test cases. Let’s go through a controller test case. We:

  1. startTest(), which begins a transaction
  2. setUp(), which bootstraps the application, which in turn puts a new ‘database’ object in Zend_Registry
  3. endTest() calls rollBack(), which explodes with No active transaction

This basically means that we’re running beginTransaction() and rollBack() on a different database connection.

One solution for this problem is to use a persistent PDO connection. This will most likely cause both methods to run on two different PHP objects, but same connection resource. To do this, just add the following setting to your database config:

params.persistent = true

I wanted something more robust, though, so I rollBack() manually right before bootstrapping and beginTransaction() immediately afterwards. It’s plumbing again, but at least it’s abstracted in a base class. Here’s an example:

<?php

abstract class Our_Test_ControllerTestCase extends Zend_Test_PHPUnit_ControllerTestCase
{
	function setUp()
	{
		Zend_Registry::get('database')->rollBack();
		parent::setUp();
		Zend_Registry::get('database')->beginTransaction();
	}
}
  

Problem #2

Second caveat was quite obvious. Some of our tables were using MyISAM (which doesn’t support transactions), not InnoDB. I solved it by creating an appropriate migration, like so:

ALTER TABLE Country ENGINE = InnoDB;

Problem #3

Some tests were still failing after all these changes. The problem was that they were assuming primary key values for newly created rows, i.e. expected a new row to have an ID of 1. This was true when we were truncating, but is no longer true for rollbacks, since autoincrement counters are not reset. I refactored it in a following way:

<?php

class Admin_ContentTranslationsControllerTest extends Our_Test_ControllerTestCase
{
	function testEditFindTranslationByIdTheOldWay()
	{
		$this->table->insert(array('key' => 'foo'));
		$this->get('edit', array('id' => '1'));
		$this->assertEquals(1, $this->assigns->translation->id);
	}

	function testEditFindTranslationByIdTheNewWay()
	{
		$id = $this->table->insert(array('key' => 'foo'));
		$this->get('edit', array('id' => $id));
		$this->assertEquals($id, $this->assigns->translation->id);
	}
}
  

Problem #4

Not really a problem for me, but I should probably mention this. We could observe the performance boost on all developers’ machines with Ubuntu. Some of our colleagues are using Windows, and they had a very small gain in speed (around 5 seconds). But their benchmarks are generally tilted, since sometimes the suite would finish under 2 minutes, and sometimes after 5+ minutes. YMMV.

Conclusion

There you go, with a conceptually simple and elegant solution you can achieve quite a dramatic improvement in performance of PHPUnit suite for a Zend Framework application. My next idea is to cache the bootstrap input (i.e. routing). What’s yours?

And here’s the evidence for sceptics:

Before:

$ phpunit
PHPUnit 3.5.0 by Sebastian Bergmann.

............................................................  60 / 630
............................................................ 120 / 630
..............S.SSSS............................S........SS. 180 / 630
S........................................................... 240 / 630
..................................S.S....................... 300 / 630
............................................................ 360 / 630
.........................SS.......F.S...........S........... 420 / 630
.....................S...................................... 480 / 630
..........................I................................. 540 / 630
..............................................S............. 600 / 630
..............................

Time: 01:24, Memory: 136.75Mb

FAILURES!
Tests: 630, Assertions: 830, Failures: 1, Incomplete: 1, Skipped: 17.

After:

$ phpunit 
PHPUnit 3.5.0 by Sebastian Bergmann.

............................................................  60 / 630
............................................................ 120 / 630
..............S.SSSS............................S........SS. 180 / 630
S........................................................... 240 / 630
..................................S.S....................... 300 / 630
............................................................ 360 / 630
.........................SS.......F.S...........S........... 420 / 630
.....................S...................................... 480 / 630
..........................I................................. 540 / 630
..............................................S............. 600 / 630
..............................

Time: 29 seconds, Memory: 131.50Mb

FAILURES!
Tests: 630, Assertions: 830, Failures: 1, Incomplete: 1, Skipped: 17.

3 Responses to “Zend Framework application’s PHPUnit suite 3x faster”

  1. funkyfly Says:

    What if I’m using transactions in my applications? How to deal with them?

  2. tree2054 Says:

    Keep in mind that if your RDBMS has deferrable constraints that the constraint check could be deferred until you call COMMIT in which case you will never see any failures from constraint violations. This may or may not be ok depending on your application.

  3. alokin Says:

    There’s no need for creating custom method for truncating data, as Zend_Test_PHPUnit_DatabaseTestCase offers API for achieving same effect but on more easier and elegant way. You can find out more about that in manual: http://framework.zend.com/manual/en/zend.test.phpunit.db.html

    And also, if you need to combine database testing with the ControllerTestCase, there’s a solution, too:
    http://framework.zend.com/manual/en/zend.test.phpunit.db.html#zend.test.phpunit.db.testing.controllerintegration