In this article I introduce the topic of Acceptance Testing (aka Functional Testing), something more PHP programmers should be starting to practice. I’m sure many of us are well aware of Unit Testing and even Integration Testing so where does this third wheel come into play for web applications given our growing obsession with Web 2.0 and AJAX and how does it differ from the former two practices? Below I’ll explain this. I will also introduce how to implement Acceptance Testing using the killer combination of PHPUnit and Selenium.
Why Acceptance Tests?
Acceptance Testing is a high level testing procedure which ensures that an application behaves as expected by the client (whomever commissioned the application). Now that a definition is out of the way…
It’s sometimes difficult to see where Acceptance Testing fits into our testing practices. Some people will find it suspiciously similar (if not identical) to Integration Testing. Integration Testing is another high level testing process, distinct from Unit Testing, which verifies that high level components of the application work as expected. As with Unit Tests, this testing usually takes place isolated from the rest of the code using mocks, stubs and specialised test classes.
The primary differences from Acceptance Testing are pretty simple. Integration tests operate similarly to Unit Tests but on groups of associated classes. The group concept is essential. We’re not testing each class in isolation as with Unit Tests, but groups of classes which together generate a desired result. Integration Tests are therefore written by and for programmers. On the other hand, Acceptance Tests operate on a fully integrated application (no isolation of classes/components) normally testing against the user interface, whether HTML for browsers or the XML/JSON response from web services. They are written to assure an application performs its purpose as defined by the client. In fact, the client may even be responsible for writing the tests!
Now that we know Acceptance Testing is a distinct practice, why should we do it?
- It captures the expectations of the client (and not the developer!)
- It measures when functionality valued by the client is complete (a “knowing when to stop” signal)
- It ensures future behaviour which deviates from the expected is quickly identified (regression testing)
- Like any good set of tests they support refactoring in the same way as Unit and Integration Tests
Of User Stories and Acceptance Tests
For those who practice Extreme Programming, Acceptance Tests are normally written to verify that the implementation of a “User Story” is complete and stays that way over time. If you recall, I noted that Acceptance Tests are client driven to the point where they may even write them.
A User Story is a short description written by the client about some unit of valuable functionality the completed application will offer, i.e. the expected behaviour Acceptance Tests should verify. In a sense, XP replaces the traditional reliance on specification which tries to plan every step of development at the cost of rigidity with a more flexible approach where collections of User Stories track current requirements, and are maintained under the assumption that any release plans based on them are subject to frequent change.
Let’s take a quick foray into writing User Stories.
A customer may login to a personal account.
As PHP developers some of us may view this as a stupidly obvious requirement. Try not to. To the client this is a valuable feature. So stifle those giggles! Assuming any continuing discussion with the client raises no further changes to this User Story we can preempt any coding with a set of Acceptance Tests. The process likely sounds familiar to Unit Testers – test first, code later. You measure the success of an implementation based on whether it passes all its pre-written Acceptance Tests (in our case these test the web interface). Here, the client has put in some thought and after a discussion with the programmers written up the following tests.
- Login page displays a login form
- Submitting a valid identity/password combination results in a successful login.
- Submitting an invalid identity/password combination redisplays the login form with errors.
- The login form is always accompanied with a “Forgot identity or password?” hyperlink.
Pretty straight forward, right? As chance would have it, the fourth Acceptance Test is also a second User Story, an extra nugget of valuable functionality the client came up with after going through their conditions of success.
A customer may retrieve a forgotten password or identity.
If each of the four Acceptance Tests pass we can assume the two User Stories have been completed. Also, when the Acceptance Tests are passed it’s a clear signal that it’s time to stop programming for this feature. Unless the client comes up with new User Stories (or elaborations thereof) there’s no point throwing more resources at it. It’s done! Move on!
The Iteration Plan
Since our crack team of PHP code monkeys is at the forefront of Extreme Programming they have also been busy assigning User Stories to a specific “Iteration”. An Iteration, in broad terms, is a fixed period of development at the end of which we should have a fully tested, working (albeit incomplete) version of the application which will pass all Acceptance Tests for all User Stories assigned to that Iteration. I’m sure many of you have bumped into the Iteration Programming ideal before. An Iteration is typically no more than a few weeks long. Considering a project may take months there are going to be many Iterations on the release schedule, each building on the other.
Back to our example. The login User Stories have been assigned to Iteration #1. Secure that it’s time to implement the Stories, you whip out your favourite IDE/editor and set to. Next step? Write the Acceptance Tests in a form where automated testing is possible.
Preparing For Acceptance Testing
- PHPUnit 3.0 http://www.phpunit.de
- Testing_Selenium http://pear.php.net/
- Java 5 (1.5.0) is needed for Selenium RC http://java.sun.com
- Selenium Remote Control (RC) http://openqa.org/
Many PHP programmers tend towards a PHP library to write Acceptance Tests and this is the approach explained here. In this article we use PHPUnit which from 3.0 has been packaged with the PHPUnit_Extensions_SeleniumTestCase class which can be used to define Acceptance Tests which rely on Selenium Core to perform the testing. It remains important to note that these tests are not Unit Tests. Yes, PHPUnit is a Unit Testing library but the framework is just as useful for Integration and Acceptance Testing.
The PHPUnit Manual page for Selenium can be found at http://www.phpunit.de/pocket_guide/3.1/en/selenium.html. PHPUnit’s Selenium extension also requires that the PEAR Testing_Selenium package is installed. Both PHPUnit and Testing_Selenium can be installed from PEAR (PHPUnit 3 is only available from the phpunit.de PEAR channel so read the installation manual for it at http://www.phpunit.de/pocket_guide/3.1/).
Note that because it has been some time since the last public release of Selenium RC 0.9.0 (back in November 2006), this article will assume you have the courage to use a recent Selenium 0.9.2 snapshot. This is required since the public 0.9.0 version does not work well with recent versions of Firefox 2 and Internet Explorer 7. You can download the latest snapshot from <
The basic form of an Acceptance Test using PHPUnit and Selenium RC is very simple.
/** PHPUnit_Extensions_SeleniumTestCase */
class GoogleIndexTest extends PHPUnit_Extensions_SeleniumTestCase
protected function setUp()
* '*firefox' => Firefox 1 or 2
* '*iexplore' => Internet Explorer (all)
* '*custom /path/to/browser/binary => Other browsers (incl. Firefox on Linux)
* '*iehta' => Experimental Embedded IE
* '*chrome' => Experimental Firefox profile
$this->setBrowserUrl('http://www.google.ie/'); // set website being tested
public function testTitle()
$this->open('http://www.google.ie/'); // open the index page
$this->assertTitleEquals('Google'); // test the expected title is present
- Element Locators
Accessors examine state. They can be used to store values in variables for later use or comparison. They also automatically generate several Assertions. The advanced use of Accessors can be hugely helpful but we’re not covering them here.
Assertions are like accessors in that they examine state. However they also allow comparison to expected values. The two main types are prefixed with either “assert” or “verify”. In Selenium Core a failed assert will end a test run while a failed verify is logged without ending the test. To gain an informative test result, try to use assert methods only for essential assertions where a failure means continuing testing is pointless.
The full list of possible Actions, Accessors and Assertions is listed on the Selenium Core Reference at http://www.openqa.org/selenium-core/reference.html. There is quite a long list so there’s no end of fun to be had in testing.
Patterns are methods for specifying text strings to search for. The default method is “glob” (i.e. use of common */? syntax for wildcards in a search term). Also available is “exact” (matches exact text), and where would we be without “regexp”! We’ll see examples of some of these as we continue.
Implementing The User Stories (or Here’s One I Prepared Earlier)
To keep this article on point, we’ll assume the User Stories are already implemented. To do it justice you should download and install a local copy. The download includes all application code (written using the Zend Framework) and completed test code for PHPUnit and Selenium.
To install, just import /INSTALL.sql to a database, copy /src/config/config.ini.dist to /src/config.ini and edit for your specific details, and do the exact same for /tests/TestConfiguration.php. In this test config file, you can change the value of TESTS_SELENIUM_BROWSER for any browser option (see the list in our original GoogleIndex test). The baseurl option should also be edited to point to the app’s /www directory.
Finally? Assuming an installation to the document root: visit http://localhost/www/. You can run all tests we write and explain below by visiting: http://localhost/tests/AllTests.php
Writing And Running The Acceptance Tests
Let’s dig into our own testing now. I’m sure we’ve had enough theory at this point! So, what was the first Acceptance Test we had our client define?
1. Login page displays a login form
Our login form is location at the relative url “/login” from the application’s index location. This gives us the URL we want Selenium Core to open. Secondly our test should check for the existence of a login form. We’ve already (see /src/default/views/scripts/login_index.phtml) defined the login form and given it a unique id “login-form” which we can search for. We know from the same template that two input fields should also exist with ids of “identity” and “password”. To offer some extra insight we’re going to mix and match Element Locators in this test.
/** PHPUnit_Extensions_SeleniumTestCase */
class LoginIndexTest extends PHPUnit_Extensions_SeleniumTestCase
protected function setUp()
$this->setBrowser('*firefox'); // or *iexplore for IE
public function testLoginFormExists()
If you downloaded the pre-implemented application, you’ve already likely run this test. The full test file is located at /tests/AcceptanceTests/UserLoginTest.php. From above we can guess that adding browser and url options directly is only going to invite frequent editing – so the tests in the download use a set of Constants instead from the /tests/TestConfiguration.php.dist file.
Are these tests sufficient? There is a risk in that if we make the tests too specific we’ll spend a long time editing tests whenever the user interface design changes. However some extras should be tested. For example, does the maxlength attribute values match the database VARCHAR limits for these fields?
$this->assertElementPresent("xpath=//input[@id='identity' and @maxlength='20']");
$this->assertElementPresent("xpath=//input[@id='password' and @maxlength='64']");
Note: Since we’re using a snapshot of Selenium RC here we cannot yet rely on the PHPUnit getElementAttribute() method to return an attribute value. The next Selenium RC release should fix this but for now we’ll use an assertElementPresent(), instead of a typical assertEquals(), using an XPath query to check the maxlength values. Thus is the price of living on the edge (and dealing with rapidly evolving browsers!).
2. Submitting a valid identity/credential combination results in a successful login.
So we’ll append a submission test to our test case which will submit the form with valid data and check for the welcome message defined in the template.
public function testValidAuthentication()
// Fill out the form!
// Submit the form!
$this->waitForPageToLoad(30000); // 30 second default
// Verify the login was successful
// And that no Error is printed
Above we meet several new Action and Assertion methods. The first is type(). type() simulates a user typing data into a text field and we use it here to tell Selenium Core to fill the user details into the login form. click() defines an element for Selenium Core to simulate a click on (there is also a submit() method you could use instead). The additional waitForPageToLoad() method tells Selenium this click will generate a server request (i.e. the form submission) and it should wait for a response before continuing. PHPUnit and Selenium both also support a clickAndWait() method to combine click() and waitForPageToLoad(). As I write this it doesn’t appear functional on any system I’ve tried it with so I assume it’s another “next release” wait and see…
We also meet the assertTextPresent() and assertTextNotPresent() methods. Both of these assertions accept a Pattern. If you recall, Patterns are used to locate text on a page using a variety of search methods including glob, regexp and exact. Above we are searching for an exact match. If you did not define a Pattern type, the default “glob” option is assumed by Selenium Core. While this is not bad, you might fall afoul of locating strings which carry glob characters like an asterix or question mark.
3. Submitting an invalid email/password combination redisplays the login form with errors.
public function testInvalidAuthenticationWithError()
// Fill out the form!
// Submit the form!
$this->waitForPageToLoad(30000); // 30 secs
// Verify the login was unsuccessful...
// ...and that an Error exists!
Nothing new here except for the use of a regexp Pattern since an error could display any valid login name. So we take a broad approach and invalidate the presence of any conceivable username. This would mean the regexp Pattern matching our application’s validation filter logic as close (or a little more broadly) as possible.
Finally, our fourth test!
4. The login form is always accompanyied with a “Forgot identity or password?” hyperlink.
This is less a new test, and more a modification of our testLoginFormExists test case. We just append the following:
public function testLoginFormExists()
// assert the "Forgot Password" link is present
$this->assertElementPresent('link=regexp:^Forgot identity or password?$');
When we put this entire test file together we finally have Acceptance Tests sufficient to verify that the User Story has been implemented, is present, and works. It’s likely minimal since we’re not including developer oriented tests such as checking the maxlength attribute values of input fields to ensure they correspond to our database VARCHAR sizes but it’ll do for now.
How to run the tests?
1. Start the Selenium RC server
java -jar selenium-server-standalone.jar
You might want to ensure your browsers executable and java are appended to your system’s PATH variable first.
2. Run the AllTests.php file from the download in the /tests directory from the browser
If all goes well, you’ll see three browser instances being spawned and closed (you could use Selenium RC’s proxy injection option to reuse the same browser session), and the results will be printed to the browser as a PHPUnit text result. If this does not happen there are a few possible causes. Firstly check you’re using the snapshot version of the Selenium RC Server (it’s much better at operating recent browsers at the cost of a handful of bugs described earlier). Secondly try disabling any active internet connections which can sometimes interfere with the Selenium RC server (esp. with Internet Explorer). You can also check the Selenium documentation online for some other scenarios. One less common issue is tested across the internet using a dial up connection (you can’t change IE’s proxy settings on dial up or VPN) in which case use Firefox or another alternative.
Acceptance Testing is a valuable practice to learn. In assessing how our applications operate in reality against a set of testable expectations we can measure completion and maintain that assurance across time. In the absence of Acceptance Tests we have no such assurance and we’ll find ourselves relying more and more on manual testing of the application. Manual testing requires far more time, and it’s less certain in whether it captures all potential problems. Likewise, Integration and Unit Testing is not a replacement for proper Acceptance Testing.
With PHPUnit and Selenium, or indeed let’s not forget alternatives like SimpleTest and it’s webtest support (or its Selenium support in CVS!), Acceptance Testing is a simple affair which requires little practice to get started. The journey is worthwhile and I hope this article helps you on your way!