Acceptance Testing of Web Applications with PHP

Introduction

In this article I introduce the topic of Acceptance Testing (aka Functional Testing), something more PHP programmers should be starting to practice. I’m sure many of us are well aware of Unit Testing and even Integration Testing so where does this third wheel come into play for web applications given our growing obsession with Web 2.0 and AJAX and how does it differ from the former two practices? Below I’ll explain this. I will also introduce how to implement Acceptance Testing using the killer combination of PHPUnit and Selenium.

Why Acceptance Tests?

Acceptance Testing is a high level testing procedure which ensures that an application behaves as expected by the client (whomever commissioned the application). Now that a definition is out of the way…

It’s sometimes difficult to see where Acceptance Testing fits into our testing practices. Some people will find it suspiciously similar (if not identical) to Integration Testing. Integration Testing is another high level testing process, distinct from Unit Testing, which verifies that high level components of the application work as expected. As with Unit Tests, this testing usually takes place isolated from the rest of the code using mocks, stubs and specialised test classes.

The primary differences from Acceptance Testing are pretty simple. Integration tests operate similarly to Unit Tests but on groups of associated classes. The group concept is essential. We’re not testing each class in isolation as with Unit Tests, but groups of classes which together generate a desired result. Integration Tests are therefore written by and for programmers. On the other hand, Acceptance Tests operate on a fully integrated application (no isolation of classes/components) normally testing against the user interface, whether HTML for browsers or the XML/JSON response from web services. They are written to assure an application performs its purpose as defined by the client. In fact, the client may even be responsible for writing the tests!

Now that we know Acceptance Testing is a distinct practice, why should we do it?

  1. It captures the expectations of the client (and not the developer!)
  2. It measures when functionality valued by the client is complete (a “knowing when to stop” signal)
  3. It ensures future behaviour which deviates from the expected is quickly identified (regression testing)
  4. Like any good set of tests they support refactoring in the same way as Unit and Integration Tests

Of User Stories and Acceptance Tests

For those who practice Extreme Programming, Acceptance Tests are normally written to verify that the implementation of a “User Story” is complete and stays that way over time. If you recall, I noted that Acceptance Tests are client driven to the point where they may even write them.

A User Story is a short description written by the client about some unit of valuable functionality the completed application will offer, i.e. the expected behaviour Acceptance Tests should verify. In a sense, XP replaces the traditional reliance on specification which tries to plan every step of development at the cost of rigidity with a more flexible approach where collections of User Stories track current requirements, and are maintained under the assumption that any release plans based on them are subject to frequent change.

Let’s take a quick foray into writing User Stories.

A customer may login to a personal account.

As PHP developers some of us may view this as a stupidly obvious requirement. Try not to. To the client this is a valuable feature. So stifle those giggles! Assuming any continuing discussion with the client raises no further changes to this User Story we can preempt any coding with a set of Acceptance Tests. The process likely sounds familiar to Unit Testers – test first, code later. You measure the success of an implementation based on whether it passes all its pre-written Acceptance Tests (in our case these test the web interface). Here, the client has put in some thought and after a discussion with the programmers written up the following tests.

  1. Login page displays a login form
  2. Submitting a valid identity/password combination results in a successful login.
  3. Submitting an invalid identity/password combination redisplays the login form with errors.
  4. The login form is always accompanied with a “Forgot identity or password?” hyperlink.

Pretty straight forward, right? As chance would have it, the fourth Acceptance Test is also a second User Story, an extra nugget of valuable functionality the client came up with after going through their conditions of success.

A customer may retrieve a forgotten password or identity.

If each of the four Acceptance Tests pass we can assume the two User Stories have been completed. Also, when the Acceptance Tests are passed it’s a clear signal that it’s time to stop programming for this feature. Unless the client comes up with new User Stories (or elaborations thereof) there’s no point throwing more resources at it. It’s done! Move on!

The Iteration Plan

Since our crack team of PHP code monkeys is at the forefront of Extreme Programming they have also been busy assigning User Stories to a specific “Iteration”. An Iteration, in broad terms, is a fixed period of development at the end of which we should have a fully tested, working (albeit incomplete) version of the application which will pass all Acceptance Tests for all User Stories assigned to that Iteration. I’m sure many of you have bumped into the Iteration Programming ideal before. An Iteration is typically no more than a few weeks long. Considering a project may take months there are going to be many Iterations on the release schedule, each building on the other.

Back to our example. The login User Stories have been assigned to Iteration #1. Secure that it’s time to implement the Stories, you whip out your favourite IDE/editor and set to. Next step? Write the Acceptance Tests in a form where automated testing is possible.

Preparing For Acceptance Testing

Requirements:

Many PHP programmers tend towards a PHP library to write Acceptance Tests and this is the approach explained here. In this article we use PHPUnit which from 3.0 has been packaged with the PHPUnit_Extensions_SeleniumTestCase class which can be used to define Acceptance Tests which rely on Selenium Core to perform the testing. It remains important to note that these tests are not Unit Tests. Yes, PHPUnit is a Unit Testing library but the framework is just as useful for Integration and Acceptance Testing.

The PHPUnit Manual page for Selenium can be found at http://www.phpunit.de/pocket_guide/3.1/en/selenium.html. PHPUnit’s Selenium extension also requires that the PEAR Testing_Selenium package is installed. Both PHPUnit and Testing_Selenium can be installed from PEAR (PHPUnit 3 is only available from the phpunit.de PEAR channel so read the installation manual for it at http://www.phpunit.de/pocket_guide/3.1/).

Selenium is a web application Acceptance Testing tool created by ThoughtWorks. It includes several packages including Selenium Core, Selenium RC and Selenium IDE. It’s purpose is to run tests in a real browser (with all the warts each comes with!) using Selenium Core to perform user actions, tests and reporting of test results. Selenium Core is written in Javascript so it makes a powerful “BrowserBot”. Selenium Remote Control is an additional server process you need to run the tests in this article. It’s purpose is to allow any programming language to interact with Selenium Core running in the browser by passing simple HTTP GET requests to the RC server. It all sounds complex but it’s a doddle to work with.

Note that because it has been some time since the last public release of Selenium RC 0.9.0 (back in November 2006), this article will assume you have the courage to use a recent Selenium 0.9.2 snapshot. This is required since the public 0.9.0 version does not work well with recent versions of Firefox 2 and Internet Explorer 7. You can download the latest snapshot from <>. After extracting the package you only need one JAR file called “selenium-server-standalone.jar”. You need no other file from the snapshot package.

The basic form of an Acceptance Test using PHPUnit and Selenium RC is very simple.

The setUp() method is used to setup our test, starting here by determining which browser to use. Firefox is just one option. If Selenium RC does not include a default reference for your preferred Javascript-enabled browser you can use the “*custom” prefix to set the path to your selected browser. setBrowserUrl() sets a base URL from which all tests are run. This will typically be the index directory of your application (i.e. where index.php may be located). A page is only ever available for testing when it’s opened (if that’s not obvious). Therefore we start each test case with an open() call to the URL of the page to commence testing with.

Selenium Core, the Javascript BrowserBot running in the browser, is simple enough to understand with some practice. There are five concepts you first need to get your head around:

  1. Actions
  2. Accessors
  3. Assertions
  4. Element Locators
  5. Patterns

Actions are simply all actions Javascript can expect a user to perform and which can manipulate the web interface’s state. This includes clicking links, pressing keys, mouse positions (e.g. onmouseover), etc. Anything a user can do, Selenium Actions can do too. Many actions can be told to wait for a condition. The simplest is the “AndWait” suffix which can be attached to actions like click(), e.g. clickAndWait(). This tells Selenium Core our action will send a request to the web server and it should wait for a response before continuing. However due to the ever shifting nature of Selenium RC I recommend you avoid the “WaitFor” suffix (fails miserably in the current version, and the snapshot) and use a separate call to “waitForPageToLoad()” – we’ll see this in action later.

Accessors examine state. They can be used to store values in variables for later use or comparison. They also automatically generate several Assertions. The advanced use of Accessors can be hugely helpful but we’re not covering them here.

Assertions are like accessors in that they examine state. However they also allow comparison to expected values. The two main types are prefixed with either “assert” or “verify”. In Selenium Core a failed assert will end a test run while a failed verify is logged without ending the test. To gain an informative test result, try to use assert methods only for essential assertions where a failure means continuing testing is pointless.

Hint: When testing AJAX enabled web interfaces the “WaitFor” methods come in extremely handy for determining when an AJAX manipulation of the underlying DOM has occured, after which you can follow up with more traditional Verify/Assert tests of the altered page. The King here is waitForCondition() which accepts a Javascript expression which is continually evaluated until it’s true or it times out (timing out counts as a failure).

The full list of possible Actions, Accessors and Assertions is listed on the Selenium Core Reference at http://www.openqa.org/selenium-core/reference.html. There is quite a long list so there’s no end of fun to be had in testing.

The final two concepts are Element Locators and Patterns. An Element Locator is a method for locating an HTML/XML element for testing. Selenium Core supports Locators using id and name attributes, Javascript DOM expressions, XPath, CSS Selectors, and Link text. Another current Selenium issue is that using the Selenium RC renders the CSS Selector Locators unusable for now. It’s a pity, but the next public release should fix this.

Patterns are methods for specifying text strings to search for. The default method is “glob” (i.e. use of common */? syntax for wildcards in a search term). Also available is “exact” (matches exact text), and where would we be without “regexp”! We’ll see examples of some of these as we continue.

Implementing The User Stories (or Here’s One I Prepared Earlier)

To keep this article on point, we’ll assume the User Stories are already implemented. To do it justice you should download and install a local copy. The download includes all application code (written using the Zend Framework) and completed test code for PHPUnit and Selenium.

http://downloads.astrumfutura.org/devzone/Acceptance_Testing_Tutorial_Application.tar.bz2

To install, just import /INSTALL.sql to a database, copy /src/config/config.ini.dist to /src/config.ini and edit for your specific details, and do the exact same for /tests/TestConfiguration.php. In this test config file, you can change the value of TESTS_SELENIUM_BROWSER for any browser option (see the list in our original GoogleIndex test). The baseurl option should also be edited to point to the app’s /www directory.

Finally? Assuming an installation to the document root: visit http://localhost/www/. You can run all tests we write and explain below by visiting: http://localhost/tests/AllTests.php

Writing And Running The Acceptance Tests

Let’s dig into our own testing now. I’m sure we’ve had enough theory at this point! So, what was the first Acceptance Test we had our client define?

1. Login page displays a login form
Our login form is location at the relative url “/login” from the application’s index location. This gives us the URL we want Selenium Core to open. Secondly our test should check for the existence of a login form. We’ve already (see /src/default/views/scripts/login_index.phtml) defined the login form and given it a unique id “login-form” which we can search for. We know from the same template that two input fields should also exist with ids of “identity” and “password”. To offer some extra insight we’re going to mix and match Element Locators in this test.

If you downloaded the pre-implemented application, you’ve already likely run this test. The full test file is located at /tests/AcceptanceTests/UserLoginTest.php. From above we can guess that adding browser and url options directly is only going to invite frequent editing – so the tests in the download use a set of Constants instead from the /tests/TestConfiguration.php.dist file.

Above we meet assertElementPresent(). This checks for the existence of a defined HTML/XML element. The argument is a string where the Element Locator type is defined to the left of the equals sign, and the argument for that Element Locator is defined on the right. I’ve introduced three differing Element Locators. An ID checks the existence of an element with id=”login-form”. The existence of the two input fields is checked using Javascript DOM expressions. The presense of a submit button is checked using an XPath query. As noted earlier, CSS Selectors are not yet supported with Selenium RC (coming soon I hope!).

Are these tests sufficient? There is a risk in that if we make the tests too specific we’ll spend a long time editing tests whenever the user interface design changes. However some extras should be tested. For example, does the maxlength attribute values match the database VARCHAR limits for these fields?

Note: Since we’re using a snapshot of Selenium RC here we cannot yet rely on the PHPUnit getElementAttribute() method to return an attribute value. The next Selenium RC release should fix this but for now we’ll use an assertElementPresent(), instead of a typical assertEquals(), using an XPath query to check the maxlength values. Thus is the price of living on the edge (and dealing with rapidly evolving browsers!).

2. Submitting a valid identity/credential combination results in a successful login.
So we’ll append a submission test to our test case which will submit the form with valid data and check for the welcome message defined in the template.

Above we meet several new Action and Assertion methods. The first is type(). type() simulates a user typing data into a text field and we use it here to tell Selenium Core to fill the user details into the login form. click() defines an element for Selenium Core to simulate a click on (there is also a submit() method you could use instead). The additional waitForPageToLoad() method tells Selenium this click will generate a server request (i.e. the form submission) and it should wait for a response before continuing. PHPUnit and Selenium both also support a clickAndWait() method to combine click() and waitForPageToLoad(). As I write this it doesn’t appear functional on any system I’ve tried it with so I assume it’s another “next release” wait and see…

We also meet the assertTextPresent() and assertTextNotPresent() methods. Both of these assertions accept a Pattern. If you recall, Patterns are used to locate text on a page using a variety of search methods including glob, regexp and exact. Above we are searching for an exact match. If you did not define a Pattern type, the default “glob” option is assumed by Selenium Core. While this is not bad, you might fall afoul of locating strings which carry glob characters like an asterix or question mark.

3. Submitting an invalid email/password combination redisplays the login form with errors.

Nothing new here except for the use of a regexp Pattern since an error could display any valid login name. So we take a broad approach and invalidate the presence of any conceivable username. This would mean the regexp Pattern matching our application’s validation filter logic as close (or a little more broadly) as possible.

Finally, our fourth test!

4. The login form is always accompanyied with a “Forgot identity or password?” hyperlink.

This is less a new test, and more a modification of our testLoginFormExists test case. We just append the following:

When we put this entire test file together we finally have Acceptance Tests sufficient to verify that the User Story has been implemented, is present, and works. It’s likely minimal since we’re not including developer oriented tests such as checking the maxlength attribute values of input fields to ensure they correspond to our database VARCHAR sizes but it’ll do for now.

How to run the tests?

1. Start the Selenium RC server

You might want to ensure your browsers executable and java are appended to your system’s PATH variable first.

2. Run the AllTests.php file from the download in the /tests directory from the browser

If all goes well, you’ll see three browser instances being spawned and closed (you could use Selenium RC’s proxy injection option to reuse the same browser session), and the results will be printed to the browser as a PHPUnit text result. If this does not happen there are a few possible causes. Firstly check you’re using the snapshot version of the Selenium RC Server (it’s much better at operating recent browsers at the cost of a handful of bugs described earlier). Secondly try disabling any active internet connections which can sometimes interfere with the Selenium RC server (esp. with Internet Explorer). You can also check the Selenium documentation online for some other scenarios. One less common issue is tested across the internet using a dial up connection (you can’t change IE’s proxy settings on dial up or VPN) in which case use Firefox or another alternative.

Conclusion

Acceptance Testing is a valuable practice to learn. In assessing how our applications operate in reality against a set of testable expectations we can measure completion and maintain that assurance across time. In the absence of Acceptance Tests we have no such assurance and we’ll find ourselves relying more and more on manual testing of the application. Manual testing requires far more time, and it’s less certain in whether it captures all potential problems. Likewise, Integration and Unit Testing is not a replacement for proper Acceptance Testing.

With PHPUnit and Selenium, or indeed let’s not forget alternatives like SimpleTest and it’s webtest support (or its Selenium support in CVS!), Acceptance Testing is a simple affair which requires little practice to get started. The journey is worthwhile and I hope this article helps you on your way!