After attending the Silicon Valley Ruby Conference and seeing the Selenium, the open-source web-testing framework in action, I decided to give it a try. Selenium makes it easy to write test scripts using a macro language done in pure javascript. Here's a sample script in "Selenese" that performs a login action.
|open|/news||
|type|user_screen_name|johnny|
|type|user_password|password|
|click|commit||
If you ignore the '|' which is a separator, its just plain english. If you don't want to manually type in Selenese, you can download Selenium IDE, a macro recorder that converts your actions into test scripts. You can also take a look at this excellent
screencast.
So far so good, you can automate tests in just 2 steps: 1) Use macro recorder to record a script 2) run the script as a test. But you still need to verify the results of your tests. Selenium can parse the contents of a webpage using regular expressions or XPATH. Eg. to verify that the login test passed, you can search for the phrase "Welcome, XXXXXX".
The real question is whether Selenium can test all aspects of your website. The answer is yes, but you probably don't want to do it that way. Selenium is a pure javascript framework so there are somethings it cannot do. It cannot fill in the upload box of a javascript form because of security restrictions. It cannot access state variables such as the session or the database because it runs only in your browser. See my last post on Pivotal Labs: they have created a Rails plugin which runs Selenium in a separate process and allows full access to Rails framework. But it is necessary? I don't think so -- at least not for me.
Lets not forget, Selenium is designed to run inside the browser so it can test the user experience. You want to verify that the user's actions produce the correct results in the browser and not just in the database. All this requires a little bit of creativity and some javascript. Take the case where the user registers and receives an email with confirmation code. Selenium can fill out the user registration form but can't see the email. So I add a test-only method that will render the email message as a HTML page.
Test Script
1) User registers as "bob@yahoo.copm"
2) Call test method to render email message for "bob@yahoo.com"
3) Use javascript to extract registration code of "XXXXX"
4) Goto page /user/confirm?code=XXXXX
5) Verify user is shown his new profile page
The entire test takes place inside the browser. The best part is there is no need to run Selenium in a separate process. Executing the test is very simple (checkout Selenium on Rails), open testrunner in your browser and run your test suite. I have been able to do all of testing using a simple setup. There are scenarios that are difficult to test, eg. uploading files as I mentioned earlier. But some of that you can do inside functional or unit tests. Overall, I have found Selenium to be an indepensible piece of the testing suite, precisely because it tests what matters to your end-users, what they see in their browser.
