Over the last day or so I've been researching to see if I could find a decent tool for automating Web UI tests. I work on a number of projects where automating the (currently manual) regression web tests would prove to be a great time saver and rescue our testers from the repetitive boredom that is inevitably part and parcel of such a task. You can only run that historical trends report so many times before you nod off in your chair...
This was my wish list:
- Must be usable by those who are not necessarily developers. Many testers do not come from a formal geeky coding background :-)
- Able to record browser sessions. Once the test recording is complete, the ability to modify the various steps is needed
- Test case reusability - e.g. set up the user logon process once and reuse across tests
- Must be browser DOM based (and give me easy access to it please)
- Have a usable 'learnable by exploration' IDE
- Allow me to add/drop into custom code should I need to - work at various levels of abstraction
- Support what we now class as standard web app features such as javascript usage, pop-ups, AJAX, DHTML bla bla.
- Ideally integrate with Visual Studio 2008
- Multi-browser support, although admittedly a lot of the Intranet front-ends I work on are Corporate IE6+ standardised ones (I know slapped hands...)
There are plenty out there and not being a dedicated tester/QA guru I do not pretend to have done an exhaustive search!
Initial thoughts - VSTS TE
We had a license for Visual Studio Team System Test Edition (VSTS TE), so I thought it made sense to start with that. I set up a VM, installed VSTS TE and recorded my first web test:
It nicely recorded the http requests, preserved session state etc (bit like the old ACT), and the steps that it went through to complete the test were all clearly shown:
but after some experimenting for an hour or so, I stopped. Why? Well, the whole set up is clearly aimed at techies which is not necessarily a bad thing, but I really couldn't see any of our QA testers using this in a hurry. Shame - an undoubtedly powerful tool for developer level testers but it felt more like a testing framework than an integrated Web UI testing tool. It's all there (see web test overview on MSDN), but it needs some sort of abstraction and nice UI over the top of it.
Back to the drawing board then...but don't stop reading yet... after some digging I found what I think could be a solution and one that plugs nicely into the VSTS I had already installed, namely ArtOfTest's WebAii Automation Design Canvas (built on top of its free WebAii .Net Automation Framework).
WebAii Automation Design Canvas
Lets cut to the chase. Watch the following 4 minute WebAii demo which shows how you construct a simple web recorded test:
A simple example I know but this is more like it. I particularly like the element highlighting and the ability to do 'quick verifications' on those elements during the recording. You can also be as specific as you wish, having full access to the browser DOM tree:
and from there you can examine/compare individual attributes using the verification builder:
this gives you pretty good control. For example, to compare an entire HTML table of results use (notice it is already populated with the relevant markup from the recorded web page):
I like the way the cancel and OK buttons are reversed in the above too, nice anti-usability touch ;-)
Another useful feature is the storyboard view which shows small screen shots from each page taken during the web recording:
This makes a great (albeit simple) documentation tool for each step. The screen shots also show which control is being actioned upon.
Other nice features
Here's a run down of some of the other bits I have tried so far:
- You can play back your tests against IE or Firefox.
- You can do simple data driven tests, for example, run the same test multiple times, but use different usernames/passwords for the logon on each iteration.
- supports AJAX and Silverlight but not tried this in anger yet.
- You can call other tests (e.g. a logon test) as a single test step.
- You can convert a step to .Net code (a one way thing). This is a nice feature and allows you to use recording to capture the main outline of the test and then drill down into various tests by using the "convert step to code" option and hand crank any .Net code you need.
- Web elements can be identified in various ways, including by attribute matching and XPath queries. This is useful for picking up any dynamically 'added on the fly' controls.
System requirements and cost
You need Visual Studio Team Edition 2008 (or Pro edition).
The WebAii Automation Design Canvas cost itself is around £1700 ($2499) per machine (see price list). I think this is pretty good value for a machine license as you can then share usage across different users. VMs and VPC images are also ok to use. A common sense licensing model from ArtOfTest, nice.
What next
Next step for me is to trial this on a project before the trial license expires to ensure it covers all that we need. I want to integrate it into an existing project, TFS it up and use it in anger with some of our main web UI regression tests we already have in place. I'll bring in the QA guys too, to assess its usage and usability from a less technical perspective.
Clarkey
