Reliable Web Testing

WebTest is a free Open Source tool for automated testing of web applications in a very effective way:

  • It's simple
  • It's fast
  • It provides great summaries
  • It runs anywhere
  • It's easy to extend

Excellent Reporting!

Paul King: "WebTest's excellent reporting provides us with a great way to get feedback directly from the customers about whether the test is doing what is intended".

Save time and resources!

WebTest has an easy syntax with steps having meaningful names that can be understood even when you don't know WebTest. Additionally, the recorder allows you to quickly generate a first draft of your tests that you can adapt and refactor to build robust tests suites. Moreover, WebTest doesn't download CSS or images and doesn't need to compute page rendering.

Very low TCO

Some user testimonials say it in a nutshell: "The ROI on WebTest is many orders of magnitude higher than any other tool I've used", "Support = excellent and free", "It simply delivers", "It has been remarkably robust, mature, and bug-free". WebTest is plain Java and runs everywhere as long as you have a JDK for your operating system.

Quickly understand the failure cause!

Perhaps the most important feature. JUnit like reports are not enough to understand why a test fails and WebTest reports provide all the information you need to quickly understand the cause of failure.

Simply create your own steps!

No matter which tool you use, you will inevitably reach its limits and need to extend it for your custom needs. With WebTest you can simply create your own steps without technology mix and access the whole API of HtmlUnit, commons-httpclient, etc.

Straightforward integration!

WebTest scripts ARE Ant scripts. This provides that you with a direct and easy integration with continuous integration tools such as Cruise Control, support in your favorite IDE, etc.

Canoo WebTest White Paper

Testing is an important part of any serious development effort. For web applications it is crucial.Defects in your corporate website may be only cause of irritation once at but they can cost you real money at other times, they can lower your market value and may even put you out of business.Canoo WebTest helps you to reduce the defect rate of your web application.

What our customers care about

  • Quality: Quality improvements are hard to achieve if you cannot see the effects of your measures. Canoo WebTest measures the externally observable quality of your application.

  • Development Risk: Is the development team on track? What progress did it achieve? What does it mean, if they say that 80% is working? Is it really? Canoo WebTest reports the real progress in terms of running Use Cases.

  • Operations Risk: Can we put our application into production safely? Will it work? Will it not do any harm when running? Canoo WebTest tells you whether it will work.

  • Delivery: Did the development team really deliver everything they promised? Canoo WebTest tells you what was delivered and whether it works as expected.

  • Costs: The costs for testing must not exceed its benefits. Canoo WebTest is free of charge, tests are easy and quick to write. They can be run countless times unsupervised and automatically. In fact it is cheaper and faster than testing manually.

What programmers care about

As programmers we want to be sure that our web application works as expected. We want to validate our work. We need some backing so that we can boldly say: "Yes, we have done it correctly. Yes, it works. Yes, we are finished with this. No, we have not broken any old functionality."

If we apply the full set of tests to the system every day then it is easy to find the cause of any reported defect, because it must be something we checked in yesterday.

If testing finds a defect, we want to solve it quickly. Therefore, we need to reproduce the unexpected behavior. What were the steps that led to this error? What was the sequence? What were the intermediate results? How much easier would it be to track down the error if we only had this information!

No matter how hard we try, there will always be defects that slip through our testing. They get reported by our users. We want to make sure that their feedback does not get lost, that the defect really gets solved, that it never appears again in future releases. The best solution is to write an automated test that exposes the bug. It will fail as long as the bug is unsolved. It will stay forever in our suite of tests.

We have to read a lot of documentation every day. Bad experience made us suspicious about the correctness of any external documentation. We don't really like writing documentation ourselves because we know that it is only a matter of time until it is out of sync with the system and all our effort will be wasted. If the documentation is done via automated tests, it is assured to be up to date, making it a reliable source of information. We are much more motivated to invest our time for this.

The same holds true for requirements specifications. It would be really convenient if we could automatically prove that we comply with the requirements spec. Therefore the spec needs to be formal enough to allow automated compliance tests. It must still be easy to understand so that the customer, the requirements analyst and the development team can all easily understand the spec. The specification language needs to be flexible enough to express page contents, workflow and navigational structures.

You may claim that all the above would be really helpful but impossible to implement under the constraints of real projects. We have done it ourselves and we have helped others doing it. The effect is tremendous: to the quality of the system, to the satisfaction of the customer and to the motivation and self-esteem of the development crew.

Testing is not for free, but it pays off.

How Canoo WebTest works

Canoo WebTest lets you specify test steps like

get the login page
validate the page title to be Login Page
fill scott in the username text field
fill tiger in the password field
hit the ok button
validate the page title to be Home Page

The example steps above make up a sequence of steps that only make sense if executed in exactly this order and within one user session. We call this a use case or a scenario. Canoo WebTest offers the appropriate abstraction for this. Refer to the Manual Overview for a complete list of step types.

Converting the textual description into a Canoo WebTest is easy, as you see below. Note how close it is to the textual description.

This is XML and you will get all the support from your preferred XML editor, including syntax highlighting and code completion based on the WebTest.dtd. Canoo WebTest leverages the advantages of XML even further. You may have noticed the line &config;. This is an XML entity that refers to the content of a file. The XML parser inlines the file at test execution time. It is one of the possible ways to share common settings for all test steps. Here the settings for protocol, host, port and webapp name are shared.

If you are familiar with the ANT build automation tool you will have recognized that Canoo WebTest makes use of this. Canoo WebTest exploits ANT's ability to structure a "build" into modules that can either be called separately or as a whole. That way, you can run any WebTest in isolation. You can also group tests into a testsuite that again can be part of a bigger testsuite. In the end you have a tree of testsuites, where each node and subtree can be executed.

The execution of the several test steps is currently implemented by using the HtmlUnit API, again an Open Source package. Test results are reported in either plain text or in XML format for later presentation via XSLT. Standard reporting XSLT stylesheets come with the Canoo WebTest distribution. They can easily be adapted to your corporate style and reporting requirements.

A sidebar: Do you think that the above example is so easy that you do not need an automatic test for this? Consider the following variations:

  • Bookmark: What if I try to get the Home Page directly without login?

  • Other pages: We have to test that no page is shown without proper login and that we get the requested page after proper login.

  • Bad Login: Bad login should keep us on the login page.

This is quite a number of scenarios to be tested. Now imagine a manual tester checking all this. Very soon he will get bored and unobservant, not to mention that resetting his session for every single test requires a lot of work. Is he really checking again all the possible variations at every full test?

Pragmatic Considerations

Test automation is key to better quality. Manual checks are more flexible and less expensive to do one time. They are more expensive and less reliable when tests need to be done over and over again. We advise to do manual checks for everything that cannot break after it worked once. Everything else should be automated if the automation can be done without excessive costs. We feel that testing with Canoo WebTest reaches the break-even point for 90% of our tests after a few test runs at latest.

We want to use what we already know. We don't want to learn a new language for the test automation. We want to rely on standard formats.

Functional testing can be classified as being either data driven or record/replay. Canoo WebTest follows the data driven approach. Record/replay is appealing at first, because you can create a lot of tests in a short time. A proxy logs what pages you request and stores the results. It can then replay the requests and compare the results against the stored ones. You typically have to tweak this procedure to tell the program what parts of the page are expected to change. The actual date and time are the most obvious examples. Every small change to your webapp causes a lot of these tests to fail. These failures must be manually processed to separate the "real" failures from the "false negatives". Doing this is almost as tedious and error prone as the manual testing and is therefore discouraged e.g. by the Automated Testing Specialists group.

Any automated test should fit snugly into your build process. If you are already using ANT for your build automation, it is no effort to integrate Canoo WebTest. An Example of this is Canoo WebTest itself. It contains a selftest that is written with Canoo WebTest. Every new build of Canoo WebTest triggers that selftest. You can explore this behavior online, starting at the Build Info link of the Canoo WebTest distribution page. Note that this is very convenient for nightly builds and even for use with a continuous integration platform like CruiseControl.

If your build process is not ANT based, calling Canoo WebTest is still easy. It means starting a Java Application. This can easily be done with every build script language that we know.

"Regression tests" is the concept of testing that asserts that everything that worked yesterday still works today. To achieve this, our tests must not be dependent on random data. Also, the expected result must be clear in advance as opposed to the "guru checks output" approach, where a specialist validates changing results. Tests must give a thumps up indication when successful and a detailed error indication otherwise. Well, this is pretty much like compiler messages.

Functional tests do not replace unit tests. They work together hand-in-hand. Consider the following example: Your Webapp displays an html table that is filled with data from the database. The maximum number of rows should be 20 and if there is more data available, a link should be shown that points to the page that contains the next 20 entries. If there is no data, no table should be shown, but the message "sorry, no data". We would test this with a) no data b) one row c) 5 rows d) exactly 20 rows e) 21 rows f) 40 rows g) 41 rows. A naive way of testing this would be to manipulate the database (maybe by using an administration servlet that we can call via "invoke") prior to calling the page. But this is not only very slow but also a little dangerous. What if two tests run concurrently against the same test database? They will mutually destroy their test setup. What if the test run breaks? Is the state of the test database rolled back? The whole job is difficult to do for a functional test, but easy and quick for a unit test. A unit test can easily call the table rendering and assert the proper "paging" without even having a database! What is left for the functional test is to assert that the table rendering logic was called at all.

Canoo WebTest is an Open Source Java project and totally based on Open Source packages. If you are not satisfied with any of the functionality, you can adapt it to your requirements. Having the sources, you even gain the ability to start the test in the debugger, revealing everything that goes on.

Canoo WebTest is free of charge. The downside is, that there is no guaranteed support. However, you can ask Canoo for special support incidents, a support contract and on-site help for introducing automated testing in your project.

Canoo WebTest is not restricted to any special technology on the server side. It makes no difference if you use Servlets, JSP, ASP, CGI, PHP or whatever as long as it produces html. Canoo WebTest executes client side JavaScript just like your browser does. In fact, it uses HtmlUnit for this purpose, which can be seen as a faceless browser that is able to mimic the behaviour of e.g. MSIE or Firefox.

Browser dependencies are the menace of web programming. One possibility is to check manually against all the "supported browsers". Our approach is to validate our html to comply with the specification. A full and pedantic validation is outside the scope of Canoo WebTest, but every validation step calls the Neko Html parser (part of HtmlUnit) and will warn you on improper html. That has proven to be very helpful. If your manual tests reveal that certain html constructions produce different behavior in your supported browers (like empty table cells in IE and Netscape), you can set up a test that checks against the usage of these constructs.