Canoo WebTest White Paper
Testing is an important part of any serious development effort.
For web applications it is crucial.Defects in your corporate website may be only cause of irritation once at but
they can cost you real money at other times, they can lower your
market value and may even put you out of business.Canoo WebTest helps you to reduce the defect rate of your web application.
What our customers care about
- Quality: Quality improvements are hard to achieve if you cannot see the effects of your measures. Canoo WebTest measures the
externally observable quality of your application.
- Development Risk: Is the development team on track? What progress did it achieve? What does it mean, if they say that 80% is working? Is it really?
Canoo WebTest reports the real progress in terms of running Use Cases.
- Operations Risk: Can we put our application into production safely? Will it work? Will it not do any harm when running?
Canoo WebTest tells you whether it will work.
- Delivery: Did the development team really deliver everything they promised? Canoo WebTest tells you what was delivered and whether it works as expected.
- Costs: The costs for testing must not exceed its benefits. Canoo WebTest is free of charge, tests are easy and quick to write. They can
be run countless times unsupervised and automatically. In fact it is cheaper and faster than testing manually.
What programmers care about
As programmers we want to be sure that our web application works as
expected. We want to validate our work. We need some backing so that we
can boldly say: "Yes, we have done
it correctly. Yes, it works. Yes, we are finished with this.
No, we have not broken any old functionality."
If we apply the full set of tests to the system every day then it is
easy to find the cause of any reported defect,
because it must be something we checked in yesterday.
If testing finds a defect, we want to solve it quickly.
Therefore, we need to reproduce the unexpected behavior. What were the
steps that led to this error? What was the sequence? What were the
intermediate results? How much easier would it be to track down the
error if we only had this information!
No matter how hard we try, there will always be defects that slip
through our testing. They get reported by our users. We want to make
sure that their feedback does not get lost, that the defect really gets solved,
that it never appears again in future releases. The best solution
is to write an automated test that exposes the bug. It will fail
as long as the bug is unsolved. It will stay forever in our suite of tests.
We have to read a lot of documentation every day.
Bad experience made us suspicious about the correctness of any external
documentation. We don't really like writing documentation ourselves because
we know that it is only a matter of time until it is out of sync with the
system and all our effort will be wasted. If the documentation is done
via automated tests, it is assured to be up to date, making it a reliable
source of information. We are much more motivated to invest our
time for this.
The same holds true for requirements specifications. It would be really
convenient if we could automatically prove that we comply with the
requirements spec. Therefore the spec needs to be formal enough to
allow automated compliance tests. It must still be easy to understand
so that the customer, the requirements analyst and
the development team can all easily understand the spec.
The specification language needs to be flexible enough to express
page contents, workflow and navigational structures.
You may claim that all the above would be really helpful but
impossible to implement under the constraints of real projects.
We have done it ourselves
and we have helped others doing it. The effect is tremendous: to
the quality of the system, to the satisfaction of the customer and
to the motivation and self-esteem of the development crew.
Testing is not for free, but it pays off.
How Canoo WebTest works
Canoo WebTest lets you specify test steps like
- get the login page
- validate the page title to be
scott in the username text field
tiger in the password field
- hit the ok button
- validate the page title to be
The example steps above make up a sequence of steps that only make
sense if executed in exactly this order and within one user session. We call
use case or a
Canoo WebTest offers the appropriate abstraction
for this. Refer to the
for a complete list of step types.
Converting the textual description into a
Canoo WebTest is easy, as you see
below. Note how close it is to the textual description.
This is XML and you will get all the support from your preferred XML editor, including
syntax highlighting and code completion based on the WebTest.dtd.
Canoo WebTest leverages the advantages of
XML even further. You may have noticed the line
&config;. This is an
XML entity that refers to the content of a file.
The XML parser inlines the file at test execution time. It is one of the possible ways to share common settings for all test steps. Here
the settings for protocol, host, port and webapp name are shared.
If you are familiar with the ANT build automation tool you will have recognized that Canoo WebTest makes use of this.
Canoo WebTest exploits ANT's ability to structure a "build" into modules that can either be called
separately or as a whole. That way, you can run any WebTest in isolation. You can also
group tests into a testsuite that again can be part of a bigger testsuite. In the end
you have a tree of testsuites, where each node and subtree can be executed.
The execution of the several test steps is currently implemented by using the HtmlUnit API, again an Open Source package. Test results are reported in either plain text
or in XML format for later presentation via XSLT. Standard reporting XSLT stylesheets come with the Canoo WebTest distribution. They can easily be adapted to your corporate style and reporting
A sidebar: Do you think that the above example is so easy that
you do not need an automatic test for this? Consider the following variations:
- Bookmark: What if I try to get the Home Page directly without login?
- Other pages: We have to test that no page is shown without proper login and that we get the requested page after proper login.
- Bad Login: Bad login should keep us on the login page.
This is quite a number of scenarios to be tested.
Now imagine a manual tester checking all this. Very soon he will
get bored and unobservant, not to mention that resetting his session
for every single test requires a lot of work. Is he
checking again all the possible variations at every full test?
Test automation is key to better quality. Manual checks are more flexible and less
expensive to do
one time. They are more expensive and less reliable when tests
need to be done over and over again. We advise to do manual checks for everything that
cannot break after it worked once. Everything else should be automated if the automation
can be done without excessive costs. We feel that testing with
Canoo WebTest reaches the
break-even point for 90% of our tests after a few test runs at latest.
We want to use what we already know. We don't want to learn a new language
for the test automation. We want to rely on standard formats.
Functional testing can be classified as being either data driven or record/replay.
Canoo WebTest follows the data driven approach. Record/replay is appealing at first, because you can
create a lot of tests in a short time. A proxy logs what pages you request and stores
the results. It can then replay the requests and compare the results against the stored
ones. You typically have to tweak this procedure to tell the program what parts of the
page are expected to change. The actual date and time are the most obvious examples.
Every small change to your webapp causes a lot of these tests to fail. These failures
must be manually processed to separate the "real" failures from the "false negatives".
Doing this is almost as tedious and error prone as the manual testing and is therefore
discouraged e.g. by the Automated Testing Specialists group.
Any automated test should fit snugly into your build process. If you are already using ANT for your build automation, it is no effort to integrate
Canoo WebTest. An Example of this is Canoo WebTest itself. It contains a selftest that is written with
Canoo WebTest. Every new build of Canoo WebTest triggers
that selftest. You can explore this behavior online, starting at the Build Info
link of the Canoo WebTest distribution page. Note that this is very convenient for
nightly builds and even for use with a continuous integration platform
If your build process is not ANT based, calling Canoo WebTest is still easy.
It means starting a Java Application. This can easily be done with every build script
language that we know.
"Regression tests" is the concept of testing that asserts that everything that
worked yesterday still works today. To achieve this, our tests must not be dependent
on random data. Also, the expected result must be clear in advance as opposed to the
"guru checks output" approach, where a specialist validates changing results.
Tests must give a thumps up indication when successful
and a detailed error indication otherwise. Well, this is pretty much like compiler
Functional tests do not replace unit tests. They work together hand-in-hand.
Consider the following example: Your
Webapp displays an html table that is filled with data from the database. The
maximum number of rows should be 20 and if there is more data available, a link
should be shown that points to the page that contains the next 20 entries.
If there is no data, no table should be shown, but the message "sorry, no data".
We would test this with a) no data b) one row c) 5 rows d) exactly 20 rows
e) 21 rows f) 40 rows g) 41 rows. A naive way of testing this would be to manipulate the database
(maybe by using an administration servlet that we can call via "invoke") prior
to calling the page. But this is not only very slow but also a little dangerous.
What if two tests run concurrently against the same test database? They
will mutually destroy their test setup. What if the test run breaks? Is the
state of the test database rolled back? The whole job is difficult to do for a
functional test, but easy and quick for a unit test. A unit test can easily
call the table rendering and assert the proper "paging" without even having a
database! What is left for the functional test is to assert that the table rendering logic
was called at all.
Canoo WebTest is an Open Source Java project and totally based on Open Source
packages. If you are not satisfied with any of the functionality, you can adapt it
to your requirements. Having the sources, you even gain the ability to start the test in the
debugger, revealing everything that goes on.
Canoo WebTest is free of charge. The downside is, that there is no guaranteed
support. However, you can ask Canoo for special support incidents, a support contract
and on-site help for introducing automated testing in your project.
Canoo WebTest is not restricted to any special technology on the server side.
It makes no difference if you use Servlets, JSP, ASP, CGI, PHP or whatever as long
your browser does. In fact, it uses HtmlUnit for this purpose, which can
be seen as a faceless browser that is able to mimic the behaviour of e.g.
MSIE or Firefox.
Browser dependencies are the menace of web programming. One possibility is
to check manually against all the "supported browsers". Our approach is to
validate our html to comply with the specification. A full and pedantic validation
is outside the scope of Canoo WebTest, but every validation step calls the
Neko Html parser (part of HtmlUnit) and will warn you on improper html.
That has proven to be very
helpful. If your manual tests reveal that certain html constructions produce
different behavior in your supported browers (like empty table cells in
IE and Netscape), you can set up a test that checks against the usage
of these constructs.