This entry was first sent out as part of the Test Automation Tips. Visit here to subscribe.
After the summer vacation, starting with the new school-year I am resuming the Test Automation Tips.
Instead of a set of examples this time I'll give an overview of TAP, the Test Anything Protocol.
TAP - Test Anything Protocol
The idea is that in some cases (or would it be most cases?) it is useful to separate the test execution and raw data generation from the test management and reporting.
When using TAP you have a test script or program or whatever you would like to call it that exercises your application and print the results to the screen (called TAP producer). You also have another tool that can parse this output (called TAP parser or harness) and generate nice or at least useful reports.
TAP output examples
A successful one looks like this
1..3 ok 1 - stderr is empty ok 2 ok 3 - foo was added
A failed run looks like this:
1..3 ok 1 - stderr is empty not ok 2 # Failed test at t/add_foo_to_db line 49. # In row 0 # Expected: A1=569.1 # Received: A1=-1.23872428244378e-263 ok 3 - foo was added # Looks like you failed 1 test of 3.
The output basically consists of three parts. A header where you declare the planned number of test units:
Then for each test unit (also called assertion in some languages) you print
ok N - name of unit
if the assertion was successful or
not ok N - name of unit
if it failed.
N is a counter that in our case should go from 1 to 42 as declared in the header and "name of unit" is some arbitrary and optional text to describe the unit.
Sprinkled throughout this output there can be comments staring with a # sign.
These lines are used both as comments and as a way to provide details in case of a failure.
The third part of the output is generated only when there was failure, or something that looks like a failure. It is a short summary that is similar to the comments but it gives a summary of what happened. Something like this:
# Looks like you failed 2 test of 42.
So far this was a simplified description on how TAP looks like.
Why is this separation good ?
For this to work we need two entities. TAP producers that can generate TAP output and TAP consumers that can read this output and provide nice reports. Actually we might need three parts as we also need a tool called harness to run the test scripts. In most current implementations the consumer and the harness are in one tool.
This will be especially important for application where several technologies or programming languages are involved.
If you think about some networking device. Who said that the self-tests of the device running on the device cannot generate TAP?
Other system will have other complexities.
Actually producing TAP output is not too difficult as you only need to print some simple strings. Nevertheless it is nice to wrap it in some testing library so others can reuse it.
The default TAP consumer in Perl called Test::Harness is actually a combination of both a harness tool that can run the test scripts and that of a parser that can analyze the TAP output and generate textual report.
In our case this summary is actually longer than the TAP output and looks like this:
t/add_foo_to_db....1/3 # Failed test at t/add_foo_to_db line 49. # In row 0 # Expected: A1=569.1 # Received: A1=-1.23872428244378e-263 # Looks like you failed 1 test of 3. t/add_foo_to_db.... Dubious, test returned 1 (wstat 256, 0x100) Failed 1/3 subtests Test Summary Report ------------------- t/add_foo_to_db (Wstat: 256 Tests: 3 Failed: 1) Failed test: 2 Non-zero exit status: 1 Files=1, Tests=3, 0 wallclock secs ( 0.01 usr 0.00 sys + 0.11 cusr 0.00 csys = 0.12 CPU) Result: FAIL
In a real situation though, you would have tens or hundreds of test scripts each with tens or hundreds of assertions. In such case the TAP output would be thousands of lines long while the summary created by the harness does not get much longer.
Test::Harness has a command line counterpart called prove that does similar things.
The nice thing of prove is that you can actually separate the two steps as well. You can run run prove with the -a option and archive your TAP output for later use. That is, prove is now only used to run the tests it does not actually parse the output.
You can take the output, archive it for later processing. Send it over the network to store it in a remote database and/or generate other types of reports, more pleasing to the eyes. Especially to the eyes of management.
TAP as HTML
So you could use TAP::Formatter::HTML of Steve Purkis
to generate reports like this one
You can also use Smolder of Michael Peters.
to collect your TAP output and provide historical reports To see Smolder in action visit this public Smolder server
I guess as TAP will become even more widespread TAP parsers will be written in other languages as well, not only in Perl.
RFC #### or TAP as an IETF standard
Recently the IETF together with some of the core TAP developers have started a process to turn TAP into an RFC.
This can be a big help in getting cross platform and cross language test running and reporting done. For details check out the TAP_at_IETF:_Notes
Your input will be appreciated there!
Testing Automation Planet
Some time ago I setup an content aggregator planet
for test automation blogs. Appropriately I called
it TAP, though it stands for
Testing Automation Planet
and not for Testing Anything Planet.
Anyway, you are welcome to read it and suggest other blogs I should include.
TIP - Testing in Python
Recently I have stumbled upon a great mailing list called TIP - Testing in Python. While obviously it is mostly concerned about Python related things there are discussions that are interesting even if you are not a Python programmer.
There was a very successful QA Hackathon in Oslo in April 2008 where among other things the IETF process has started.
There are going to be at least two similar events in 2009.
One, the Israeli QA Hackathon is planned to be in February in Israel focusing on TAP in a cross language setting.
The other QA Hackathon is planned to be in April in Birmingham, UK which looks like more focused on the Perl aspects.Published on 2008-09-02 by Gabor Szabo blog comments powered by Disqus