Welcome back

After the summer vacation, starting with the new school-year I am resuming the Test Automation Tips.

Instead of a set of examples this time I'll give an overview of TAP, the Test Anything Protocol.

TAP - Test Anything Protocol

The idea is that in some cases (or would it be most cases?) it is useful to separate the test execution and raw data generation from the test management and reporting.

When using TAP you have a test script or program or whatever you would like to call it that exercises your application and print the results to the screen (called TAP producer). You also have another tool that can parse this output (called TAP parser or harness) and generate nice or at least useful reports.

TAP output examples

A successful one looks like this

  1..3
  ok 1 - stderr is empty
  ok 2
  ok 3 - foo was added

A failed run looks like this:

  1..3
  ok 1 - stderr is empty
  not ok 2
  #   Failed test at t/add_foo_to_db line 49.
  # In row 0
  # Expected: A1=569.1
  # Received: A1=-1.23872428244378e-263
  ok 3 - foo was added
  # Looks like you failed 1 test of 3.

TAP details

The output basically consists of three parts. A header where you declare the planned number of test units:

  1..42

Then for each test unit (also called assertion in some languages) you print

   ok N - name of unit

if the assertion was successful or

   not ok N - name of unit

if it failed.

N is a counter that in our case should go from 1 to 42 as declared in the header and "name of unit" is some arbitrary and optional text to describe the unit.

Sprinkled throughout this output there can be comments staring with a # sign.

   # comments

These lines are used both as comments and as a way to provide details in case of a failure.

The third part of the output is generated only when there was failure, or something that looks like a failure. It is a short summary that is similar to the comments but it gives a summary of what happened. Something like this:

  # Looks like you failed 2 test of 42.

So far this was a simplified description on how TAP looks like.

Why is this separation good ?

For this to work we need two entities. TAP producers that can generate TAP output and TAP consumers that can read this output and provide nice reports. Actually we might need three parts as we also need a tool called harness to run the test scripts. In most current implementations the consumer and the harness are in one tool.

The point is that nothing in the above said anything about the language the TAP producer was written in nor does it say anything about the language of the TAP parser. They can be in any language. In fact there are TAP producers implemented in several languages including Perl, Python, PHP, Javascript, C and even PgSQL with some more listed on testanything.org

TAP Producers

This will be especially important for application where several technologies or programming languages are involved.

If you think about a web application it will require HTML, CSS, Javascript, SQL and at least one high-level server side language such as Perl, Python, Ruby or PHP. In some cases even more.

If you think about some networking device. Who said that the self-tests of the device running on the device cannot generate TAP?

Other system will have other complexities.

Actually producing TAP output is not too difficult as you only need to print some simple strings. Nevertheless it is nice to wrap it in some testing library so others can reuse it.

TAP Consumers

The default TAP consumer in Perl called Test::Harness is actually a combination of both a harness tool that can run the test scripts and that of a parser that can analyze the TAP output and generate textual report.

In our case this summary is actually longer than the TAP output and looks like this:

   t/add_foo_to_db....1/3
   #   Failed test at t/add_foo_to_db line 49.
   # In row 0
   # Expected: A1=569.1
   # Received: A1=-1.23872428244378e-263
   # Looks like you failed 1 test of 3.
   t/add_foo_to_db.... Dubious, test returned 1 (wstat 256, 0x100)
     Failed 1/3 subtests

   Test Summary Report
   -------------------
   t/add_foo_to_db (Wstat: 256 Tests: 3 Failed: 1)
     Failed test:  2
     Non-zero exit status: 1
   Files=1, Tests=3,  0 wallclock secs
   ( 0.01 usr  0.00 sys +  0.11 cusr  0.00 csys =  0.12 CPU)
   Result: FAIL

In a real situation though, you would have tens or hundreds of test scripts each with tens or hundreds of assertions. In such case the TAP output would be thousands of lines long while the summary created by the harness does not get much longer.

Test::Harness has a command line counterpart called prove that does similar things.

The nice thing of prove is that you can actually separate the two steps as well. You can run run prove with the -a option and archive your TAP output for later use. That is, prove is now only used to run the tests it does not actually parse the output.

You can take the output, archive it for later processing. Send it over the network to store it in a remote database and/or generate other types of reports, more pleasing to the eyes. Especially to the eyes of management.

TAP as HTML

So you could use TAP::Formatter::HTML of Steve Purkis

to generate reports like this one

Smolder

You can also use Smolder of Michael Peters.

to collect your TAP output and provide historical reports To see Smolder in action visit this public Smolder server

I guess as TAP will become even more widespread TAP parsers will be written in other languages as well, not only in Perl.

RFC #### or TAP as an IETF standard

Recently the IETF together with some of the core TAP developers have started a process to turn TAP into an RFC.

This can be a big help in getting cross platform and cross language test running and reporting done. For details check out the TAP_at_IETF:_Notes

Your input will be appreciated there!

Testing Automation Planet

Some time ago I setup an content aggregator planet for test automation blogs. Appropriately I called it TAP, though it stands for Testing Automation Planet and not for Testing Anything Planet.
At least today.

Anyway, you are welcome to read it and suggest other blogs I should include.

TIP - Testing in Python

Recently I have stumbled upon a great mailing list called TIP - Testing in Python. While obviously it is mostly concerned about Python related things there are discussions that are interesting even if you are not a Python programmer.

QA Hackathon

There was a very successful QA Hackathon in Oslo in April 2008 where among other things the IETF process has started.

There are going to be at least two similar events in 2009.

One, the Israeli QA Hackathon is planned to be in February in Israel focusing on TAP in a cross language setting.

The other QA Hackathon is planned to be in April in Birmingham, UK which looks like more focused on the Perl aspects.