The idea is that in some cases (or would it be most cases?)
it is useful to separate the test execution and raw data
generation from the test management and reporting.
When using TAP you have a test script or program or whatever
you would like to call it that exercises your application
and print the results to the screen (called TAP producer).
You also have another tool that can parse this output
(called TAP parser or harness) and generate nice or at least
useful reports.
TAP output examples
A successful one looks like this
1..3
ok 1 - stderr is empty
ok 2
ok 3 - foo was added
A failed run looks like this:
1..3
ok 1 - stderr is empty
not ok 2
# Failed test at t/add_foo_to_db line 49.
# In row 0
# Expected: A1=569.1
# Received: A1=-1.23872428244378e-263
ok 3 - foo was added
# Looks like you failed 1 test of 3.
TAP details
The output basically consists of three parts.
A header where you declare the planned number of test units:
1..42
Then for each test unit (also called assertion in some languages)
you print
ok N - name of unit
if the assertion was successful or
not ok N - name of unit
if it failed.
N is a counter that in our case should go from 1 to 42 as
declared in the header and "name of unit" is some
arbitrary and optional text to describe the unit.
Sprinkled throughout this output there can be comments staring
with a # sign.
# comments
These lines are used both as comments and as a way
to provide details in case of a failure.
The third part of the output is generated only when there
was failure, or something that looks like a failure.
It is a short summary that is similar to the comments but
it gives a summary of what happened. Something like this:
# Looks like you failed 2 test of 42.
So far this was a simplified description on how TAP looks like.
Why is this separation good ?
For this to work we need two entities. TAP producers that
can generate TAP output and TAP consumers that can read
this output and provide nice reports. Actually we might need
three parts as we also need a tool called harness to run the
test scripts. In most current implementations the consumer
and the harness are in one tool.
The point is that nothing in the above said anything about
the language the TAP producer was written in nor does it say
anything about the language of the TAP parser. They can be
in any language. In fact there are TAP producers implemented
in several languages including Perl, Python, PHP, Javascript,
C and even PgSQL with some more listed on
testanything.org
TAP Producers
This will be especially important for application where several
technologies or programming languages are involved.
If you think about a web application it will require HTML, CSS,
Javascript, SQL and at least one high-level server side language
such as Perl, Python, Ruby or PHP. In some cases even more.
If you think about some networking device. Who said that the
self-tests of the device running on the device cannot generate
TAP?
Other system will have other complexities.
Actually producing TAP output is not too difficult as you only need
to print some simple strings. Nevertheless it is nice to wrap it in
some testing library so others can reuse it.
TAP Consumers
The default TAP consumer in Perl called Test::Harness is actually
a combination of both a harness tool that can run the test scripts
and that of a parser that can analyze the TAP output and generate
textual report.
In our case this summary is actually longer than the TAP output
and looks like this:
t/add_foo_to_db....1/3
# Failed test at t/add_foo_to_db line 49.
# In row 0
# Expected: A1=569.1
# Received: A1=-1.23872428244378e-263
# Looks like you failed 1 test of 3.
t/add_foo_to_db.... Dubious, test returned 1 (wstat 256, 0x100)
Failed 1/3 subtests
Test Summary Report
-------------------
t/add_foo_to_db (Wstat: 256 Tests: 3 Failed: 1)
Failed test: 2
Non-zero exit status: 1
Files=1, Tests=3, 0 wallclock secs
( 0.01 usr 0.00 sys + 0.11 cusr 0.00 csys = 0.12 CPU)
Result: FAIL
In a real situation though, you would have tens or hundreds of
test scripts each with tens or hundreds of assertions. In such
case the TAP output would be thousands of lines long while the
summary created by the harness does not get much longer.
Test::Harness has a command line counterpart called
prove that does similar things.
The nice thing of prove is that you can actually separate the
two steps as well. You can run run prove with the -a option
and archive your TAP output for later use. That is, prove is
now only used to run the tests it does not actually
parse the output.
You can take the output, archive it for later processing.
Send it over the network to store it in a remote database
and/or generate other types of reports, more pleasing to the eyes.
Especially to the eyes of management.
to collect your TAP output and provide historical reports
To see Smolder in action visit this
public Smolder server
I guess as TAP will become even more widespread TAP parsers will be
written in other languages as well, not only in Perl.
RFC #### or TAP as an IETF standard
Recently the IETF together with
some of the core TAP developers have started a process
to turn TAP into an RFC.
This can be a big help in getting cross platform and cross
language test running and reporting done.
For details check out the
TAP_at_IETF:_Notes
Your input will be appreciated there!
Testing Automation Planet
Some time ago I setup an content aggregator planet
for test automation blogs. Appropriately I called
it TAP, though it stands for
Testing Automation Planet
and not for Testing Anything Planet.
At least today.
Anyway, you are welcome to read it and suggest other
blogs I should include.
TIP - Testing in Python
Recently I have stumbled upon a great mailing list called
TIP - Testing in Python.
While obviously it is mostly concerned about Python related things
there are discussions that are interesting even if
you are not a Python programmer.
