It is not enough to run tests and get OKs and not OKs, it is also
important to have a nice way to report these results. In this
entry we are going to look at some of the ways TAP streams can
be displayed in a nicer way that pure text report.
In preparation to the Test Automation classes in Oslo two
weeks from now I am going over my comments I made
last month in Frankfurt. One of the missing items was
showing how the results of the tests that are in TAP
format can be presented better.
After all a textual report such as this:
tap/01-success..........ok
tap/02-failure..........1/4
# Failed test 'this fails'
# at tap/02-failure.t line 8.
# Failed test '2+2 should be 4'
# at tap/02-failure.t line 9.
# got: '4'
# expected: '5'
# Looks like you failed 2 tests of 4.
tap/02-failure.......... Dubious, test returned 2 (wstat 512, 0x200)
Failed 2/4 subtests
tap/03-exception........1/5 # planning 5 but running only 3 - with exception
a real exception at tap/03-exception.t line 11.
# Looks like you planned 5 tests but ran 3.
that goes on for several pages is still too much and hard to read.
Besides people with less technical background won't like it.
So let's take a look at how one can improve this.
TAP the Test Anything Protocol
Just a reminder, we are talking about test results in TAP
format. TAP originated from the
Perl world but there are implementations of it in every major programming
language and even in PostgreSQL.
Turning TAP into an IETF standard is on its way.
The basic idea is that there is a separation between
the part that runs the tests and generates raw output
and the tool that creates reports from this raw output.
The raw output has an "ok" or "not ok" line for every
test unit or assertion with some additional information
between those lines.
Something like this:
1..3
ok 1 - Accessing web page
ok 2 - Title is correct
not ok 3 - Clicking on link 'release software'
Test::Harness
Traditionally in the perl world the tests are executed either when running
"make test" or "Build test" depending on the packaging system.
There is also a tool called "prove" that allows us to run some test files.
These all used the Test::Harness module to execute the tests and
then create a textual report.
People can also run the test script using the plan "perl" command and
they will see the raw, unaltered output of the test script.
Other implementations have their own tools how to run the tests.
The problem earlier was that it was quite hard to replace the formatting
options of Test::Harness.
Since the release of TAP::Harness 3 by Andy Armstrong about two
years ago the back-end of the Harness became much cleaner that allows
the implementation of various other formatting options.
The simples way to use it and to show it is by passing it as the
formatter option to "prove".
The following command will run all the .t files in the t/ subdirectory
merging the STDOUT and STDERR (-m) and running quietly (-Q).
The output is a single HTML file using the JQuery Javascript library
and a few external Javascript and CSS files.
In order to to show many parts of it I created several test script,
ran the above command on it and placed the result
on my web server.
Go ahead, check it out.
Collecting test data
As nice as that report is you don't always want to install
TAP::Formatter::HTML on every system you run your tests.
After all you might not even use Perl for generating the TAP stream.
In addition, later you might want to create additional reports
based on historical test results.
For this, we should have a way to collect the results of the test
in raw data format. Move them to a central machine and generate
the nice reports there, keeping the raw data files as well.
The archiving option of prove was created exactly for this:
prove -a tap.tar.gz t/
This will run the tests and generate a tarbal from the resulting
TAP stream along with a meta.yml file that contains some meta data
on the execution. You can take this tar.gz file and move it to another
server.
A warning though, the TAP streams of each test file is saved in
a file with the exact same name as the test file was. So if you create the
archive and the untar it in the same place you will overwrite your test
scripts with the TAP streams. Better to open it in another directory.
Generating report from the archived files
Once you have the tar.gz file on the central machine you should
be able to create the HTML report.
Unfortunately I could not find a nice way to do it but
Ash Berlin and Steve Purkis
showed me a workaround:
Yes, this will only work on Unix, maybe on windows one can replace 'cat' with
'type' but I have not tried it. In any case I hope soon there will a better
solution to this.
Smolder
There is another way to collect and display test archive which is starting
to be a more integrated solution.
Michael Peters has developed
Smolder more than a year ago but it made it to
CPAN only a week ago after the
QA Hackathon that was held
in Birmingham.
Smolder is a web based application to collect and display TAP streams.
Once installed it can be launched with the "smolder" command.
By default it provides its own web server and uses SQLite so you don't
have to worry about further configuration. Once you see it and decide
that you'd like to use it in production with many test systems reporting
to it you'll probably invest more in the installation, using Apache as the
front-end server and MySQL as the database but at the beginning you can
use the simple installation.
Once the system is setup you have a web based administrative interface
to add projects, users and associate them with each other. Every user
can then upload archived tap results as generated by the -a flag of
prove. The system then provides various views of the tests results.
You can view the details of a single test run and you can already
see some historical data as well.
Not only that, but if you are running an open source project you can ask
Michael if can still add your project to use this public server.
Personally I'd really like to see some open source project that are not
written in perl or related to the Perl community having some data there
but of course Michael is the one who know if the machine still has capacity
to add your project.
Setting up Smolder
So in order to install Smolder you'll have to configure your CPAN
shell and then type
cpan install Smolder
That should install all the dependencies.
Once you installed it, as of version 1.35 you have to do the following
to start the server:
In addition you have to change the /etc/hosts file so smolder.local
will resolve to 127.0.01
You can of course use any name with at least one . in it.
/etc/hosts needs to have the following line in it:
127.0.0.1 smolder.local
Then you can launch Smolder using the following command:
Now you can setup your test machines or instruct your developers
to upload their results after every test run. This will give you
plenty of data quickly.
