Parrot is a virtual machine
for dynamic languages such as PHP, Perl 5, Python, Ruby, Scheme, Tcl with its
main focus on Perl 6.
As one day it is going to replace the engine of Perl 5 it must run on the
50-so platforms where Perl 5 currently runs. It should be also tested on
those platforms. Therefore having and easy way to setup a smoke test
environment and to report the test results should be easy.
Development
Parrot is written in C.
For version control the developers use Subversion. You can find instructions
on how to get the source code in the download
area of their web site. Links to other development and testing related pages can be found under
resources.
Testing
At the time of this writing there were 7380 unit tests written for Parrot
and 3118 unit tests for the language implementations.
As they don't have an automatic way to report test coverage, the team
provides a document describing the level of coverage for each subsystem
on the
Parrot Testing Status
page.
Similar to the standard in Perl, tests can be found in the
t/ directory in the source code.
Tests are written on several levels and in several languages: There are tests written
in C, PIR (Parrot Intermediate Representation), PASM (Parrot Assembler) and Perl5.
In addition, test for the various language implementations, that is for the compilers
of language X to Parrot Assembler, are usually written in their respective
language and can usually be found in their own directories under
languages/
The output of all the tests follows the
TAP - Test Anything Protocol used in
Perl 5 and in various other languages.
Smoke testing
It is extreamly easy to get involved in smoke testing Parrot.
You will only need a C compiler and a recent version of Perl5 installed
along with Test::TAP::HTMLMatrix with its prerequisites from CPAN.
Results of the Parrot smoke tests
can be found following the above link. The reports are automatically posted
there, when you run make smoke.
The exact instructions to run and submit smoke test reports are these:
perl Configure.pl
make
make smoke
Obviously one has to check out the latest version from SVN first by
svn co http://svn.perl.org/parrot/trunk parrot
and then update to the latest version every time before running the tests.
My scheduled job looks like this:
cd /home/gabor/work/parrot
svn up
make clean
perl Configure.pl --cc=cc --cxx=CC --link=cc --ld=cc
make
make smoke
make languages-smoke
the implementation of the Smoke client and server can be found in
tools/util/ smokeserv-server.pl and smokeserv-client.pl and
smokeserv-README.pod
On the smoke report page one can see a table divided based on platform.
Within each platform you can see one row for each report. This is a
summary of the results.
A sample row looks like this:
Parrot 0.4.14 r20749 20 Aug 2007 20:03 Mon languages 11.40 min
99.23 % ok 3118: 3094, 24, 123, 29, 1 >> >>
The columns are:
Parrot version number (the latest release before the test run) (0.4.14)
revision number (from the Subversion revision count) (r20749)
Date when the test was executed (20 Aug 2007 20:03 Mon)
The remaining two >> signs lead us to the more detailed reports
Potential flags can bee looked up in the parrot source code by typing perldoc Configure.pl.
Specifically languages means that we are testing some of the language implementations hence
the total number of test is only 3118.
In the details, the meaning of Success and Failed are obvious. TODO means test that were implemented already
before the actual feature was implemented. So we expect those tests to fail. Unexpected success means that there were tests marked as TODO (that is expected to fail)
that succeeded. This can happen in various cases, for example when someone implements a feature
but forgets to update the test.
Tests marked as skipped are those that might be platform dependent (and the test is running
on a platform where they are not relevant) or that they have some missing prerequisites.
In the parrot test report
on the left side you can see filename. The test suit is organized into several files.
Each file contains one or more unite tests (or assertions, or ok calls). Each rectangular
next to the file name represents one such unit test. The size of the rectangulars does not
matter - they are different only to fit the screen. What is interesting is the number of
rectangulars. On one hand in scheme/t/io/basict.t (4th row) you can see 2 rectangulars meaning there
are two assertions in the file on the other hand scheme/t/arith/logic.t has many small rectangulars
meaning it has many (I counted 42) units.
The colors mean
Green: ok
Dark green: TODO test that failed as expected
Light green: SKIPed tests
Red: failure
Yellow: TODO test that was unexpectedly successful
Placing the mouse over one of the rectangulars, you'll see the tool-tip showing the exact name of
the assertion that usually explains what does that test. In case of the TODO and SKIPed tests
it usually also says why is that in TODO or why was that skipped. (Obviously this will only
work on the real report and not on this image).
On the right side of each row you can see a small summary of that file,
the percentage of the tests that were successful.
Wish list
What I am missing form this is a report where I could see each assertion on which platform
was it successful and where did it fail. I would also like to see some statistics on how
the number of test and the success/failure rate changed throughout the development.
I would also like to see an aggregated report from all the platforms.
