Testing GHC, the Glasgow Haskell Compiler

This is part of a series of articles I started with Quality Assurance and Automated Testing in Open Source Software.

GHC - The Glasgow Haskell Compiler is probably the strongest compiler and interactive environment for the Haskell functional language .

Development

The GHC team uses Trac to manage their project. The main developer page provides a lot of information on how to get involved.
They are using Darcs for version control, which itself is written in Haskell.

To get the source look at the Getting The Sources page. It has the detailed instructions but there are many of them and each one takes a long time. So be patient!

Testing

Writing and running tests on the GHC Test framework is documented rather well.
Tests are divided into separate files. The tests themselves are written in Haskell. For each such test file (where the code lives) you have to provide three additional files:

  • a file containing what should be placed on STDIN
  • one containing the expected output on STDOUT
  • and another one containing the expected output on STDERR

The test framework is written in Python. For each test file the author has to give an expected behavior which (such as normal, skip, expect_broken(#trac bug number), expect_broken_if_platform(bug_number, platform), etc... These are all Python functions provided by the framework.

Smoke Testing

The GHC team uses Buildbot for automatic smoke testing.
In order to setup a build slave interested people need to get a username/password - which I got in a few hours.
Setting up client (or slave) part of BuildBot is rather simple. There are some prerequisites for setting it up and building GHC, but they are all nicely described on the GHC BuildBot page.

So far I managed to setup the BuildBot slave but it seems the build did not start in the scheduled time. Once the buildbot is running I'll be able to see the results on the status page.

TODO

I still would like to see how test are running, maybe write a simple one and I would like to get my buildbot slave do its job.