testing SCons processes

Consodoc is (at the moment) vaporware for XML publishing as PDF. It has passed unit tests, and now I'm going to test the program as the whole.

Consodoc is based on the software build system SCons. From the user perspective, he uses an usual SConstruct with additional calls to consodoc code.

Each functional test for consodoc has the following layout:

before/
after/
stdout
stderr
retcode

The folder before contains data for test run, including the file SConstruct. After running SCons, the content of the folder before should be equal to the content of the folder after. Also, files stdout, stderr and retcode contain expected output from scons: standard output, standard error and exit code, respectively.

Each test SConstruct finishes with:

Default(...)

import os
check = os.getenv('cdoc_testtest_sconscript')
if check:
  SConscript(check)

If the environment variable is set, SCons interprets additional script, which defines the target check:

node = Local('check')
env['BUILDERS']['test'] = Builder(action = run_check)
env.test(node, None)
env.Alias('check', node)

Code of run_check is straightforward. It uses the class Popen3 from the Python module popen2 to re-run SCons to build the default target. Then is compares the output with the expected data.

Ok, an individual test case is described. Now, a script to run all the test cases is required. This script is very project-dependent. In my project, it executes the following steps:

1. delete temporary folder and re-create it again,
2. unpack the test cases to the temporary folder,
3. execute the tests.

For the last step, I decided to re-use functionality of the module unittest. I had to inject test cases to the TestCase class dynamically. I thought it would be very easy, but due to lack of closures in Python (more precisely, closures do exist, but they are counter-intuitive for lisp programmers), I failed. Finally, I found a way using a global variable. I exploit the fact that unittest uses cmp to arrange test cases. Therefore, if we have an array of test names and sequential number of running the common function, we know the name of the current test. Here is a sample code:

import sys, unittest
test_names = ['aaa', 'ddd', 'bbb']
test_names.sort()
test_index = 0

class DynamicTestCase(unittest.TestCase):
  def one_test(self):
    global test_index
    test_name  = test_names[test_index]
    test_index = test_index + 1
    # Ok for 'aaa' and 'bbb', failure for 'ddd'
    assert test_name in ['aaa', 'bbb']

for test_name in test_names:
  func_name = 'test_' + test_name
  setattr(DynamicTestCase, func_name, DynamicTestCase.one_test)

suite = unittest.makeSuite(DynamicTestCase, 'test_')
if not unittest.TextTestRunner().run(suite).wasSuccessful():
  sys.exit(1)

Update

I've overlooked a simple way to make the right closures. Peter Otten suggested a better way:

import sys
import unittest

test_names = ['aaa', 'ddd', 'bbb']
test_names.sort()

class DynamicTestCase(unittest.TestCase):
    pass

def make_test(test_name):
    def one_test(self):
        self.assert_(test_name in ['aaa', 'bbb'])
    return one_test

for test_name in test_names:
    setattr(DynamicTestCase, 'test_' + test_name, make_test(test_name))

if __name__ == "__main__":
    unittest.main()

26 July 2006, update

I don't use packed tests anymore. It's too much complication without benefits. Instead, the code recursively walk down the current directory. If some folder contains the the subfolders "before" and "after", then the folder contains a test case.

I also found that it's better to run scons with the parameter "--debug=stree --debug=explain". This parameter reveals more scons thinkings and therefore makes testing more accurate.

Categories: consodoc

Updated: