ScriptTest

Most of the programs that I write for myself have command line interfaces. My functional tests for command line programs were always a little clunky and I never had a satisfactory solution for testing file system modifications that occur when the program creates or updates files. Functional test frameworks for web and GUI apps are quite common, but not so much for command line programs.

A few months ago I discovered ScriptTest which solves the problem quite nicely. I've been using it in my Tom's TV Utility project and am very pleased with the results. ScriptTest is quite easy to use but the online documentation is a little out of date so I recommend reading the __doc__ strings. Oh yes, ScriptTest is written in Python.

The online documentation describes a simple test so I'll show a more complex one from my project.

import os
from scripttest import TestFileEnvironment

class TestTomsTV:
    def setup(self):
        self.env = TestFileEnvironment(
                base_path=os.path.abspath('tests/test-output'),
                cwd=os.getcwd(),
                script_path=[os.getcwd()],
                environ={
                    'PATH': '/usr/local/bin:/usr/bin:/bin',
                    'HOME': os.path.abspath('tests/test-output'),
                   },
                template_path=os.path.abspath('tests/data'),
                ignore_hidden=False)

OK, maybe that's a little too complex ;-) but it shows ScriptTest's flexibility.

• base_path sets the directory where the utility will write files. The default is ./test-output.
• cwd sets the working directory for the program under test. The default is base_path.
• script_path limits where the framework will search for the program under test. If you do not give the full path of the program, it will be found the normal $PATH which is not desirable in development.
• environ passes a custom environment to the program under test. My utility depends on $HOME so I ensure that the variable points to the base_path directory.
• template_path sets the directory where preset files will be found. This is handy feature which allows tests to initialize files in the base_path area.
• ignore_hidden toggles how the framework reports hidden file and directories. My program creates a hidden directory so I require the framework to record that detail.

Here's a sample test.

    def test_list(self):
        'Test: tomstv list'
        self.env.writefile('.tv/db.sqlite', frompath='file1.sqlite')
        cmd = 'tomstv list'
        result = self.env.run(cmd)
        assert result.returncode == 0,\
                'Expect returncode=0, got %r' % result.returncode
        assert result.stderr == '',\
                'Expect stderr=%r, got %r' % result.stderr
        expect = data.file1_list
        print result.stdout.split('\n')
        for r, e in zip(result.stdout.split('\n'), expect.split('\n')):
            assert r == e, '\nExpect %r\ngot %r' % (e, r)

The writefile method copies a test database from the template_path mentioned before and the run method executes the program. When it completes, you can verify output is correct and check files that where changed, created, or deleted, but this test doesn't do that.

If you write functional test for command line programs, you should definitely add ScriptTest to your toolbox.