Design

There are four components in the system:

The driver reads the test scripts and asks the shim(s) to evaluate the individual tests therein. It then checks the shim's answers against expected values, which are also contained in the test scripts.

The watchdog listens for messages from the driver about its activities. If it does not hear from the driver within a certain time-frame, it attempts to kill the driver and any known shims before shutting itself down.

The Catechesis distribution includes the driver and watchdog, which are written in Perl 5. One shim is needed for each implementation of the spec or API you need to test, and it should be written in the same language as your implementation.

Running Tests

Tests are run with catechist, the included driver program. For complete documentation, see 'man catechist' after installing Catechesis. The general form is: 

catechist --shim SHIM [OPTIONS] TEST_FILE [TEST_FILE..]

So to run the example suite from within the Catechesis distribution: 

$ cd examples
$ ../bin/catechist --shim ./perl5_shim test_suite.txt

Writing Test Scripts

Test script files contain stanzas of various types, comments, and blank lines. Leading whitespace is consumed before any other parsing is done, so scripts may be indented however is most readable.

Lines which begin with a pound/hash are comments. There are no multi-line comments, and hashes inside a line are not recognized as comments. 

# this line is a comment and will be ignored
# so will this one
but this one isn't # and this text isn't either

Stanzas

Everything in a test script, except comments, must be contained in a stanza. 

begin [type]
    [directive_name] [directive_value] 
end

Stanzas are delimited by lines with begin or end as the first word.

Every stanza has a type, which is given on the begin line. There are two valid types of stanzas: test and environment.

Stanzas contain directives, which are instructions on what should be done.

Directives

The first word in a directive line is the directive name. The remainder of the line is the directive value. Values may span more than one physical line by ending lines with a backslash: 

begin [type]
    directive1 all these words become the value of the "directive1" directive
    # the next directive spans three physical lines
    directive2 text \
               more text\
               still more text
end

The value of directive2 would be 'text more text still more text'.

Environment stanzas may contain two directives:

plan
Specifies how many tests are intended to be run in this script. A plan must be given before a test stanza is encountered. If the actual number run does not match the plan, the script as a whole will be considered a failure.
diag
Causes its value to be printed as a diagnostic line in the TAP output stream

Test stanzas may contain four directives, two of which are required:

send
Required. The JSON which will be sent to the shim, carrying instructions and data for the test.
expect
Required. The JSON which you expect to get from the shim after it has carried out the instructions contained in the send directive.
desc
A freeform text description of the test, which will be printed in the TAP output stream alongside the test result
diag
Causes its value to be printed as a diagnostic line in the TAP output stream

The format of the send and expect directives are between you and your shim, so long as they are valid JSON and vivify into a hash/dictionary/object. You will likely want them to mirror your spec or API. See the example Math API and the standard test script for an example.

Writing Shims

Shims are uncomplicated. They are simple programs which form a thin interface between Catechesis and the implementation you are testing (hence: shim).

They communicate with the driver simply by reading from STDIN and writing to STDOUT. They don't (and shouldn't) know anything about Catechesis. They should know only three things:

Remember that communication with the driver is via pipes, so your shim must ensure that its writes to STDOUT are unbuffered/autoflushed, or are manually flushed with every write. If this does not happen, deadlock will occur and the watchdog will kill the shim process and the driver.

Finally, all shims should terminate execution when receiving a message that contains the key QUIT with a value of QUIT.