There are four components in the system:
The driver reads the test scripts and asks the shim(s) to evaluate the individual tests therein. It then checks the shim's answers against expected values, which are also contained in the test scripts.
The watchdog listens for messages from the driver about its activities. If it does not hear from the driver within a certain time-frame, it attempts to kill the driver and any known shims before shutting itself down.
The Catechesis distribution includes the driver and watchdog, which are written in Perl 5. One shim is needed for each implementation of the spec or API you need to test, and it should be written in the same language as your implementation.
Tests are run with catechist
, the included driver
program. For complete documentation, see 'man
catechist'
after installing Catechesis. The general form is:
catechist --shim SHIM [OPTIONS] TEST_FILE [TEST_FILE..]
So to run the example suite from within the Catechesis distribution:
$ cd examples $ ../bin/catechist --shim ./perl5_shim test_suite.txt
Test script files contain stanzas of various types, comments, and blank lines. Leading whitespace is consumed before any other parsing is done, so scripts may be indented however is most readable.
Lines which begin with a pound/hash are comments. There are no multi-line comments, and hashes inside a line are not recognized as comments.
# this line is a comment and will be ignored # so will this one but this one isn't # and this text isn't either
Everything in a test script, except comments, must be contained in a stanza.
begin [type] [directive_name] [directive_value] end
Stanzas are delimited by lines with begin
or end
as the first word.
Every stanza has a type, which is given on
the begin
line. There are two valid types of
stanzas: test
and environment
.
Stanzas contain directives, which are instructions on what should be done.
The first word in a directive line is the directive name. The remainder of the line is the directive value. Values may span more than one physical line by ending lines with a backslash:
begin [type] directive1 all these words become the value of the "directive1" directive # the next directive spans three physical lines directive2 text \ more text\ still more text end
The value of directive2
would be 'text more text still
more text'.
Environment stanzas may contain two directives:
plan
diag
Test stanzas may contain four directives, two of which are required:
send
expect
send
directive.
desc
diag
The format of the send
and expect
directives are between you and your shim, so long as they are valid
JSON and vivify into a hash/dictionary/object. You will likely want
them to mirror your spec or API. See the
example Math API and the
standard
test script for an example.
Shims are uncomplicated. They are simple programs which form a thin interface between Catechesis and the implementation you are testing (hence: shim).
They communicate with the driver simply by reading
from STDIN
and writing to STDOUT
. They
don't (and shouldn't) know anything about Catechesis. They should
know only three things:
Remember that communication with the driver is via pipes, so your
shim must ensure that its writes to STDOUT
are
unbuffered/autoflushed, or are manually flushed with every write. If
this does not happen, deadlock will occur and the watchdog will kill
the shim process and the driver.
Finally, all shims should terminate execution when receiving a
message that contains the key QUIT
with a value
of QUIT
.