Greg is a framework for testing other programs and libraries. Its purpose is to provide a single front end for all tests and to be a small, simple framework for writing tests. Greg leverages off the Guile language to provide all the power (and more) of other test frameworks with greater simplicity and ease of use.
The simplicity of the Greg framework makes it easy to write tests for any program, but it was specifically written for use with GNUstep-Guile to permit direct testing of the GNUstep libraries without the necessity to run a separate driver program.
The core functionality of Greg is a Guile module which can be loaded into any software with an embedded Guile interpreter. Any program which uses Guile as it's scripting language can therefore use Greg to test itself directly!
For testing external programs, Greg provides a compiled module that may be dynamically linked into Guile to permit you to run an application as a child process on a pseudo-terminal. In conjunction with the standard Guile `expect' module, this lets you test external programs.
Also provided is greg - a Guile script to invoke the Greg test framework in much the same way that runtest is used in DejaGNU.
All tests have the same output format (enforced by the greg-testcase procedure). Greg's output is designed to be both readable and readily parsed by other software, so that it can be used as input to customised testing processes.
Greg provides most of the functionality of DejaGNU but is rather simpler. It omits specific support for cross-platform/remote testing since this is really rather trivial to add where required and tends to vary from site to site so much that an attempt at a generic solution is pretty pointless. What Greg does do, is provide hooks to let you easily introduce site specific code for handling those sorts of situations.
The current version of Greg can normally be found on GNU ftp sites and at http://www.tiptree.demon.co.uk/gstep/guile/greg-0.6.tar.gz with documentation online at http://www.tiptree.demon.co.uk/gstep/guile/gregdoc_toc.html
or, for the bleeding edge - availably by anonymous cvs as part of the GNUstep-Guile package in the GNUstep project -
CVSROOT=":pserver:anoncvs@cvs.net-community.com:/gnustep"
export CVSROOT
cvs login (password is `anoncvs')
cvs -z3 checkout guile
To run tests from an existing collection, try running
make check
If the check
target exists, it usually saves you some
trouble--for instance, it can set up any auxiliary programs or other
files needed by the tests.
Alternatively, if you are in the top-level source directory of an existing
testsuite (ie. there are subdirectories containing files with a .scm
extension), you can get the greg script to test all the tools in the
directory by typing -
greg
If you have a test suite that is intended to be used for `embedded' testing - You need to start the application to be tested, gain access to it's Guile command line (or other guile interface) and enter the commands -
(use-modules (ice-9 greg)) (greg-test-all)
Each Greg test is a Guile
script; the tests vary widely in
complexity, depending on the nature of the tool and the feature tested.
; ; GNUstep-guile interface library test ; ; Create an object using the NSString [stringWithCString:] method and ; check that the resulting object is of the correct class. ; (greg-testcase "The `stringWithCString:' method creates an NSString object" #t (lambda () (define obj ([] "NSString" stringWithCString: "Hello world")) (gstep-bool ([] obj isKindOfClass: ([] "NSString" class))) ))
Though brief, this example is a complete test. It illustrates some of the main features of Greg test scripts:
gstep-bool
and []
provided by the GNUstep-Guile library.
greg-testcase
to record the test outcome.
You pass this procedure a string (`assertion') describing the testcase,
a boolean specifying the expected outcome, and a `thunk' (parameterless
procedure) that performs the actual test.greg-testcase
so that, in the event of an error, the greg-testcase
procedure can trap it and report an unresolved test.(define obj ([] "NSString" stringWithCString: "Hello world"))
could have
appeared outside the testcase, but that would have been less safe.
Greg was written to support regression testing for the GNUstep libraries. It was inspired by an earlier test framework (by Ovidiu Predescu) that used DejaGNU along with a `driver' program (to make the calls to the library) and a suite of TcL scripts to control the driver.
There were three main problems (inherent in the nature of DejaGNU) with that approach -
So, something different was required, a test framework in a safer, simpler language that made it easy to create thin interfaces to libraries, so simplifying the task of producing testcases.
Of course, the good points of DejaGNU needed to be retained - clear output, Posix compliance, the ability to test separate programs as well as libraries.
A couple of additional goals seemed worthwhile -
Hopefully, Greg meets all its goals.
This section is copied almost directly from the DejaGNU documentation with minor modifications -
Greg is believed to conform to the POSIX standard for test frameworks.
POSIX standard 1003.3 defines what a testing framework needs to provide, in order to permit the creation of POSIX conformance test suites. This standard is primarily oriented to running POSIX conformance tests, but its requirements also support testing of features not related to POSIX conformance.
The POSIX documentation refers to assertions. An assertion is a description of behavior. For example, if a standard says "The sun shall shine", a corresponding assertion might be "The sun is shining." A test based on this assertion would pass or fail depending on whether it is daytime or nighttime. It is important to note that the standard being tested is never 1003.3; the standard being tested is some other standard, for which the assertions were written.
As there is no test suite to test testing frameworks for POSIX 1003.3 conformance, verifying conformance to this standard is done by repeatedly reading the standard and experimenting. One of the main things 1003.3 does specify is the set of allowed output messages, and their definitions. Four messages are supported for a required feature of POSIX conforming systems, and a fifth for a conditional feature. Greg supports the use of all five output messages; in this sense a test suite that uses exactly these messages can be considered POSIX conforming. These definitions specify the output of a test case:
PASS
UPASS
PASS
, instead of UPASS
, is returned
for test cases which were not expected to pass but did. This means that
PASS
is in some sense more ambiguous than if UPASS
is also
used.
FAIL
FAIL
message is based on the test case only. Other messages are used to
indicate a failure of the framework.
XFAIL
FAIL
, instead of XFAIL
, is returned
for test cases which were expected to fail and did not. This means that
FAIL
is in some sense more ambiguous than if XFAIL
is also
used.
UNRESOLVED
PASS
or
FAIL
before a test run can be considered finished.
Note that for POSIX, each assertion must produce a test result
code. If the test isn't actually run, it must produce UNRESOLVED
rather than just leaving that test out of the output. With Greg this is
not a problem - any unexpected termination of a greg-testcase
procedure will produce UNRESOLVED
.
Here are some of the ways a test may wind up UNRESOLVED
:
UNTESTED
The only remaining output message left is intended to test features that are specified by the applicable POSIX standard as conditional:
UNSUPPORTED
Greg uses the same output procedures to produce these messages for
all test suites, and these procedures are already known to conform to
POSIX 1003.3. For a Greg test suite to conform to POSIX
1003.3, you must set the variable greg-posix
to be true (or run
the greg
command with the --posix
flag).
Doing this will ensure that non-posix extensions are not used.
Go to the first, previous, next, last section, table of contents.