Writing tests

Before starting on writing test cases you need some familiarity with the GNUstep base library, Guile, Gstep-Guile and Greg. I recommend that you get the documentation for all these packages together on your machine and set bookmarks in your favorite web browser to point to it (and to this documentation of course).

The test suite for the gstep-base library consists of the basic framework plus directories containing tests for individual classes.

Mostly you should be able to pick up how to use this stuff simply by looking at the existing tests - pay special attention to what the initialisation file (`begin.grg') is doing.

One thing to note, if you are not familiar with Guile, is the scope of variables - if you want to pass information from one testcase to another you need to declare the variables outside the testcases (using the `define' primitive) -

(define arith-ok #f)	; For passing info between testcases

;
; A testcase to check an instance of numeric addition
;
(greg-testcase "One plus One is two" #t
(lambda ()
  (if (eq? (+ 1 1 ) 2)
    (begin (set! arith-ok #t) #t)
    #f)
))

;
; A testcase to check arithmetic - only supported if we have addition.
;
(greg-testcase "X multiplied by 2 is X plus X" #t
(lambda ()
  (if arith-ok
    (eq? (+ 1 1) (* 1 2))
    (throw 'unsupported))
))

Similarly, if you wish to restrict the scope of a variable to a group of testcases, wrap the whole lot in a `begin' -

(begin
  (define arith-ok #f)	; For passing info between testcases

;
; A testcase to check an instance of numeric addition
;
  (greg-testcase "One plus One is two" #t
  (lambda ()
    (if (eq? (+ 1 1 ) 2)
      (begin (set! arith-ok #t) #t)
      #f)
  ))

;
; A testcase to check arithmetic - only supported if we have addition.
;
  (greg-testcase "X multiplied by 2 is X plus X" #t
  (lambda ()
    (if arith-ok
      (eq? (+ 1 1) (* 1 2))
      (throw 'unsupported))
  ))
)

; The 'arith-ok' variable is no longer in scope.

initialisation

The initialisation file `begin.grg' is run before any of the tests are started. This file performs basic startup tasks including -

• Defining the default order in which tests are executed. When you add tests for a new class, you must modify this file so that it knows about the new class and when it should be tested.
• Initialising some useful variables. Handles for memory allocation zones etc.
• Defining procedures to make testing easier.
• Loading the `Protocols.scm' file (which defines procedures for checking that objects conform to specific protocols).

Tests for an individual class are placed in a directory whose name is the name of the class. Since the test scripts within a directory are normally executed in alphabetical order, you may want to add a `begin.grg' to override that default sequence of execution of scripts. If this has been done, you must modify the `begin.grg' file whenever you add a new script to the dictionary, so you may prefer to select your script names so that the default alphabetical ordering causes them to run in the correct sequence.
By convention we use - `basic.scm' for a script calling helper procedures to test that the class exists and conforms to the protocols it should.
`testXX.scm' where `XX' is a number from `00' to `99' for the other test scripts - so that they will be run in the order indicated by the numeric parts of their names.

A script file will contain one or more testcases - each of which constitutes a test of a single well defined feature of the `tool' that the script is meant to test. A testcase is always written using the `greg-testcase' procedure.

helper procedures

The testing framework defines various helper procedures (in `begin.grg' and `Protocols.scm') that you should probably use as the basis for a new testsuite for a class.

You should start any testsuite for a class with `test-alloc' to check that basic operations work. Next you should define a few variables for test objects, assign instances of the class to them (inside a testcase), and pass them to the `test-NSObject' procedure to check that they conform to the NSObject protocol. Then, before writing your own specific tests, you might use other procedures to check conformance to other protocols.

The helper procedures defined in `begin.grg' are -

• test-alloc
  The `test-alloc' procedure takes the name of a class and checks that instances of the class can be correctly created using all the standard allocation and initialisation operations.
  This procedure returns `#t' if all it's tests pass, `#f' otherwise.

• test-exception
  The `test-exception' procedure allows you to easily test that erroneous code fragments raise exceptions the way they should. This procedure takes three arguments -
  • assertion
    This is a brief text description of what the code fragment is doing and will be used in the output describing the test case.
    eg.
    "attempting to access beyond the end of an array"
  • exception
    This is the variable containing the OpenStep exception name constant.
  • thunk
    This is the code fragment (usually a lambda expression) that should be executed to demonstrate the exception.
  This procedure returns `#t' if the test passes, `#f' otherwise.

  
  ; Test NSString range checking exception.
  (test-exception
    "NSString extracting substring with range beyond end of string"
    NSRangeException
    (lambda ()
      (define r (NSMakeRange 6 4))
      (define s ([] "NSString" stringWithCString: "Hello"))
      ([] s substringWithRange: r)	; Should raise exception here.
      #f					; Shouldn't get to this line.
    )
  )
  
  

• test-alloc
  The `test-alloc' procedure takes a class name (string) as an argument and checks basic common stuff to do with creating objects of the class. It checks that the class exists and that the standard allocation and and initialisation methods work to create objects of that class.

  
  ; Test to see if NSString handles basic object creation tasks.
  (test-alloc "NSString")
  
  

protocol checkers

• test-NSCoding
  This procedure tests the NSCoding protocol by archiving and unarchiving objects and then using the [-isEqual:] method to check that the unarchived versions of the objects are equal to the originals.
  The only parameter used by this procedure is a list of objects to be tested.

  
  ; Test to see if NSString conforms to the NSCoding protocol properly
  (define myString ([] "NSString" stringWithCString: "Hello world.\n"))
  (test-NSCoding (list myString))
  
  

• test-NSCopying
  This procedure tests the NSCopying protocol. It takes five arguments, the name of an immutable class, the name of the corresponding mutable class, a list of objects to be copied, a flag to specify if a copy of the immutable object should be a simple retain (irrespective of zone) and a a flag to specify whether the copy of the immutable object should be a true copy (irrespective of zone). If both flags are false, the retain behavior required by the test is assumed to be that returned by the OpenStep NSShouldRetainWithZone() function. The procedure makes test copies using the default memory allocation zone and a test zone. It checks that the copy operation works to produce an object that [-isEqual:] to the original. If an object to be copied is immutable, the procedure checks to see that where the object should simply be retained rather than copied it is.

  
  ; Test to see if NSString conforms to the NSCopying protocol properly
  (define testString ([] "NSString" stringWithCString: "Hello"))
  (test-NSCopying "NSString" "NSMutableString" (list testString) #f #f)
  
  

• test-NSMutableCopying
  This procedure tests the NSMutableCopying protocol. It takes three arguments, the name of an immutable classe, the name of the corresponding mutable class, and a list of objects to be copied. The procedure makes test copies using the default memory allocation zone and a test zone. It checks that the copy operation works to produce an object that [-isEqual:] to the original. It also checks that the copy is a true copy, not just the original retained.

  
  ; Test to see if NSString conforms to the NSMutableCopying protocol properly
  (define testString ([] "NSMutableString" stringWithCString: "Hello"))
  (test-NSMutableCopying "NSString" "NSMutableString" (list testString))
  
  

• test-NSObject
  This procedure tests the NSObject protocol. It takes two arguments, the name of the class of the objects to check, and a list of objects to be tested for conformance to the protocol. All the objects to be tested must belong to the same class (or be instances of it's subclasses). Almost all the methods in the NSObject protocol are tested

  
  ; Test to see if NSString conforms to the NSObject protocol properly
  (define myString ([] "NSString" stringWithCString: "Hello\n"))
  (test-NSObject "NSString" (list myString))
  
  

greg-testcase

The `greg-testcase' procedure takes three arguments -

• The assertion
  This is a string describing the test to be performed. It should take the form of an assertion that is expected to hold true.
• The expected outcome
  This is a boolean value - `#t' if the test is expected to return `#t' (ie. the assertion is expected to be proved correct), and `#f' if the test is expected to return `#f' (ie. the assertion is expected to be proved to be incorrect).
• The test itself
  This is a `thunk' (a Guile procedure that takes no arguments) and is normally a `lambda expression'. The return value of this procedure should be `#t' if the assertion is proved correct, `#f' if it is proved incorrect.

The Guile programming language permits the `thunk' to return in four ways -

• Return the boolean value - `#t'
  In which case, the test has succeeded, the assertion has proved correct, and Greg will record either an expected success (PASS) or an unexpected success (UPASS) depending on the expected outcome passed to `greg-testcase'.
• Return the boolean value - `#f'
  In which case, the test has failed, the assertion has proved incorrect, and Greg will record either an expected failure (XFAIL) or an unexpected failure (FAIL) depending on the expected outcome passed to `greg-testcase'.
• Return a non-boolean value
  In which case, the test is unresolved (UNRESOLVED).
• Throw an exception
  In which case Greg's behaviour depends on the exception thrown -
  • 'pass
    Greg acts as if the test had returned the boolean `#t'
  • 'fail
    Greg acts as if the test had returned the boolean `#f'
  • 'unsupported
    Greg reports this case as unsupported (UNSUPPORTED)
  • 'untested
    Greg reports this case as untested (UNTESTED)
  • any other exception
    Greg reports this case as unresolved (UNRESOLVED)

As there are no other ways in which the `thunk' may be exited, it is impossible for a testcase to produce a result that doesn't fit into the framework (unless your testcase manages either to crash Guile or enter an infinite loop - in which case you won't get any output).

The value returned by the `greg-testcase' procedure is a boolean - `#t' if the test resulted in an expected pass, `#f' otherwise.
You can use this return value to make the execution of subsequent testcases dependent on the success of an earlier testcase.

;
; A testcase to check an instance of numeric addition
;
(greg-testcase "One plus One is two" #t
(lambda ()
  (eq? (+ 1 1 ) 2)
))

;
;  The above testcase will generate output -
;  'PASS: One plus One is two'
;

Go to the first, previous, next, last section, table of contents.