xautotest README

			Last modified: October 20, 1993

New in this release:

	- Change named from xrecplay to xautotest to avoid confusion
	  as to what this distribution is
	- WarpCursors are now recorded - before if a client warped
	  the cursor, the playback would be off track
	- Pixmap comparison made easier and more portable for playing
	  back on either color/bw terminals when Colormap matching is off.
	- no warning messages about the colormap when colormap matching
	  is off
	- bug fix which keeps the xtmconvert client from stripping image
	  information when converting between file formats
	- bug fix so you can build the server without the extension defined
	- bug fix in XTest extension where record would record wrong
          x,y positions of the cursor if you touched the bounds of the
          screen (now records position after server constrains it)
	- partial window matches are now specified by the mouse instead of
	  the keyboard

Modifications in previous release:

        - rewrote Imakefile and put all tools under one directory
        - made modifications to includes, signals and prototypes
          for porting to Sun
        - modified exmatch.c not to check window x and y location
        - fixed variable bug in xtmrecord
        - added printf when entering and exiting command mode
        - changed exmatch to be easier on the window checking
          when not doing colormap compares (was saying windows
          werent the same when they were)
	

xautotest DESCRIPTION:
======== ============

This distribution describes how to build the X test extension into an X11R5
Sun server and contains a set of tools for automated testing of X clients.
This automated testing includes record and playback of mouse and keyboard
events and pixmap comparisons of images on the display to verify correctness.
It should work with all X clients, including Motif, Xaw and xview clients and
window managers.

To build the clients in this distribution, you need to build an X11R5 Sun
X server and the libraries (libX11 and libXext) from the MIT distribution.

Most of the credit goes to others for more of the extension and the
xtm tools, I only wrote part of the extension, packaged everything up,
did bug fixes and wrote one additional client.

The bug fixes in this release to the X test extension are crucial for
the tools to work, so if you already have a server with the X test
extension in it, you should still consider building a Sun server with
this version of the X test extension to make sure things will work correctly.

At some point, I'm sure the X sample Sun server will be distributed with
a fully implemented X test or X trap extension.  This distribution will
become obsolete at that time (although the tools may not).


Why a server extension? Why do I have to rebuild a X server?
=== = ====== ========== === == = ==== == ======= = = =======
 
There are a number of approaches to automated testing I could have worked
on.  Unfortunatly, the only one which resolved the real-world testing
issues involved making server modifications.
 
I thought about implementing a record-playback facility which sits in
the NextEvent loop of an X Client, recording events as they come into
the client and then playing them back by feeding them back to the
client though the NevtEvent loop.
 
This does not allow you to test any inter-application functionality (such as
interaction of a client with the window manager) since the only client
getting events is the one with the modified NextEvent loop.
 
Any X approach you take will have problems with the GrabServer protocol
request.  This request shuts down connections to all clients except for
the one which performed the GrabServer.  Window managers commonly do a Grab
Server when you move a windows border around the screen so it can use XOR
drawing without worrying about things changing under the border while
it does this.  If you are trying to send mouse movements to the X server
when a client does a GrabServer, the movements stop.  This problem will appear
if you are moving a window around the screen and the window manager does
a GrabServer, the machine will lock up if the client which is sending
the mouse movements gets cut off by the Grab Server.  The window manager
will be waiting for a mouse release to finish the move but the client
which is trying to send the mouse release to the server wont be able
to because the window manager has a GrabServer on the server.  In this
release while executing a playback, GrabServers only "pretend" to work.
This is a suitable solution in practice.

There are a number of commercial packages available for automated X testing.
I initially evaluated some of them before working on this distribution.  When
evaluating them, check them for the following items:
 
        1] Record a script where you are using mwm and move a window
           around by its border for 5 seconds.  Play this back.
 
        2] Ask about how bitmap comparisons are done.  Do they work
           on color and monochrome screens?  Do they work on screens
           with different widths/heights?
 
           Run a client on a color machine which allocates colors before
           recording a test script.  Record a test script with a color
           bitmap comparison.  Shut the server down and restart it without
           running the color allocating client first.  Play the script back.
           Does the test pass?
 
        3] Try recording a drag and drop operation and playing it back.
           Try recording a cut and paste operation and playing it back.
 
        4] Do the windows have to be in the same place on the screen
           during playback?  How are relative screen positions handled
           (if they are at all)?
 
        5] Check to see if double and triple clicks recorded are played
           back as expected.
 
        6] If testing a Motif app, record a script on the periodic table
           of Widgets, trying a few things on each widget and play the
           script back.
 
        7] Does the tool have a simple programmable interface so you
           can add looping, if-then constructs for test passes, etc?
 

MAKING AND RUNNING the xautotest tools:
====== === ======= === ======== ======

Building an X11R5 Sun server with the XTEST extension:

1.1]    You must have or get code for the X11R5 Sun server.
        The server should contain a fragment of code in
        mit/server/os/WaitFor.c like the following:

		#ifdef XTESTEXT1
		if (playback_on) {
		wt = &waittime;
		XTestComputeWaitTime (&waittime);
		}
		#endif /* XTESTEXT1 */

        At some point, MIT may delete this XTESTEXT1 code from the sample
        distribution (its been there for a few years, though).  If this
        happens, you will need to get an older version and re-implement
        the code (or hopefully someone will modify this distribution to
        work with it).

1.2]	Patch the server code.

	cd to the mit directory of the distribution and type:
		patch -p < <this directory>/xtest.patch

	If you don't have the patch program, you could try to make
	the modifications by hand.  In the xtest.patch file, lines
	preceeded with a '+' are to be added.  Lines preceeded with
	a '!' are to be modified.

1.3]	Build the X11R5 server (in mit directory type make World and
	muddle though the build instructions when it doesnt work).

1.4]	Replace your server with the newly built X server (its in
	mit/server/Xsun) or if you are using xinit, use your
	.xserverrc startup to run the new server (probably a better
	choice).  Exit the X environment and restart X with the
	new X server.

	To make sure you are running the new X server, use the X
	client xdpyinfo.  The extensions section should contain
	a reference to the XTestExtension like:

		number of extensions:    6
			XTestExtension1
			SHAPE
			...


The xtm tools are the clients which let you do recording/playback
and script conversion.  You need to build them with the include files
and the libraries which were built when you built the X server
in steps 1.0-1.4.  xmkmf is used to build the clients below and this
assumes you have installed the X distribution in steps 1.0-1.4.

If you are not installing the X which was built in 1.0-1.4 then you
should build the xtm tools below with the include files in
mit/extensions/include, mit/lib/X and mit/include.  You also need to
build with libraries in mit/lib/X/libX11.a and
mit/extensions/lib/libXext.a.  This will require you to modifications
to the makefiles generated by the xmkmf (the specific
modificications will depend on where you built the X server
for 1.0-1.4).

Building and verifying the xtm tools:

2.1]	% cd xtmtools
	% xmkmf
	% make

2.2]	In your .Xdefaults add:
		*CommandKey:            F21

	You may also want to add:
		*CompareColorMap:       no
		*MatchBufferSize:       80000

	Then load this change with:
		% xrdb -load .Xdefaults

	F21 should be the Pause key on your keyboard (keycode 28).
	You can check this with the X11 client xev.  Run xev and move
	the mouse cursor into the xev window.  Press the Pause key on
	your keyboard.  The keycode displayed should be 28.  Otherwise,
	find out which key is keycode 28, this will be your command
	key when using xtmrecord.  Do not change the command key to
	something else or you will experience little problems (the
	press of the command key may not go into command mode until
	you generate a couple more events because of buffering).

2.3]	% xtmtools/xtmrecord test1.scr
	move mouse
	press the "Pause" (command) key (keycode 28)
	press q

2.4]	% xtmtools/xtmexecute ../xtmrecord/test1.scr
	The session should play back your move movement and then quit.

Building the xwdiff client (optional)

3.1]	% cd xwdiff
	% xmkmf
	% make

HINTS FOR BUILDING test scripts:
===== === ======== ==== ========

1]	Don't do a full test in one script, modularize and split
	up the test.  For example a test for the bitmap client
	may look like:

	#!/bin/sh

	Playscript()
	#Passed: <script name>
	#Execute a playback script using the xtm record/playback
	#script tools. A .scr is tacked on to the script name to
	#find the actual playback file name
		{
		if xtmexecute $1.scr 1>/dev/null 2>&1; then
			if [ "$verbose" = "on" ]; then
				echo "[Xscript: pass: script $1 done]"
			fi
			return 0
		fi
		if [ "$verbose" = "on" ]; then
			echo "[Xscript: fail: script $1 aborted]"
		fi
		return 1
		}

	Waitfor()
	#Passed: <client name>
	#Wait until a client maps to the screen
		{
		code to wait for $1 to map to screen.....
		}

	test_bitmap()
		{
		bitmap -geometry +0+0 &
		Waitfor bitmap
		if Playscript script1; then #new file
			if Playscript script2; then #draw line/verify
				if Playscript script3; then #draw circle/verify
					return 0
				fi
			fi
		fi
		return 1
		}

2]	Set the window manager options to force you to place each window.
	That way, windows will appear in the same location on a playback
	run.

3]	Use shell commands to do looping. For example, something like:

	#!/bin/csh -f

	set count = "0"
	while ("$count" < "10")
		Playscript script1.scr;
		@ count++
	end

4]	Have your script wait for a client to map before starting the
	test.  See the example above, make a Waitfor function and use
	X core clients to check to see when the client has mapped (i.e.
	xwininfo).  You can do a pixmap comparison on the contents of
	the window to make sure it is "all the way up" before the
	script starts.


PROBLEMS WITH the xautotest tools:
======== ==== === ======== ======

xtmrecord is very slow saving window matches when running in an NFS
mounted directory (can take minutes to do a screen capture).  Run on
a non-NFS mounted directory and it should be much quicker (couple seconds).

There still seems to be some timing problems with double clicks being
played back as single clicks.

CREDITS:
========

Eric Swildens, while contracting at HaL Computer Systems, wrote this README,
implemented part of the Sun X test extension, fixed some bugs in it,
wrote the xwdiff client and fixed some bugs in the xtm tools. Most
of the extension was hacked from what comes with the X11R5 server.

Most (90%) of the work was done by others.  Id say who they are, but
their COPYRIGHT notice seems to frown on that.  Thanks go to them for
the X test extension and the xtm tools.

Addition thanks go Jeff Woolsey who did Beta testing.

See the COPYRIGHT notice for a full list of copyrights.

Another quality submission from:	ess@netcom.com
					Eric Swildens