31.2. External Modules

Platform Dependent: UNIX, Win32 platforms only.

31.2.1. Overview
31.2.2. Module initialization
31.2.3. Module finalization
31.2.4. Function EXT:MODULE-INFO
31.2.5. Function SYS::DYNLOAD-MODULES
31.2.6. Example
31.2.7. Module tools
31.2.7.1. Modprep
31.2.7.2. clisp.h
31.2.7.3. Exporting
31.2.8. Trade-offs: FFI vs. C modules
31.2.9. Modules included in the source distribution
31.2.9.1. Base and Full Modules

Modules on Win32

Everything described in the section will work verbatim on Win32 when using Cygwin or MinGW, except for one thing - you will need to replace the run extension in lisp.run with the Win32 executable extension exe.

For historical reasons, all examples appear to assume UNIX and use the run file type (“extension”) for the CLISP run-time. This does not mean that they will not work on Win32.

31.2.1. Overview

CLISP has a facility for adding external modules (written in C, for example). It is invoked through clisp-link.

A module is a piece of external code which defines extra Lisp objects, symbols and functions. A module name must consist of the characters A-Z, a-z, _, 0-9. The module name “clisp” is reserved. Normally a module name is derived from the corresponding file name.

clisp-link needs a directory containing:

clisp-link expects to find these files in a subdirectory linkkit/ of the current directory. This can be overridden by the environment variable CLISP_LINKKIT.

clisp-link operates on CLISP linking sets and on module sets.

A linking set is a directory containing:

makevars

some /bin/sh commands, setting the variables

CCthe C compiler
CPPFLAGSflags for the C compiler, when preprocessing or compiling
CFLAGSflags for the C compiler, when compiling or linking
CLFLAGSflags for the C compiler, when linking
LIBSlibraries to use when linking
X_LIBSadditional X Window System libraries to use
RANLIBthe ranlib command
FILESthe list of files needed when linking
modules.h
the list of modules contained in this linking set
modules.o
the compiled list of modules contained in this linking set
all the FILES
listed in makevars
lisp.run
the executable
lispinit.mem
the memory image

To run a CLISP contained in some linking set directory, call

$ directory/lisp.run -M directory/lispinit.mem

or

$ clisp -K directory

(recommended, since it also passes -B to the run-time).

A module set is a directory containing:

link.sh
some /bin/sh commands, which prepare the directory before linking, and set the variables NEW_FILES, NEW_LIBS, NEW_MODULES, TO_LOAD and optionally TO_PRELOAD
and any other files
needed by link.sh

In link.sh the module set directory is referred to as $modulename/.

Module set variables

The following variables should be defined in link.sh.

NEW_FILES
the space-separated list of files that belong to the module set and will belong to every new linking set.
NEW_LIBS
the space-separated list of files or C compiler switches that need to be passed to the C compiler when linking the lisp.run belonging to a new linking set.
NEW_MODULES
the space-separated list of the module names belonging to the module set. Normally, every #P".c" file in the module set defines a module of its own. The module name is derived from the file name.
TO_LOAD
the space-separated list of Lisp files to load before building the lispinit.mem belonging to a new linking set.
TO_PRELOAD (optional)

the space-separated list of Lisp files to load into an intermediate lispinit.mem file, before building the lispinit.mem belonging to a new linking set. This variable is usually used to create (or unlock) the Lisp PACKAGEs which must be present when the new #P".c" files are initialized. E.g., the FFI:DEF-CALL-IN functions must reside in already defined packages; see Example 31.6, “Calling Lisp from C. You can find a live example in modules/syscalls/preload.lisp and modules/syscalls/link.sh.in.

Warning

If you are unlocking a package, you must also DELETE it from CUSTOM:*SYSTEM-PACKAGE-LIST* (see Section 30.2, “Saving an Image”) here and re-add it to CUSTOM:*SYSTEM-PACKAGE-LIST* in one of the TO_LOAD files. See, e.g., modules/i18n/preload.lisp and modules/i18n/link.sh.in.

Creating linking sets

The command

$ clisp-link create-module-set module file1.c ...

creates a module set in module directory which refers (via symbolic links) to file1.c etc. The files are expected to be modules of their own.

The command

$ clisp-link add-module-set module source destination

combines a linking set in directory source and a module in directory module to a new linking set, in the directory destination which is newly created.

The command

$ clisp-link run source module ...

runs the linking set in directory source, with the module in directory module loaded. More than one module can be specified. If CLISP has been built with the configuration option --with-dynamic-modules, the loading will be performed through dynamic loading. Otherwise - this is much slower - a temporary linking set will be created and deleted afterwards.

31.2.2. Module initialization

Each module has two initialization functions:

void module__name__init_function_1 (struct module_t* module)

called only once when CLISP discovers while loading a memory image that there is a module present in the executable (lisp.run) which was not present at the time the image was saved. It can be used to create Lisp objects, e.g. functions or keywords, and is indeed used for that purpose by modprep.

You do not have to define this function yourself; modprep and FFI will do that for you.

If you use FFI, (FFI:C-LINES :init-once ...) will add code to this function.

Warning

The PACKAGEs must already exist and be unlocked, cf. TO_PRELOAD.

Warning

If you are using modprep and defining your own “init-once” function, it must call the module__name__init_function_1__modprep function!

void module__name__init_function_2 (struct module_t* module)

called every time CLISP starts. It can be used to bind names to foreign addresses, since the address will be different in each invocation of CLISP, and is indeed used for that purpose by FFI (e.g., by FFI:DEF-CALL-OUT). It can also be used to set parameters of the libraries to which the module interfaces, e.g., the pcre module sets pcre_malloc and pcre_free.

You do not have to defined this function yourself; modprep and FFI will do that for you.

If you use FFI, (FFI:C-LINES :init-always ...) will add code to this function.

name is the module name.

See also Section 30.1, “Customizing CLISP Process Initialization and Termination”.

31.2.3. Module finalization

Each module has a finalization function

void module__name__fini_function (struct module_t* module)

called before exiting CLISP.

You do not have to defined this function yourself; modprep and FFI will do that for you.

If you use FFI, (FFI:C-LINES :fini ...) will add code to this function.

name is the module name.

See also Section 30.1, “Customizing CLISP Process Initialization and Termination”.

31.2.4. Function EXT:MODULE-INFO

Function (EXT:MODULE-INFO &OPTIONAL name verbose) allows one to inquire about what modules are available in the currently running image. When called without arguments, it returns the list of module names, starting with “clisp”. When name is supplied and names a module, 3 values are returned - name, subr-count, object-count. When verbose is non-NIL, the full list of module lisp function names written in C (Subrs) and the full list of internal lisp objects available in C code are additionally returned for the total of 5 values.

When name is :FFI, returns the list of shared libraries opened using :LIBRARY. When verbose is non-NIL, return the association list of DLL names and all foreign objects associated with it.

31.2.5. Function SYS::DYNLOAD-MODULES

Platform Dependent: Only when compiled with configure flag --with-dynamic-modules.

Note

Dynamic loading does not work on all operating systems (dlopen or equivalent is required).

Note

--with-dynamic-modules precludes some efficiency optimizations which are enabled by default.

Function (SYS::DYNLOAD-MODULES filename ({name}+)) loads a shared object file or library containing a number of named external CLISP modules.

Note

This facility cannot be used to access arbitrary shared libraries. To do that, use the :LIBRARY argument to FFI:DEF-CALL-OUT and FFI:DEF-C-VAR instead.

External modules for CLISP are shared objects (dynamic libraries) that contain the module__name__subr_tab variable, among others. This serves to register external functions which operate on Lisp-level structures with CLISP.

To use dlopen with modules, you should add -fPIC to the module's compilation options. Something like cc -shared -o name.so name.o may be needed to produce the shared object file.

31.2.6. Example

To link in the FFI bindings for the GNU/Linux operating system, the following steps are needed. (Step 1 and step 2 need not be executed in this order.)

  1. Create a new module set

    $ clisp-link create-module-set linux /somewhere/bindings/linux.c
  2. Modify the newly created linux/link.sh

    1. add -lm to the libraries

      replace

      NEW_LIBS="$file_list"

      with

      NEW_LIBS="$file_list -lm"
    2. load linux.fas before saving the memory image

      replace

      TO_LOAD=''

      with

      TO_LOAD='/somewhere/bindings/linux.fas'
  3. Compile linux.lisp, creating linux.c

    $ clisp -c /somewhere/bindings/linux.lisp
  4. Create a new linking set

    $ clisp-link add-module-set linux base base+linux
  5. Run and try it

    $ base+linux/lisp.run -M base+linux/lispinit.mem -x '(linux:stat "/tmp")'

31.2.7. Module tools

There are some tools to facilitate easy module writing.

31.2.7.1. Modprep

If your module is written in C, you can pre-process your sources with modprep in the CLISP distribution and define lisp functions with the DEFUN macro:

DEFUN(MY-PACKAGE:MY-FUNCTION-NAME, arg1 arg2 &KEY FOO BAR) {
  if (!boundp(STACK_0)) STACK_0 = fixnum(0); /* BAR */
  if (!boundp(STACK_1)) STACK_1 = fixnum(1); /* FOO */
  pushSTACK(`MY-PACKAGE::SOME-SYMBOL`); /* create a symbol in the package */
  pushSTACK(`#(:THIS :IS :A :VECTOR)`); /* some vector, created once */
  pushSTACK(``MY-PACKAGE::MY-FUNCTION-NAME``); /* double `` means FUNCTION */
  VALUES1(listof(7)); /* cons up a new list and clean up the STACK */
}

Then (MY-PACKAGE:MY-FUNCTION-NAME 'A 12 :FOO T) will return (A 12 T 0 MY-PACKAGE::SOME-SYMBOL #(:THIS :IS :A :VECTOR) #<ADD-ON-SYSTEM-FUNCTION MY-PACKAGE:MY-FUNCTION-NAME>) (assuming you EXPORTed MY-FUNCTION-NAME from MY-PACKAGE).

Another useful macros are:

DEFVAR
create a GC-visible private object
DEFFLAGSET
define a C function which will remove several flag arguments from the STACK and return the combined flag value
DEFCHECKER
define a C function which will check that its argument is one of the specified keywords and return the value of the appropriate cpp macro

See modules/syscalls/ and other included modules for more examples and file modprep for full documentation.

Warning

If you manipulate Lisp objects, you need to watch out for GC-safety.

31.2.7.2. clisp.h

If your module is written in C, you will probably want to #include "clisp.h" to access CLISP objects. You will certainly need to read "clisp.h" and the code in included modules, but here are some important hints that you will need to keep in mind:

  • Lisp objects have type object.
  • Variables of this type are invalidated by lisp memory allocation (allocate_*() functions) - but not C allocations (malloc et al) - and must be saved on the STACK using cpp macros pushSTACK(), popSTACK() and skipSTACK().
  • Access object slots using the appropriate TheFoo() macro, e.g., TheCons(my_cons)->Car, but first check the type with consp().
  • Arguments are passed on the STACK, as illustrated in the above example.
  • Wrap your system calls in begin_system_call()/end_system_call() pairs. These macros, defined in "clisp.h", save and restore registers used by CLISP which could be clobbered by a system call.

31.2.7.3. Exporting

If your module uses FFI to interface to a C library, you might want to make your module package case-sensitive and use exporting.lisp in the CLISP distribution to make FFI forms and DEFUN, DEFMACRO at al export the symbols they define. See modules/netica/, modules/matlab/ and modules/bindings/ for examples.

31.2.8. Trade-offs: FFI vs. C modules

When deciding how to write a module: whether to use FFI or to stick with C and modprep, one has to take into account several issues:

Speed: C wins

FFI has a noticeable overhead: compare RAWSOCK:HTONS (defined in modules/rawsock/rawsock.c) with

(FFI:DEF-CALL-OUT htons (:name "htons") (:library :default)
  (:arguments (s ffi:short)) (:return-type ffi:short) (:language :stdc))

and observe that RAWSOCK:HTONS is almost 3 times as fast (this really does compare the FFI overhead to the normal lisp function call because htons is computationally trivial). This difference will matter only if you call a simple function very many times, in which case it would make sense to put the loop itself into C.

Portability: C wins

First of all, FFI is not as widely ported as CLISP, so it is possible that you will face a platform where CLISP runs but FFI is not present.

Second, it is much easier to handle portability in C: observe the alternative implementations of htonl et al in modules/rawsock/rawsock.c.

Third, certain C structures have different layout on different platforms, and functions may take 64-bit arguments on some platforms and 32-bit arguments on others; so the FFI code has to track those differences.

Code size: FFI wins
You need to type much fewer characters with FFI, and, if you use the :LIBRARY argument to FFI:DEF-CALL-OUT and FFI:DEF-C-VAR, you do not need to leave your CLISP session to try out your code. This is a huge advantage for rapid prototyping.
UI: C wins
To produce a nice lispy UI (using &OPTIONAL and &KEYword arguments etc), you will need to write wrappers to your FFI:FOREIGN-FUNCTIONs, while in C you can do that directly. The same goes for “polymorphism”: accepting different argument types (like, e.g., POSIX:RESOLVE-HOST-IPADDR does) would require a lisp wrapper for FFI:FOREIGN-FUNCTIONs.
Learning curve: unclear

If you are comfortable with C, you might find the CLISP C module facilities (e.g., modprep) very easy to use.

CLISP FFI, on the other hand, is quite high-level, so, if you are more comfortable with high-level languages, you might find it easier to write FFI forms than C code.

Safety: unclear
One can get a segfault either way: if your FFI:DEF-CALL-OUT form does not describe the function's expectations with respect to the arguments and return values (including ALLOCATION), you will probably learn that the hard way. If the module is written in C, all the opportunities to shoot oneself in the foot (and other body parts) are wide open (although well known to most C users). However, with C, one has to watch for GC-safety too.

31.2.9. Modules included in the source distribution

The following modules come with the source distribution of CLISP (but are not necessarily built in a particular binary distribution):

bindings

Call the operating system functions from CLISP. The following platforms are supported:

CLX

Call Xlib functions from CLISP. Two implementations are supplied:

mit-clx, from MIT ftp://ftp.x.org/R5contrib/CLX.R5.02.tar.Z
the standard implementation
new-clx, by Gilbert Baumann

faster, with additional features, but not quite complete yet. Please try it first and use mit-clx only if new-clx does not work for you. new-clx comes with several demos, please try them: run

$ clisp -K full -i modules/clx/new-clx/demos/clx-demos.lisp

and follow the intructions.

oracle
Access Oracle from CLISP; by John Hinsdale.
fastcgi
Access FastCGI from CLISP; by John Hinsdale.
libsvm
Access LibSVM from CLISP;.
matlab
Do matrix computations via MATLAB.
netica
Work with Bayesian belief networks and influence diagrams using Netica C API.
postgresql
Access PostgreSQL from CLISP.
queens
Compute the number of solutions to the n-queens problem on a n×n checkboard (a toy example for the users to explore the CLISP module system, see modules/queens/).
dirkey
Directory Access (LDAP, Win32 registry etc).
berkeley-db
Berkeley DB from Sleepycat Software interface.
regexp
The POSIX Regular Expressions matching, compiling, executing. See Section 31.2.9.1, “Base and Full Modules”.
pari
Interface to the computer algebra system PARI.
pcre
The Perl Compatible Regular Expressions matching, compiling, executing.
i18n
Internationalization of User Programs. See Section 31.2.9.1, “Base and Full Modules”.
syscalls
Use some system calls in a platform-independent way. See Section 31.2.9.1, “Base and Full Modules”.
readline
When GNU readline and FFI are available, some advanced readline and history features are exported using this module. See Section 31.2.9.1, “Base and Full Modules”.
wildcard
Shell (/bin/sh) globbing (Pathname Matching).
rawsock
Raw socket access.
zlib
Compress VECTORs using ZLIB.

To use modules, read unix/INSTALL and build CLISP in directory build-dir with, e.g.,

$ ./configure --with-module=pcre --with-module=clx/new-clx --build build-dir

then run it with

$ ./build-dir/clisp -K full

This will create a base linking set with modules i18n, regexp and syscalls; and a full linking set with modules pcre and new/clx in addition to the 3 base modules.

See Chapter 32, Extensions Implemented as Modules for individual module documentation.

31.2.9.1. Base and Full Modules

Note that the default build process includes the following modules in both base and full linking sets:

i18n
regexp
syscalls
readline (only when both GNU readline and FFI are available)

The composition of the full linking set depends on the platform and on the vendor preferences.


These notes document CLISP version 2.41Last modified: 2006-10-13