Extended LXRT - A New Concept
=============================

We try to provide hard real time services in user space, also for normal, 
i.e. non root, users. We think that it will not be as good a performer as
kernel space real time task modules, but a few microsecs more latency can be
acceptable for many applications. Many users will be glad with it for itself.
At the very least, it will be usefull in easing development and many other 
things: training, teaching and so on. It is wholly along the basic LXRT 
concept so we have seen it as an extension to LXRT.

To get hard real time in user space you need a fully preemptable kernel. The
question, within RTAI "philosphy", is how to get full preemption with minimum 
changes, possibly none, to the kernel source.

The solution calls for a compromise. We propose to accept that a hard real
time process does no Linux context kernel operations leading to a task switch.
In than sense, it is better to speak of a "user space kernel module", and we
will use the two terms interchangeably.

The approach is similar to what one of us did when he was using QNX: he always
mated a hard real time tasks with a buddy for any I/O operation that could
lead to excessive delays. In fact, even within such a fully preemptable kernel,
I/Os could lead to deadlines misses under heavy hard real time I/O load from 
many hard real time tasks. Many examples in this distribution, i.e.: clocks, 
latency calibration and sound, show you a clear picture of how easy it is to
use kernel services by mating to a buddy server process, without any problem.

So, at least on the base of our modest experience, that is not an unbearable 
constraint. Since RTAI has many good intertask services, we do not see any 
problem in using the same approach again, especially in view of with what 
Pierre has done, is doing and will do, to make it the "informed" way.
It is nonetheless possible that such a constraint will be somewhat lifted as
development proceed. Moreover the user space approach does not forbid you to
do it in kernel space, if it is eventally needed. In fact it is not seen as
an alternative to doing it in the kernel, but simply as a way of giving you 
more opportunities, at least during the development phase.

Taking into account that the present solution is just at the beginning of its
development, we see a lot of space for making it better.


How is that possible?
---------------------

We think that what you'll have here shows that it can work, even if it can be
improved. The idea is to keep soft interrupts disabled for hard real time user 
space modules. This way, kernel module hard real time tasks and hard real time 
interrupts can preempt user space modules, but user space modules cannot be 
preempted neither by Linux hard interrupt nor by Linux processes. 
Linux hardware interrupt are pended as usual for service when RTAI's real time 
tasks (both in the kernel and user space) are idle.


How does it work?
-----------------
 
Hard real time user space modules are just normal Linux processes that mate 
to a special buddy hard real time kernel task module, as done under LXRT 
already. They must be POSIX real time Linux processes locked into memory.
To distinguish them from usual LXRT firm real time processes the user simply 
calls rt_make_hard_real_time(), whereas by using rt_make_soft_real_time() he 
can return to standard Linux task switching.
Note that some of the required features, e.g POSIX real time under Linux,
require root permission. However by using the function rt_allow_nonroot_hrt() 
you are allowed to: make a process POSIX real time, lock the memory and do IO
operations, as a normal non root user. It is nonetheless necessary that the
superuser "insmod"s the required modules (rtai, rtai_sched and lxrt).

The call to rt_make_hard_real_time allows to take a normal process out from 
the Linux running queue by calling schedule() after having queued the task to 
a bottom handler. When the bh runs, the task is scheduled as a hard real time 
module by lxrt_schedule(), and Linux will not know of it, while having it still 
in its process list.

Lxrt_schedule() is also set as the signal function to be called when returning 
to the Linux context from a hard real time kernel space schedule, thus ensuring
preemption in any case.

Lxrt_schedule() clear the soft interrupt flags and mimics the Linux schedule() 
function, with scheduling policy SCHED_FIFO, even from within interrupts.

To return to soft real time, rt_make_soft_real_time() does the opposite.


What it currently does
----------------------

There are some (not so) simple test processes that runs periodically and on
which scheduling latency is measured. No doubt that it does something
different as by running the same tasks under the same load with plain LXRT
the latency goes as high as Linux 10 ms tick, compared to a few tens of
usecs under user space modules (preliminary rough measures). Note that
within this new context it is likely that you can use also Linux pthreads
both for soft and hard real time. 

In fact pthreads are normal user processes in disguise, Xavier made a
choice, i.e. pthreads as cloned processes, that is good also for LXRT.
Other examples show interacting tasks at work, while the sound task gives
an idea of IOs from user space.

The experience gathered so far indicates that, despite the availability of
more processing power, under SMP the latency for the same background load
can be double/tripled with respectg to UP. That is likely due to cache
trashing caused by processe switches and seems not to depend on the RTAI MP
scheduler you are using. So it makes a larger jitter difference, with
respect to working in kernel space, using hard real time processes under
SMP than under UP.  In fact under UP the jitter is roughly the same weather
you are using user or kernel space modules.


What it currently does not very quickly
---------------------------------------

Lxrt_schedule() can schedule in and out plain Linux processes, but to do it
safely that must happen within Linux idle tasks.  Clearly when one tests
under heavy load the starting and ending of hard real time mode can be
somewhat sluggish. In any case problems are just in starting and ending,
once user space modules are in place they are fine. The matter has been
somewhat improved by forcing the scheduling weight of the idle task, just
four lines added/modified within the kernel. We know that there can be
other ways of doing it, but all what we could conceive is likely to require
heavy kernel modifications. Once more we recall that all our "philosophy"
is to deplete the kernel with the slightest changes possible to it, better
if none.

Note that within lxrt.c we trapped the kernel sys call and interrupt
enabling to be sure that they are not called within hard real time user
space modules.  The same thing is possible, done directly in rtai.c, for
all the reserved Linux traps, but no alternative handler has been
implemented yet.
 
The new additions to lxrt:

- changed rtai_lxrt_handler to avoid ret_from_intr if returning from within a
  a hard real time process;

- added macros my_switch_to(prev,next,last), loaddebug and function 
__switch_to, all copied from Linux;

- added lxrt_schedule to schedule hard real time user space tasks among
  themselves and to and from the Linux context, with soft flags disabled
  (__cli()), a lot of new data needed are found just above it, the name
  should self explain them;

- added function lxrt_do_steal to be run from the bh timer to schedule a new
  hard real time process; 

- added the pointer rthal_enint to save the trapped trap rtahl.enint in order 
  to diagnose enable from within rt user space modules;

- added lxrt_enint to actually do the above trapping;

- added lxrt_sigfun to lxrt_schedule when getting back to Linux from the rtai
  schedulers;

- added steal_from_linux to make a Linux process a user space hard real time
  module;

- added give_back_to_linux to return a user space module to the Linux processes;

- added linux_syscall_handler to save the trapped Linux sys handler;

- added lxrt_linux_syscall_handler to diagnose calls to sys from hard real time
  processes;

- print_to_screen to allow a safe printing of diagnosting messages from within
  user space modules working in hard real time mode.

User functions:

- print_to_screen(const char *format, ...): to safely print information and
  diagnostic messages in hard real time user space modules;

- void rt_make_soft_real_time(void): to return a hard real time user space
  process to soft Linux POSIX real time;

- void rt_make_hard_real_time(void): to make a soft Linux POSIX real time
  process a hard real time LXRT process;

- rt_allow_nonroot_hrt(void): to allow a non root user the make a process
  Linux POSIX real time, lock process memory in ram and carry out IO operations
  from user space.


Tests
-----

There is a wealth of examples to show extended lxrt operations, both in soft 
and hard real time mode. They can be useful also in giving you some clues for i
your applications.


Tests list
----------

- single task (directory one);
- two tasks   (directory two);
- many tasks  (directory many);
- many tasks  (directory forked);
- many pthreads (directory threads);
- latency calibration (directory latency_calibration);
- sound test (directory sound);
- digital clock with semaphores (directory sem_clock);
- digital clock with messages (directory msg_clock).
- task resumed from an interrupt handler (directory resumefromintr).

The possibility of using pthread_create to generate Linux processes is very
usefull since it allows a task layout that is close to the structure of
modules.  That could make it easier the translation to kernel modules for
maximum performances. Also to be remarked is the possibility of resuming
user space modules directly from interrupt handlers, see example
reseumefromint.

If you want to check the jitter while one of the clocks or the sound example 
are running, you should enter the latency_calibration directory under another 
screen and type "./rt_process 1 &" followed by "./check". Try it varying Linux 
load. Be carefull, you must end it before closing the clocks/sound tests, see 
a more detailed comment within README in latency_calibration directory.

Have a look at the README files in each directory for more informations.

It is important to remark that what is found under this directory can be
used for any application but it is intended mainly for devlopment work. It
will be soon ported to lxrt-informed for a safer production use.  Thus it
is remarked that you must install a SIGINT handler if you want to safely
terminate you LXRT processes, cleaning up any RTAI resouce they use, after
Ctrl-C. Some examples show how it can be done.  

We remind once more that what you find in directory lxrt is the final
development version, the related production version is in lxrt-informed.
It may happen that under this directory you can find features not yet
ported in lxrt-informed. It will likely be so for a very short time. So
take care of abormal terminations yourself or wait for help from
lxrt-informed.

Contributed by: Pierre Cloutier, Paolo Mantegazza, Steve Papacharalambous.
