NET_RPC
=======

What found here makes RTAI a distributed system, both for kernel and user 
space applications. The use of "net_" in front of "rpc", for the name of this 
directory main module, is due to the existence of rpc functions internally 
to all RTAI schedulers, the concept of intertask Remote Procedure Call (RPC)
messaging being strongly supported within RTAI since its inception. Clearly 
nothing new, synchronous intertask message passing is an old concept and the 
basis of microkernels, either distributed or not. So this implementation is 
nothing but the RTAI specific way of doing it.

The basic specifications for making it easy are:

- use just the name of any already available function substituting "rt_..." 
  with "RT_...",
- add two initial arguments more, i.e. the "node" of and the "port" on the 
  remote machine that will execute "rt_..." functions that became "RT_...".

e.g.: 
	  rt_mbx_send(mbx, msg, msglen);
becomes: 
	RT_mbx_send(node, port, mbx, msg, msglen);.

Using 0 for the "node" forces a local execution of the corresponding "rt_..."
function. In this way you revert to a standard local application automatically.
"Port" is not used with a zero "node" but should be kept to a meaningfull value 
anyhow, to allow remote calls when "node" is set to point to a valid remote
machine again. So using "RT_..." naming with a null variable "node" allows a 
local application to be ready for becoming distributed, once you reassign a 
real "node". The only added cost will be a longer arguments list and the 
execution of a "C" "if" instruction. The "node" and "port" arguments will be 
explained shortly afterward.

Naturally you can also run a distributed application wholly in local mode by
just setting all your "node"s to the local host (dotted decimal notation: 
127.0.0.1); less effective than using a null "node" but useful for testing 
its networked execution on a single machine.

The only change in the formal arguments, with respect to the usual normal local 
"rt_..." usage, is related to any possible argument associated with a time 
value. It must be expressed in nanosecs always, instead of the RTAI standard 
internal timer units count. In this way you are not required to know the 
frequency of any timer running on a remote machine and be sure that a correct 
timing will be assured anyhow, irrespective of the node on which it will run. 

These are, more or less, the only things to know to make any RTAI application 
distributed. 

There is however a possible usage variation obtained by using "-port" instead
of "port" in any call. In fact a "port" is a positive number and making it 
negative forces an immediate return, i.e an asynchronous rpc. It is your 
responsability to ensure that an asynchronous rpc is reasonable, e.g. a 
"sem_signal" can be used in such a way, while a "sem_wait" will likely cause 
missing a possible synchronization. Notice that the remote support stub will 
answer anyhow but the returned values will be discarded. So it should never
be used if you want to know the result of your remote call. Nonetheless, as 
it will be explained later on, a mechanism is provided to allow recovering 
the results of an asynchonous call left behind.

In any case it is important to know that any further asynchronous call will 
not be sent if the previous one has not been answered yet. "Net_rpc" assumes no 
buffering at the remote node, i.e. one can be sure the remote port is ready
to accept calls only if its stub has answered to any previous service request 
anyhow. So, if the previous one has not answered yet, launching an asynchronous 
call will cause an immediate return without any action being taken. In relation 
to this it is important to note that the overall maximum message length 
allowed, both in sending and receiving is MAX_MSG_SIZE found in net_rpc.h, 
actually 1500 bytes, i.e. the number of octets defined by Linux ETH_DATA_LEN.

The user can check if the port has or has not answered an asynchronous RPC by 
calling:
	void *rt_waiting_return(unsigned long node, int port);
a non null return value implying "port" at "node" is waiting an rpc return.
Remote calls that make sense being used asynchronously can be used also in
interrupt handlers, see directory uresumefromintr for an example.

Before doing anything remotely a task working on remote services has to ask 
for a "port" at its remote peer. Such a port is obtained by a call of the type: 
	myport = rt_request_port(node); 
and released by:
	rt_release(node, myport);
when the "port" is needed nomore. A task cannot ask for more then one "port",
multiple "port" requests from a task will always return the same "port". The
assigned "port" will be an integer >= MAX_STUBS, defined in net_rpc.h.

Nonetheless a task can create and access more "port"s by using: 
	anotherport = rt_request_port(node, id);
"id" being a unique unsigned long "id" the task must have agreed with any
other local application. Releasing a "port" defined with a specific "id" 
makes no difference with the simpler request above, i.e.:
	rt_release(node, anotherport);
must be used anyhow.

In requesting a "port" there is also the possibility of providing a mailbox 
to recover results of asyncrhounous calls. So you can use either:
	myport = rt_request_port(node, mbx);
or:
	myport = rt_request_port(node, id, mbx);
"mbx" being a pointer to a mailbox. When a new rpc is made and there is the
result of any previous asynchronous call pending it will be sent to such a
mailbox. A typical use of this possibility consists in providing a server
task that reads the mailbox and uses the returned values in a manner agreed
with the original RPCs sender. A more direct way to ensure a pending return
is sent to the mailbox is to use:
	int rt_sync_net_rpc(unsigned long node, int port); 
which forces a synchronization, thus sending any pending return to a mailbox, 
if one is made available at the port request, it returns 1 always. The above
function allows to recover a pending return immediately, it is likely it
will be used in combination with "rt_waiting_return". 

A helper functions is provided to obtain any result queued in the mailbox:
	int rt_get_net_rpc_ret(
		MBX *mbx, 
		unsigned long long *retval, 
		void *msg1, 
		int *msglen1, 
		void *msg2, 
		int *msglen2, RTIME timeout, 
		int type
	);
mbx:		The mailbox
retval:		The value returned by the async call, if any. A double long 
		can contain any value returned by RTAI functions, it is up to
		you to use it properly.
msg1 and msg2:	Buffers for possibly returned messages.
msglen1, 
msglen2:	The length of msg1 and msg2, the helper function return the 
		actual length of the messages, truncating them to msglen1 and 
		msglen2 if their buffers are not larger enough to contain the
		whole returned messages.
timeout:	any timeout value to be used if needed by the mbx_receive 
type:		defined by type, the mailbox receive function to be
		used, i.e.: NET_MBX_RECEIVE for rt_mbx_receive, 
		NET_MBX_RECEIVE_WP for rt_mbx_receive_wp, NET_MBX_RECEIVE_IF 
		for rt_mbx_receive_if, NET_MBX_RECEIVE_UNTIL for 
		rt_mbx_receive_until and NET_MBX_RECEIVE_TIMED
		for rt_mbx_receive_timed.

The function is just a helper, a user can see it as a suggestion for his/her 
own optimised implementation, e.g getting just the returned value or a single 
message, because he/she knows those are the only returned values. See the test 
"uasync" for a specific example.
 
It must be rmerked again that even such an asynchronous form of "net_rpc" does 
not queue messages, as said it allows just one effective async call, but can 
help in increasing the application parallelism. A full queueing asynchronous 
form should not be overly difficult. It has not been adopted to avoid any 
assumption on the possibility of buffering messages at the remote node.

"Port" requests cause a task rescheduling so they must be called just from
within an RTAI task. Thus you have to care setting up an initializiation task 
if you want to use them in a "make it all at once" unified initialisation. 
Never call them directly in init_module functions.

"Port"s request/release need to be timedout, and possibly repeated, in case of 
collisions on the unique server "port" used preset and used by "net_rpc" to 
accept the related requests on the remote node. A Linux timer is used for this 
scope, since it is possible that no RTAI timer is running at the time a request 
is made. So "port"s request/release are not real time operations, they must be 
carried out before beginning any true real time work. Because of this assumptionthey are not time critical and can be timed out softly, without too much a 
hurry.

A further remark must be made in relation to intertask receive functions.
If you want to receive from any task there is no need to use RT_receive(x),
rt_receive(x) will receive also from remote tasks. If you want a specific
receive instead you must use RT_receive, recalling that you will receive from 
a local stub acting as the agent of the remote task. RT_receive(x) takes care 
of it, with a little added overhead since it has to find the stub associated 
to the node-port-task appearing in the RT_receive(x) argument list. You can
improve the efficiency of specific receives from remote tasks by finding
their local stubs yourself, using:
	RT_TASK *rt_find_asgn_stub(unsigned long long owner, int asgn);
where "owner" combines the remote "node" and "task". It can be built by using 
the macro: OWNER(node, task); "asgn" will assign a port-stub combination to 
the owner pair if it is different from zero and none exists yet. Any following
request for a "port" from the remote "task" will be given the one you have
created. With such a technique you can then effectively use a specific
rt_receive(x) on the task returned by rt_find_asgn_stub. The gain will be
more significant for user space applications, even if it is likely that you
will not notice any difference. 
See the 3 mode of executing found in "usoundmesg" for an example of what just
explained.

The same reasoning applies to RT_return(x). Take care of using RT_return(x)
with RT_receive(x) and rt_return with rt_receive, never mix them.  

Notice that "node" must be the networkwise integer corresponding to the
dotted decimal notation of the remote IP address. In kernel space the function 
"ddn2nl" is provided to transform the standard dotted decimal notation, 
xxx.xxx.xxx.xxx, into such an integer. In user space standard libc functions 
can be used directly, see the examples. 

The "ddn2l" prototype is:
	unsigned long ddn2nl(const char *ddn);
a null value being returned for a bad "ddn" string. A "ddn" string is assumed
to be bad if any of its dotted field contains a value greater than 0xFF.

The net_rpc driver module can be made to know its local node by being insmoded 
either with the parameter <ThisNode="xxx.xxx.xxx.xxx"> or by a call to: 
	rt_set_this_node(const char *ddn, unsigned long node);
made by the application module before using any net_rpc service. In the above 
call "ddn" is the dotted decimal notation string and node the corresponding 
"node". A call with a NULL "ddn" means that "node" contains a valid value while
a non NULL "ddn" implies that the local node is obtained from converting the 
related string, "node" being discarded.

It is important to remark that "net_rpc" uses RTNet supported APIs for its 
remote calls. RTAI CVS and distributions make available natively a soft RTNet 
support that uses standard Linux networking, thus lacking true real time
networked performances. To use it you have to insmode ""rtnet/krtnet" found 
here and execute "rtnet/urtnet" in background, i.e. "./rtnet/urtnet &".
You can avoid using "krtnet", but not "urtnet" by making "net_rpc" with the
macro "EMULATE_RTNET" defined.
For a real time networked support you should download "RTNet" from its home
site: http://www.rts.uni-hannover.de/rtnet/, recall to comment out 
"EMULATE_RTNET" in such a case.

It must be noted also that the specific implementation adopted for our RPC
scheme acts along the ideas already used for LXRT. In fact, networking
apart, the remote execution is carried out by a stub task, you can call it
buddy, proxy, agent. The possibility of expanding its use to any user specific
application is already in place, once more following the ideas used in 
extending LXRT.

To verify almost all the possibility offered by "net_rpc" there are:
- an example in kernel space (ktest),
- five examples in user space (utest, usound, usoundmsg, uresumefromintr, 
  uasync). 
Clearly if they work in user land it works also in the kernel one. That's why 
there is only one in kernel space. See aslo EXAMPLES.LIST and the READMEs in 
the related directories.

"Net_rpc" should now be a production tool and RTAI has also a production 
application, i.e. RTAI-Lab, that is fully based on "net"rpc". At the moment 
this text is the only documentation available. You should look at net_rpc.h, 
the above cited examples and RTAI-Lab to better understand what can be done 
with it. 

So the tool for integrated symmetricall local/distributed-kernel/user-space 
application is available and its usage is up to you.

For your convenience a list of the functions and macros explained in this README
follows. There is clearly no need to explain "RT_..." services since they 
follow the general rules explained at the very beginning.

- int rt_send_req_rel_port(unsigned long node, int port, unsigned long id, MBX *mbx);

- #define rt_request_port(node)           rt_send_req_rel_port(node, 0, 0, 0)
- #define rt_request_port_id(node, id)    rt_send_req_rel_port(node, 0, id, 0)
- #define rt_request_port_mbx(node, mbx)  rt_send_req_rel_port(node, 0, 0, mbx)
- #define rt_request_port_id_mbx(node, id, mbx)  rt_send_req_rel_port(node, 0, id, mbx)

- #define rt_release_port(node, port)  rt_send_req_rel_port(node, port, 0, 0)

- unsigned long ddn2nl(const char *ddn);

- unsigned long rt_set_this_node(const char *ddn, unsigned long node);

#define OWNER(node, task) \
        ((((unsigned long long)(node)) << 32) | (unsigned long)(task))

- RT_TASK *rt_find_asgn_stub(unsigned long long owner, int asgn);

- int rt_rel_stub(unsigned long long owner);

- int rt_waiting_return(unsigned long node, int port);

- int rt_sync_net_rpc(unsigned long node, int port);

- rt_get_net_rpc_ret(
	MBX *mbx, 
	unsigned long long *retval, 
	void *msg1, 
	int *msglen1, 
	void *msg2, 
	int *msglen2, 
	RTIME timeout, 
	int type
  );

What must be installed for distributed user space applications:
---------------------------------------------------------------

insmoded: rtai.o, rtai_shm.o, rtai_sched.o, rtai_lxrt.o, krtnet.o if "net_rpc"
          has not been made with "EMULATE_RTNET", 
	  net_rpc.o ThisNode="xxx.xxx.xxx.xxx"

process executed in background, as a kind of deamon: urtnet &
library: export LD_LIBRARY_PATH=lxrt/lib

It is important to insert krtnet.o first, if it is needed, followed by net_rpc.o
then run urtnet process in background.

At rmmod do not kill urtnet, it will happen at krtnet rmmod.

What must be installed for distributed kernel space applications:
-----------------------------------------------------------------

insmoded: rtai.o, rtai_shm.o, rtai_sched.o, lxrt/sched_ext/rtai_sched_ext.o,
	  krtnet.o if "net_rpc" has not been made with "EMULATE_RTNET", 
	  net_rpc.o ThisNode="xxx.xxx.xxx.xxx"

process executed in background, as a deamon: urtnet &

It is important to insert krtnet.o first, followed by net_rpc.o then run urtnet 
process in background.

At rmmod do not kill urtnet, it will happen at krtnet rmmod.
-----------------------------------------------------------------

Naturally you must care of using the complete path names needed to get to their
directories from your application subdir.

As usual ... comments and bugs fixes are welcomed.

