\section{The Vision} XORP stands for eXtensible Open Router Platform. It is called an {\it extensible} router platform for good reason. We believe that only by designing for extensibility from the outset can we simultaneously satisfy several different user groups: \begin{itemize} \vspace{-0.07in} \item Network researchers needing a platform for experimentation. \vspace{-0.07in} \item Network operators needing a low-cost stable routing platform on commodity hardware. \vspace{-0.07in} \item Network equipment vendors with special purpose hardware. \vspace{-0.07in} \item Network application writers looking for an open platform to support their applications. \end{itemize} \vspace{-0.05in} Only time will tell if we're right, but in this paper we'll try and explain where we hope to go from here. Of course we'll only get there with good luck and a lot of help from all of these communities. First we'll discuss the XORP architecture, then return to how it helps these diverse groups. \subsection{XORP Overview} XORP divides into two subsystems. The user-level subsystem consists of the routing protocols themselves, along with routing information bases and support processes. The kernel-level subsystem manages the forwarding path and provides APIs for the user-level to access. The goal is for almost all of the user-level code to be agnostic as to the details of the forwarding path. User-level XORP has a multi-process architecture, with one process per routing protocol, plus extra processes for management, configuration, and coordination. To enable extensibility we designed a novel inter-process communication mechanism, XORP Resource Locators (XRLs), for communication between these modules. %The lower level uses the Click modular router~\cite{click}, a %modular, extensible toolkit for packet processing on conventional PCs. %Further work will help this architecture span a %large range of hardware forwarding platforms, from commodity %PC hardware, through mid-range PC-based platforms %enhanced with intelligent network interfaces (such as Intel's %IXP1200~\cite{ixp1200,scoutixp}), to high-end hardware-based forwarding engines. %We may also support alternative forwarding paths, such as the %FreeBSD forwarding path with AltQ queuing extensions~\cite{altq} %or an alternative extensible forwarding path such as %Scout~\cite{pathrouter}. %Some forwarding path choices may influence the functionality available %for end users to extend or change. But for many aspects of research, such %as routing protocol experiments that don't %require access to the forwarding path, a conventional FreeBSD %lower level would suffice. \begin{floatingfigure}[l]{3.0in} %[h] \centerline{\psfig{figure=processes3.ps,width=3.0in}} \caption{XORP High-level Processes} \label{fig:processes} \vspace{-0.1in} \end{floatingfigure} Figure~\ref{fig:processes} shows a XORP router's user-level processes and forwarding engine. The shared user-level processes are the XORP architecture's most innovative feature. Four core processes are particularly worthy of comment. The \emph{router manager} process manages the router as a whole. It maintains configuration information; starts processes such as routing protocols, as required by the configuration; and restarts failed processes as necessary. The \emph{ipc finder} process stores mappings between application requests, such as ``What are this router's interfaces?'', and the particular IPC calls necessary to answer those requests. Think of it as an IPC redirector: when an application wishes to make an IPC call, it consults the finder to discover how to do it. Thus it is easy to change how application requests are handled at run-time. We can also trace XORP's internal communications by asking the finder to map abstract requests to sequences of IPCs, some for tracing and some for doing the work. %% XXX This is duptext from Eddie and Orion editing at the same time. %% Either this or the next paragraph should go. %The \emph{forwarding engine abstraction} process, or FEA, abstracts the %details of how the router's forwarding path is implemented. %Routing protocols talk to the FEA to install routes and discover properties %of interfaces, for example. %Interfaces to the FEA remain the same regardless of whether the forwarding %path is implemented in Click, in a conventional kernel, or even in external %hardware. %Again, XORP processes can bypass the FEA when required. The \emph{routing information base} process (RIB) receives routes from the routing processes, and arbitrates which routes should be propagated into the forwarding path, or redistributed to other routing processes. The forwarding path is managed by the \emph{forwarding engine abstraction} process (FEA). The FEA abstracts the details of how the forwarding path of the router is implemented and as a result, the routing processes are agnostic to whether the forwarding plane is Click \cite{click} based, a conventional UNIX kernel, or novel forwarding hardware The FEA manages the networking interfaces and forwarding table in the router, and provides information to routing processes about the interface properties and events occurring on interfaces, such as an interface being taken down. \subsection{Extensibility} XORP's design encourages the construction of useful interfaces through multi-process design. A routing protocol process, for example, must communicate with other processes to install routes and discover information about the router itself. Open inter-process interfaces, built in to the system from the beginning, form the basic source of user-level XORP's extensibility. % \begin{floatingfigure}[r]{3.3in} %[h] %\vspace{-0.05in} \centerline{\psfig{figure=xrl.ps,width=3.2in}} \caption{An example XRL} \label{fig:xrl} \vspace{-0.1in} \end{floatingfigure} \def\xrl#1{\textsf{\small #1}} Most inter-process communication within XORP takes place via XRLs. XRLs resemble the Web's URLs. They specify in human-readable form the type of IPC transport desired (the ``protocol''), the abstract name for the entity being communicated with, the method being invoked, and a list of named arguments. Unlike URLs, they can also specify the nature of the response expected. Figure \ref{fig:xrl} shows one of the XRLs for the forwarding engine abstraction (FEA) process in human readable form. The initial `\xrl{finder:}' tag specifies the protocol; in this case the actual protocol has not yet been resolved. The first time this XRL is called, the client contacts the finder, which redirects to a new XRL containing the actual protocol and parameters to be used to contact the current FEA. Subsequent communication bypasses the finder. %The XRL library, which all of our processes link against, takes an XRL %and performs argument marshaling, then it invokes the specified transport %mechanism, and handles the response or any %failure that might occur. Unlike many RPC or remote method invocation %mechanisms, XRLs don't try and hide from the programmer that %off-process communication is taking place. While this makes the %programmer's job harder at first, it is essential for robustness that %the programmer is made aware that the failure modes of IPC are different from those of %a local procedure call. %To help the programmer and improve performance, an IDL and a stub %generator exist, so most XRL clients never need to parse the human %readable form. One motivation for XRLs was to encapsulate existing protocols within our consistent IPC framework. For example, we might wish to run third-party software that uses SNMPv3 for configuration. To integrate this software into our XRL-based management framework, we might write an SNMP `protocol family' for the XRL client library. Then XORP processes could transparently interoperate with the third-party software via XRLs that start with `\xrl{snmp:}'. %XRLs are general enough to encompass simple %communication with the kernel via \texttt{ioctl}s, and even signaling %via \texttt{kill()}. At the present time, we have not had a need to %add this functionality, but should the need arise, our architecture %would support it. The current XRL library supports %XORP-specific protocols for remote procedure call, one layered over %TCP and the other over UDP, and a local procedure call mechanism for %intra-process communication. \subsubsection{XRL Example: Command-line Interface} One of the biggest issues faced by an {extensible} router is the integration of separately maintained components into a coherent system. %Consider the interaction between management mechanisms such %as a command-line interface (CLI) and a new routing protocol. The %author of each of the management processes has no knowledge of future %routing protocols. At the same time, the author of each routing %protocol has no knowledge of future management mechanisms. Our solution is for all management, including initial configuration, to take place using XRLs. To add support for a specific management mechanism, such as SNMP or a command-line interface, the protocol implementor writes simple text files that map management requests to XRL calls. These thin mapping files are easy enough to write that third parties might add them as new management interfaces become available. To get more concrete, our configuration manager has a strict hierarchy of configuration parameters, which is directly reflected in our command line interface (CLI). The configuration manager takes a directory of template files, which define the possible configurable parameters for each XORP routing protocol, and generates mappings of configuration parameters to XRL dispatches. The designer of a new routing protocol can simply supply a template file specifying the new functionality provided. The configuration manager can read the template file, discover the new functionality, and know how to communicate with the process. The new functionality is then immediately available through the CLI. For example, a fragment of a router configuration file, and the corresponding part of a template file might look like: %\vspace{-0.1in} \noindent\framebox[\textwidth][l]{ \begin{minipage}{6in} \begin{multicols}{2} \noindent \sf \small protocols ospf \{\\ \parindent 10pt \indent router-id: 128.16.64.1\\ \indent area 128.16.0.1 \{\\\ \parindent 20pt \indent interface xl0 \{\\ \parindent 30pt \indent hello-interval: 30\\ \parindent 20pt \vspace{-0.1in} \indent \}\\ \parindent 10pt \vspace{-0.1in} \indent \}\\ \} \eject \noindent \sf \small hello-interval: uint \{ \\ \parindent 10pt \indent \%set: xrl ``ospf/ospf/1.0/set\_hello\_interval?\\ \parindent 60pt \indent if:txt=\$(IFNAME)\&interval:i32=\$(VALUE)'';\\ \parindent 10pt \indent \%get: xrl ``ospf/ospf/1.0/hello\_interval?if:txt\\ \parindent 60pt \indent -$>$ interval:i32'';\\ \} \end{multicols} \end{minipage} } \subsection{Robustness} The routing and coordination processes in XORP run in user space on a regular UNIX operating system. Routing processes are protected from each other and can have their resources constrained according to administrative preference. Furthermore, routing processes can crash without affecting the kernel, forwarding plane, or each other. And if a routing protocol does crash, the RIB will remove its routes from the forwarding engine, and optionally inform the re-starting routing process of the routes it previously held. %Multiple processes are used in Zebra~\cite{zebra} and Cisco's proposed ENA %router operating system, but not by some of the larger %commercial vendors today. A significant aspect of robustness is security. One benefit of being forwarding-path agnostic is that we can abstract privileged operations, such as sending on a raw socket, into the FEA via XRLs. This allows us to run many routing protocols in a sandbox. They have no interaction with the outside world except through XRLs and packets, and so a vulnerability in a routing protocol is far more difficult to escalate into full control of the router. Untrusted code might even be run in a virtual machine such as provided by Xen\cite{xen}. %Robustness in the forwarding path is also important, but solutions %such as memory protection that work well in user-space are not %acceptable. In the Click forwarding path robustness comes from the %granularity and simplicity of Click's components. Each element is %small enough to be well understood and tested in isolation. And since %many of Click's components can be run and debugged in user-space, %confidence about their robustness can be attained before being used in %the kernel.