Package Once, Use Everywhere
============================

Cross-platform Unix software packaging with OpenPKG 
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

You might prefer Open Source software for its well-known advantages, but
sometimes regret the downside of its associated open and distributed
development when trying to apply it manually to your work environment.

To keep a work environment stable and secure, it's often necessary to
locate the latest version of an application on the Internet and collect
its most recent patches containing security and bug fixes. However,
there is often no single location for these files. After the search
finishes, a system administrator must build and install the new binaries
on every Unix box in the network, and might find that the application
needs slight tweaking on each of them. 

Then after a round of laborious build manipulation, it might not be
clear that the application will run as intended on each of the different
platforms. If the application is a daemon even more work awaits, because
most Unix flavors have their own method of starting and stopping
daemons.

If the previous operation succeeds after all, the system administrator
might be left to wonder why it is necessary to clutter up the system
with residual installation files and how do the vendors think the
residue should be removed? Also, if the C</var> file system later runs
out of disk space due to an overgrown application log, do the vendors
think that the responsibility lies with the system administrator to
implement log file rotation himself? Such redundant basic tasks can and
should be avoided.

What can a system administrator do when facing obstacles like these?
He might start by sticking with vendor packages only. Unfortunately,
there may be several different Unix variants to administer. Worse yet,
no existing vendor packaging facility supports multiple installation of
an application, a feature useful for multiple project coexistence on a
single machine and independent package testing. Another missing feature
regards the management of coexisting packaged and unpackaged software.

Finally, if the system administrator wants a software installation and
configuration guide for his vacation replacement, he might as well give
up. This is usually impossible, because every vendor packaging approach
is different and cannot be uniformly applied across all major Unix
platforms.

OpenPKG to the rescue 
---------------------

OpenPKG is a large Open Source software packaging project. 
The project began in November 2000, and has grown to be a collaborative
software development effort managed and maintained by many. The project
aims to create a modular and flexible Unix subsystem for cross-platform
software packaging and installation.

More specifically, the goals of OpenPKG stem from the historical problem
often faced in the daily operation of a datacenter environment. The
major Unix platforms in operation in datacenters include FreeBSD, Linux
and Solaris. To satisfy the demands of such a diverse Unix computing
environment, the installation and maintenance of software packages
across these platforms greatly benefits from a unified and consistent
approach.

OpenPKG is not limited to the three major platforms just mentioned,
however. OpenPKG has more ambitious goals and runs on most major Unix
platforms. To achieve such cross-platform portability, OpenPKG provides
a subsystem on top of the underlying Unix system as shown in Figure_1.
It covers every essential server software component from shells,
editors, compilers, and up to network daemons and add-on applications.
Hence, the intended target community consists of system administrators
faced with a large and diverse set of Unix servers.

Internally, OpenPKG leverages the existing packaging technology of
the popular RedHat Package Manager (RPM). However, the RPM software
included with OpenPKG is specially extended to become more unique and
self-contained. The nearly 1000 available OpenPKG packages are really
just RPM packages under the hood, but were developed from scratch in an
OpenPKG standard approach. The packages are clean and robust, because
they follow extremely strict style guidelines and environment requirements.

To meet these OpenPKG guidelines and standards, a package must be built
from pristine vendor sources in a non-privileged temporary environment.
It has to work under an arbitrary file system location, has to follow a
strict file system layout, and must be fully self-contained within its
OpenPKG instance. Furthermore, the package must be maximum independent
from external Unix facilities, has to install with a reasonable and
ready to go configuration, and has to use log file rotations and other
such administrative wonders.

These well specified package-building guidelines yield several benefits
to OpenPKG users. OpenPKG users can install an instance (the OpenPKG
subsystem and user-chosen packages) under any file system location,
and even install multiple such instances on a single Unix system.

Incidentally, the main OpenPKG project environment is hosted on a
machine with fourteen other ongoing software projects, each with their
own dedicated OpenPKG instance. To separately satisfy each project's
needs, the associated OpenPKG instance serves every required software
component from Postfix to BIND, and INN to Apache. Each project can
therefore run in its own isolated environment, much like on a virtual
machine -- but without any virtualization overhead.

Covering the whole OpenPKG package lifecycle 
--------------------------------------------

OpenPKG follows an approach of minimum OS intrusion and maximum
standalone presence. It tries hard to smooth out the differences between
the underlying vendor solutions. Sometimes this means that it has to
break with well known standards in order to provide a cleaner solution.
This is the most reasonable approach, and leads to a consistent
cross-platform solution.

A question often asked is, "why does OpenPKG use RPM as the underlying
packaging technology when other alternatives exist?" There are indeed
other similar packaging technologies available to projects like
OpenPKG, with the most prominent alternatives to RPM being the Debian
C<dpkg>/C<apt> combination, FreeBSD ports, and System V C<pkgadd>.
According to some, RPM may not be the greatest packaging system since
sliced bread. However, RPM along with its OpenPKG extensions is the only
solution covering the I<whole> package lifecycle in a fully consistent
way.

To elaborate, the OpenPKG package lifecycle starts with automated
tracking, fetching, unpacking, patching, and building the source package
from pristine vendor sources. It goes on to build the binary package
in an unprivileged environment, and finishes its life term with the
installation, upgrade, and uninstallation of the binary package on the
target OpenPKG instance. This all works in a self-contained environment,
and is driven by concise yet complete package specifications (RPM
C<.spec> files). So, to finally answer the question, OpenPKG adopts RPM
as its underlying packaging technology because no other can fulfill
these steep requirements.

It is important to note that OpenPKG is primarily about packaging,
not porting. A requirement of the OpenPKG packaging philosophy is
that the vendor software has to be inherently portable to begin with.
Minor platform porting issues are fixed by the OpenPKG packagers, but
fundamental changes are not considered. In fact, the main reason for
some platforms lacking full OpenPKG support is that the amount of
overhead in building software on them is not within reason.

It may surprise some that OpenPKG officially discourages the use of
binary packages, and only provides them for bootstrapping (development
tools not available) and emergency (tight time constraints) purposes.
Experience has shown that installing binary packages built from source
packages on the target machine outperforms other binary methods in
respect to security and robustness.

There are simply too many subtle differences between most build and
install systems that can influence the binary at run time and cause
trouble. Some important run time parameters, such as the maximum size
of shared memory segments, are compiled into the binary on the build
machine. Among many examples of such a run time build dependency is
a situation in which an Apache package is built with mod_ssl and OSSP mm.
The dependency details of such a combination are overwhelming, when
sorting out the run time parameters. To avoid such trouble, OpenPKG
believes that the only reasonable solution is to always start with
source packages.

Bootstrapping OpenPKG for the first time 
----------------------------------------

The OpenPKG bootstrapping process is wrapped into a tricky shell script
that when run, will create a new instance of OpenPKG. This process is as
self-contained as possible, and requires a minimum amount of operating
system support and tools to unpack and compile itself. In the best case,
the script will search the C<$PATH> for the development tools C<tar>,
C<make> and C<cc> and use them in its processing. If any of these tools
are missing, an alternative approach exists in which a shell script
containing binaries provides the missing tools. FIXME?

The first step in bootstrapping involves dedicating a unique
file system prefix to the instance along with user and
group ids. The generic bootstrap building script called
C<openpkg->I<version>C<->I<release>C<.src.sh> requires these arguments
and creates a platform specific bootstrap installation script named
C<openpkg->I<version>C<->I<release>C<.>I<arch>C<->I<os>C<->I<tag>C<.sh>.
When run, this script installs the OpenPKG instance under the specified
prefix with all files owned by the user and group (Figure_2).

This bootstrapping process links the OpenPKG instance with the
underlying Unix system in a very mild way with only a few anchor points.
Subsequent package installations do not touch the system at all, and if
OpenPKG itself is uninstalled, the anchor points vanish.

After creating a self-contained hierarchy the bootstrap process
registers itself as the C<openpkg> package, and can thus be upgraded
or treated just like any other package. To make upgrade an already
bootstrapped OpenPKG instance easier, C<.rpm> versions of the bootstrap
package are available as well.

A step by step example of a complete installation and uninstallation of
an OpenPKG instance with a RSYNC server package is given in Listing_4.
To understand the OpenPKG RPM commands used, see the quick reference in
Table_2.

OpenPKG file system layout 
--------------------------

Every file system standard sucks. OpenPKG's file system aims to suck
less (Figure_2). Basically, its package area resembles the traditional
layout found under C</usr> on most popular Unix systems. Additionally,
it contains its own OpenPKG RPM package management information in a sub-area for
purposes of self-containment plus a local area for adding unpackaged
components.

OpenPKG does break with tradition in one aspect of its file system
layout, however. It unconventionally uses a separate subdirectory of
I<prefix>C</etc/>, I<prefix>C</share/> and I<prefix>C</var/> for each
installed package. These subdirectories are easy to manage, because
each is named after its associated package. This provides for a better
structure than the usual mess of files, and every OpenPKG package
adheres to this layout scheme (even when requiring a lot of effort to
override the different vendor package intentions).

Looking again at the RSYNC example in Listing_4, you can conclude right
away that the RSYNC configuration is in I<prefix>C</etc/rsync/>, and it
logs to somewhere in I<prefix>C</var/rsync/>. Such ease of maintenance
makes backups easier, moving whole instances without hassle, and more.

Managing OpenPKG packages 
-------------------------

When building packages, the temporary files are placed into
subdirectories of I<prefix>C</RPM/> by default. A package builder can
obtain the necessary subdirectory access by either being a member of
the associated OpenPKG group, logging in under the management user
id of the OpenPKG instance, or logging in as C<root>. A carefully
written C<~/.rpmmacros> file can alternatively redirect the paths to a
specified location (see the default macros C<%_sourcedir>, C<%_specdir>,
C<%_builddir>, C<%_tmppath>, C<%_rpmdir>, C<%_srcrpmdir> in
I<prefix>C</etc/openpkg/rpmmacros>) and this way allow even an arbitrary
user to build packages.

To build a binary package I<pkg-bin> from a source package I<pkg-src>,
use C<openpkg rpm --rebuild >I<pkg-src>. OpenPKG RPM will read the
C<.spec> information of the I<pkg-src>, build the package based
on the information, and place the resulting binary package in
I<prefix>C</RPM/PKG/>I<pkg-bin>.

To finally install the binary package so that it becomes part of the
OpenPKG instance, just use C<openpkg rpm -Uvh >I<pkg-bin>. Strictly
speaking this upgrades the package. To OpenPKG RPM, installation is
nothing more than the special case of upgrading from nothing.

As a side note, some packages provide alternative build variants through
boolean variables named C<with_>I<name>. To determine which variables
are available (if any at all), run "C<openpkg rpm -qpi >I<pkg-src>C< | grep
with_>". To build a binary package using such variables, add C<--with >I<name>
or C<--without>I<name> to the C<openpkg rpm --rebuild> command to
override the default value.

OpenPKG RPM is very clever when it comes to keeping configuration files
during an upgrade, as shown in Table_1. An old configuration file
is kept if the system administrator stuck to default configuration
which remained the same, or if the configuration was changed but
coincidentally matches the default configuration of the new package. In
practice, a administrator-changed configuration must be reapplied in few
cases of package upgrade.

In any case, should a configuration file not be kept, OpenPKG RPM saves the
old configuration file with the extension C<.rpmsave> before saving
a new default in its place. This ensures that changes to a default
configuration can be recovered and reapplied so that an upgraded package
runs correctly. Should a new default configuration file replace an old
one that retains its original (but old) default, OpenPKG RPM will rename it
with the extension C<.rpmorig>.

To make this delightful mechanism work properly, the configuration files
of each package have to be explicitly tagged. OpenPKG packages all
follow this principle, which further contributes to OpenPKG's robust
nature. OpenPKG RPM does the intuitive right thing by making sure that
a changed configuration file is kept in place if possible and if not,
then preserves it for manual consideration and application.

Finally, after the installation of a package, you can query a lot
of its information. The command C<openpkg rpm -qi >I<pkg-name>
summarizes a single installed package, while C<openpkg rpm -qa> lists
the names of all installed packages. C<openpkg rpm -qlv >I<pkg-name>
lists all the files associated with a package and C<openpkg rpm -qf
>I<prefix>/I<path>/I<to>/I<file> reveals to which package the given file
belongs. You can even check a package's integrity using C<openpkg rpm -V
>I<pkg-name> to verify which files have been tampered with or somehow
munged. For more details on this refer to Table_2.

The OpenPKG run-command facility 
--------------------------------

You might have noticed that in the previous example installation of
RSYNC, the server was started using the command C<openpkg rc rsync
start>. The workhorse behind this simple statement is the powerful
OpenPKG run-command facility. Run-commands for every package are
conveniently named I<prefix>C</etc/rc.d/rc.>I<pkg-name>. What each
of them offers is the functionality of several shell script segments
encapsulated in a single file. The sections of a run-command file are
identified by left-aligned labels prefixed with 'C<%>'. Listing_2 shows
C<rc.rsync> as an example.

The C<openpkg rc> command takes I<pkg-name> as the first argument and one or
more section labels as additional arguments. The run command segments
corresponding with the desired section labels are then extracted from
the C<rc.>I<pkg-name> file and executed in the order given on the
command line. The reserved package name C<all> serves as a wildcard and
refers to all installed OpenPKG packages, causing the processing of all
run-command files in a specified order. In this case, the run-command
facility will order the run-command processing according to the priority
field (C<-p >I<number>) of the given section label in each run-command
file. Another popular field in a section label is C<-u >I<user>, which
directs the script code to execute with the privileges of I<user>.

Most sections in a run-command file have arbitrary labels intended to
be used as command line arguments to the run-command facility. However,
some sections have special meaning. The section labels of these are
reserved names used internally by the run-command facility. For example,
the C<%common> section functions as a library and contains script code
useful to some or all of the other sections. Its script code is run
before any other script code.

Just like its cousin, the C<%common> section, the C<%config> section
can appear only one time in each run-command file. It contains
variables used to configure the behavior of the other sections
residing in the same run-command file. This means that the logging
and enabling variables in a C<%config> section will only affect the
associated package, for example. Such variables can be overridden in
I<prefix>C</etc/rc.conf> in a per-hierarchy scope, however. Technically,
the run-command facility assembles a large script file from the
C<%config> section, the I<prefix>C</etc/rc.conf> file, the C<%common>
section, and finally the user-defined section given as an argument (in
that order). The fat script it then executed.

The sections C<%monthly>, C<%weekly>, C<%daily>, C<%hourly> and
C<%quarterly> also have special meaning, as the OpenPKG bootstrap
process sets up C<cron> jobs to execute them accordingly. Another label
often seen is C<%env>, which is intended to be used with the C<--eval>
option explained below.

Regarding configuration through variables, it should be mentioned
that the C<rc.>I<pkg-name> file is intentionally not tagged as a
configuration file and will be overwritten on updates with no questions
asked. The I<prefix>C</etc/rc.conf> file is tagged as a configuration
file and is intended for overriding variables.

With OpenPKG, all daemon packages are released with scripts that
recognize the value of a variable I<pkg-name>C<_enable> (default value
"C<yes>"). Setting this variable to "C<no>" disables all run-commands of
the daemon in question. As seen with the RSYNC server example, this can
be quite useful when installing a package just to get a client piece.
Should the server piece not be of interest, then a simple variable
shuts it off completely. Similarly, to disable the automatic startup
of all daemons in a hierarchy, just add a C<openpkg_rc_all="no"> to
I<prefix>C</etc/rc.conf>. In this case, daemons can still be started
manually. This feature may be of interest to system administrators
wanting control over daemons with finer granularity.

The OpenPKG run-command facility has many other interesting features.
Use C<openpkg rc --query >I<variable> to see the effective value of any
configured variable, or use C<openpkg rc --config> to see a complete list of
all available variables with their default and effective values. Last
but not least the run-command facility offers a very handy feature to
allow packages to extend the user shell environment. For instance the
bootstrap package C<openpkg> uses this to add the OpenPKG instance
into your C<PATH>, C<MANPATH>, C<INFOPATH>, etc. Just execute C<eval
`>I<prefix>C</bin/openpkg rc --eval openpkg env`> to perform this
environment extension for your current shell session.

OpenPKG RPM vs. RedHat RPM 
--------------------------

As mentioned, OpenPKG is based on a uniquely adjusted and extended
RPM-based packaging facility which allows for very concise and
clean package specifications and building of I<every> package in
an unprivileged environment. Is this any different than what the
RedHat, SuSE, or Mandrake implementations offer? To understand the
added value of the OpenPKG implementation, let's take as example the
OpenPKG packaging of the RSYNC program. The OpenPKG packaging consists
of three files: the OpenPKG RPM specification (C<rsync.spec>, Listing_1),
the run-commands (C<rc.rsync>, Listing_2), and the default daemon
configuration (C<rsync.conf>, Listing_3). In comparison with the
RPM-based RSYNC package of other vendors, the OpenPKG RPM-based package
is full featured yet very concise and clean. This is due to the use of
OpenPKG RPM extensions and strict style guidelines.

To offer more portable and concise shell scripting, OpenPKG RPM uses GNU
shtool. All manual installation and patching is done with the C<shtool>
command. A companion tool, C<rpmtool>, complements C<shtool> with
OpenPKG RPM and OS-specific features. The C<rpmtool> allows all OpenPKG
packages to generate their file list (C<%files>) on the fly and makes
the packaging information smaller. It reduces the required maintenance
when vendor version updates occur as well.

OpenPKG RPM additionally provides a set of local macros (C<%{l_xxx}>)
to abstract system specifics and also help in removing redundancy from
packaging specifications. For example, the C<%{l_prefix}> is the file
system I<prefix> of the associated OpenPKG instance. Using OpenPKG's
local macros offers a clear advantage, because packages no longer need
hard-coded path prefixes and can therefore be built for arbitrary
OpenPKG instances.

Macros exist for the most often used build variables. The C<%{l_cc}>
macro expands to either I<prefix>C</bin/cc> (in case the OpenPKG C<gcc>
package is installed) or defaults to just C<cc>. The same goes for
"C<%{l_cflags -O}>": it expands to the optimized C compiler flags. In
case C<gcc> is installed, it expands to "C<-O2 -pipe>". Otherwise,
it expands to just "C<-O>" by default. The variables C<%{l_make}> and
C<%{l_mflags}> work together in a similar way. If C<%{l_make}> points
to a known C<make> which supports parallel building and the underlying
system has more than one CPU, then "C<%{l_mflags -O}>" expands to the
necessary flags to leverage the system's multiple processing power. For
example, on a 2 CPU FreeBSD machine with BSD C<make>, "C<%{l_mflags
-O}>" expands to "C<-j4 -B>" while on a 4 CPU Linux machine with GNU C<make>,
"C<%{l_mflags -O}>" expands to "C<--no-print-directory -j8>".

Many packages also deploy a tricky OpenPKG-specific package build
option feature based on OpenPKG RPM's C<%define> macro and the C<Provides>
header: A package specification C<foo.spec> can contain zero or more
"C<%option> I<opt-name> I<opt-default-value>" lines which expand to both
a "C<%ifndef> I<opt-name> C<%define> I<opt-name> I<opt-default-value>
C<%endif>" construct and a "C<Provides: foo::>I<opt-name> =
I<opt-value>" header definition. First, this allows the package itself
to conditionally build with variations through the use of following
'C<%if "%{>I<opt-name>C<}" == ">I<opt-value>C<"> ... C<%endif>'
constructs inside the C<foo.spec> file. 

The compared effective option value is either I<opt-default-value>
or the I<opt-override-value> from a "C<--define '>I<opt-name>
I<opt-override-value>C<'>" or "C<--with >I<opt-name>" or or "C<--without
>I<opt-name>" command line options. Second, the resulting source RPM
can be queried with "C<openpkg rpm -qp --provides>" or "C<rpm -qpi>" in order
to lookup the provided options and their default values. The same can
be done for the binary RPM to lookup the options and their effective
values which were used to built the package. Third, another package
depending on C<foo> usually just uses "[C<Build>]C<PreReq:> ... C<foo>".
In OpenPKG it also can depend on a particular build variation of C<foo>
by using "[C<Build>]C<PreReq:> ... C<foo>, C<foo::>I<opt-name>C< =
>I<opt-value>". The consequent of this functionality provides both a
clean, precise and flexible packaging.

Additionally, all OpenPKG packages follow I<exactly> the same style
as the RSYNC example (see Listing_1, Listing_2, and Listing_3). The
header order, indentation, etc. is fully standardized, and allows developers
to easily query and even semi-automatically edit package information
directly from the source. Incidentally, the indices on the OpenPKG FTP
server and the OpenPKG release engineering procedures are auto generated
by exploiting this standard scheme.

I<Every> OpenPKG package is able to build in an unprivileged (non-C<root>
user) environment and with read-only access to an OpenPKG instance.
This allows safe (no development system intrusion) and precise (no
trashed or missing files) packaging. Such security and precision
is achieved by consistently using the C<BuildRoot> feature of RPM
for I<all> packages. In short, this means that when rolling a binary
package the software is redirected to install into a shadow area
(I<prefix>C</RPM/TMP/>I<pkg-name>C<-root/>I<prefix>). The package is
then made from the shadow area just as if it were located in the real
file system location (I<prefix>). This important improvement to the
standard RPM behavior may sound trivial and easy to achieve, but is
actually one of the trickiest steps in packaging software for OpenPKG.

Sometimes (as with RSYNC), it is just a matter of overriding variables
(C<prefix> in the example) on the "C<make install>" step. Other times the
solution is more involved. For some OpenPKG packages it takes a lot of
effort to find a reasonable way to redirect the vendor installation to
the C<BuildRoot> location, but the extra effort is always worthwhile and
results in safer and more precise packaging.

Finally, OpenPKG's RPM implementation provides proxy packages, a tricky
and appealing mechanism for reusing the packages of a master OpenPKG
instance. Proxy packages can reside in multiple slave OpenPKG instances,
and allow the system administrator to avoid redundant building and
maintaining of the same software package in multiple OpenPKG instances.
For example, C<gcc> is typically required by many packages at build
time. A C<gcc> OpenPKG package is usually needed in every OpenPKG
instance.

A savvy system administrator will install a single C<gcc> package
in a master OpenPKG instance and then only proxy packages (pointing
to the real C<gcc>) in the other OpenPKG instances by running
I<slave-prefix>C</bin/openpkg makeproxy> on the C<gcc> binary RPM of
the master instance. OpenPKG's RPM will then produce a binary RPM
package for the slace instance, containing a shadow tree resembling the
contents of the master instance. The shadow tree is technically nothing
more than symbolic links to the master (non-proxy) package's files
and directories. This mechanism can save a lot of time and storage,
however it should only be applied to packages with global configuration
dependencies only or with no configuration dependencies at all.

Integrating unpackaged software 
-------------------------------

No matter how many packages OpenPKG provides, the world will always have
appealing yet unpackaged software. Ambitious system administrators can
package the software themselves for local purposes and even contribute
such new packages to the OpenPKG community. Alternatively, the C<local>
subdirectory of an OpenPKG instance exists for the purpose of containing
unpackaged software, and can be instrumental in integrating a base of
OpenPKG packages with other unpackaged software in an easy to maintain
way. OpenPKG also provides a corresponding C<lsync> tool to aid such
integration.

To integrate unpackaged software into an OpenPKG instance, each
unpackaged software component may be installed into the C<bin>,
C<sbin>, C<man>, C<info>, C<include> and C<lib> subdirectories of
I<prefix>C</local/PKG/>I<pkg-name>C</> and then virtually linked into
the corresponding top-level directories under I<prefix>C</local/> by
running I<prefix>C</bin/openpkg lsync>. This easy to administer strategy
leads to a very clean and maintainable OpenPKG instance, even with
its new coexisting unpackaged software in I<prefix>C</local/>.
This especially makes it easy to uninstall a package. Just remove
I<prefix>C</local/PKG/>I<pkg-name>C</> with all its contents and run
C<openpkg lsync> again.

This strategy even allows for installation of different
versions of the same software. Just install into
I<prefix>C</local/PKG/>I<pkg-name>C<->I<version>C</> and add a symbolic
link pointing from I<prefix>C</local/PKG/>I<pkg-name>C</> to this
directory. This works, because C<openpkg lsync> skips subdirectories of
I<prefix>C</local/PKG/> with version numbers attached. To upgrade an
older C<foo-0.7.41> to C<foo-0.7.42>, just repeat the installation in
the same way, altering the symlink I<prefix>C</local/PKG/foo> to point
to C<foo-0.7.42> instead and running C<openpkg lsync> again. C<openpkg
lsync> will automatically update symlinks, creating new links if
required and removing outdated dangling ones. As might be guessed, it is
just as easy to go back to the old version if the new one keeps dumping
core or something like that. For an example of such multiple unpackaged
software installation see Figure_3.

OpenPKG release engineering 
---------------------------

A carefully crafted release process is part of the OpenPKG project, and
the fruits of the whole project are available to the public according
to Open Source standards. All sources (package specifications, source
patches, website sources, the handbook, this article, etc) are located
in a publicly readable central CVS repository, which can be browsed
anonymously by conventional C<cvs> commands or through the website
for added convenience. Additionally, all developer commits to this
repository are tracked and summarized with postings to public mailing
lists and public newsgroups. People can easily follow all developments
by subscribing to the list or reading the newsgroup.

For stability and to reduce conflicts between development milestones,
OpenPKG has three release branches (which technically directly
map to CVS branches.) These are OpenPKG-SOLID, OpenPKG-STABLE and
OpenPKG-CURRENT. OpenPKG-SOLID is the security update branch of the last
public OpenPKG release. OpenPKG-STABLE is the stable branch from whose
contents the next public release is made. OpenPKG-CURRENT is the current
state of the development branch, and contains packages of beta-grade
stability. 

In any case, the branch from which a package was built can easily
be determined by the OpenPKG RPM file name, because they follow a
consistent naming scheme: I<pkg-name>C<->I<version>C<->I<YYYYMMDD>
(for CURRENT), I<pkg-name>C<->I<version>C<->I<N>C<.>I<YYYYMMDD> (for
I<N>-STABLE), I<pkg-name>C<->I<version>C<->I<N>C<.>I<M>C<.>I<X> (for
I<N.M.X>-SOLID). Once such a source OpenPKG RPM package is built, the
new binary OpenPKG RPM package filename contains even more information,
such as operating system, hardware, and the OpenPKG instance tag.

The OpenPKG developer team is extremely very fast in keeping
OpenPKG-CURRENT packages up to date and in sync with the latest upstream
vendor versions. This is due to the fact that the versions of all
externally available vendor sources are fully automatically tracked on a
bi-daily basis. An OpenPKG package for a new vendor software version is
often available before the software is even announced on I<Freshmeat.net>.

Finally, OpenPKG takes security very seriously. Experience has shown
that "security through obscurity" does not work, and that public
disclosure leads to quicker and better solutions to security problems.
In that vein, OpenPKG tries to release fixed packages as fast as
possible after a vulnerability was discovered. The OpenPKG security
release and advisory process notifies the community by publishing
official security advisories in the security section of the website and
on the mailing lists.

Conclusion 
----------

OpenPKG is an Open Source software project existing
since November 2000. The implementation relies on RPM 4 for
its basic packaging mechanism, but offers more than RPM alone. To
meet its goal of becoming a modular and flexible Unix subsystem for
cross-platform software packaging and installation, OpenPKG includes
tricky bootstrapping logic that installs a customized implementation of
RPM 4 on any of the supported target platforms.

OpenPKG is in production use since April 2001 in datacenter
environments of large ISPs. Since its public release in January 2002,
OpenPKG users have profited from an increase from 220 to nearly 1000 software
packages. The project is continuously improved by a diverse team of
developers who also daily update and add packages.

The base of OpenPKG software packages is expected to increase even more,
partly due to the ease of writing specifications and building packages.
Most OpenPKG users find it deceivingly simple to build a basic package.
New users interested in such packaging can use the RSYNC example in this
article as a blueprint. Accordingly, package contributions are always
appreciated by the members of the OpenPKG project and the members
of the OpenPKG Foundation.

References
----------

OpenPKG:
U<http://www.openpkg.org/>
U<ftp://ftp.openpkg.org/>

OpenPKG Community Forums:
U<mailto:openpkg-users@openpkg.org>
U<mailto:openpkg-dev@openpkg.org>
U<nntp://news.openpkg.org/openpkg.users>
U<nntp://news.openpkg.org/openpkg.dev>

About the authors 
-----------------

Ralf S. Engelschall is a computer scientist and Open Source software
hacker. He is the author of well-known software like Apache mod_ssl,
GNU Pth, and GNU Shtool and the founder of Open Source software
projects like OpenSSL, OpenPKG, and OSSP.
He can be contacted at: rse@engelschall.com

Thomas Lotterer is a network professional and consultant working as a
Unix software developer. He gained experience in cross-platform system
integration and software distribution by working previously as a system
administrator and technical trainer. Today, Thomas works actively on the
OpenPKG and OSSP projects.
He can be contacted at: thomas@lotterer.net

Christoph Schug is a senior Unix system administrator. His revolutionary
ideas and visions often result in additional lines in Ralf's TODO list.
When not in the office, Christoph might be found in the Alps steering
the screaming and smoking tires of his Miata MX-5 roadster.
He can be contacted at: chris@schug.net

Michael Schloh von Bennewitz is a software engineer. He was an active
contributor to both the OpenPKG and OSSP projects.
He can be contacted at: michael@schloh.com

(Last Modified: 2006-09-26)

______________________________________________________________________________


Figure_1: The OpenPKG subsystem principle
------------------------------------------------------------------------------
article.principle.fig
------------------------------------------------------------------------------

Figure_2: The OpenPKG file system layout
------------------------------------------------------------------------------
article.layout.fig
------------------------------------------------------------------------------

Figure_3: File system layout of non-OpenPKG software installed
------------------------------------------------------------------------------
article.layout-local.fig
------------------------------------------------------------------------------

Listing_1: OpenPKG packaging for rsync (rsync.spec: OpenPKG RPM specification)
------------------------------------------------------------------------------
##
##  rsync.spec -- OpenPKG RPM Package Specification
##  Copyright (c) 2000-2006 OpenPKG Foundation e.V. <http://openpkg.net/>
##  Copyright (c) 2000-2006 Ralf S. Engelschall <http://engelschall.com/>
##
##  Permission to use, copy, modify, and distribute this software for
##  any purpose with or without fee is hereby granted, provided that
##  the above copyright notice and this permission notice appear in all
##  copies.
##
##  THIS SOFTWARE IS PROVIDED ``AS IS'' AND ANY EXPRESSED OR IMPLIED
##  WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF
##  MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED.
##  IN NO EVENT SHALL THE AUTHORS AND COPYRIGHT HOLDERS AND THEIR
##  CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
##  SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
##  LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF
##  USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND
##  ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
##  OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT
##  OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
##  SUCH DAMAGE.
##

#   package information
Name:         rsync
Summary:      Remote Filesystem Synchronization
URL:          http://rsync.samba.org/
Vendor:       Andrew Tridgell & Paul Mackerras
Packager:     OpenPKG
Distribution: OpenPKG
Class:        CORE
Group:        Filesystem
License:      GPL
Version:      2.6.8
Release:      20060904

#   package options
%option       with_lastmatch  no
%option       with_timelimit  no

#   list of sources
Source0:      http://rsync.samba.org/ftp/rsync/rsync-%{version}.tar.gz
Source1:      rsync.conf
Source2:      rc.rsync
Patch0:       rsync.patch

#   build information
Prefix:       %{l_prefix}
BuildRoot:    %{l_buildroot}
BuildPreReq:  OpenPKG, openpkg >= 20060823
PreReq:       OpenPKG, openpkg >= 20060823
AutoReq:      no
AutoReqProv:  no

%description
    Rsync is a replacement for rcp(1) that has many more features.
    Rsync uses the "rsync algorithm" which provides a very fast method
    for bringing remote files into sync. It does this by sending just
    the differences in the files across the link, without requiring
    that both sets of files are present at one of the ends of the link
    beforehand. At first glance this may seem impossible because the
    calculation of differences between two files normally requires local
    access to both files.

%track
    prog rsync = {
        version   = %{version}
        url       = http://rsync.samba.org/ftp/rsync/
        regex     = rsync-(\d+\.\d+\.\d+)\.tar\.gz
    }

%prep
    #   unpack vendor sources
    %setup -q
    %patch -p0
%if "%{with_lastmatch}" == "yes"
    %{l_patch} -p1 <patches/last-match.diff
%endif
%if "%{with_timelimit}" == "yes"
    %{l_patch} -p1 <patches/time-limit.diff
%endif

%build
    #   configure vendor sources
    CC="%{l_cc}" \
    CFLAGS="%{l_cflags -O}" \
    ./configure \
        --prefix=%{l_prefix} \
        --with-rsyncd-conf=%{l_prefix}/etc/rsync/rsync.conf \
        --disable-debug \
        --disable-locale \
        --with-included-popt \
        --with-rsh=ssh

    #   build vendor sources
    %{l_make} %{l_mflags -O}

%install
    #   perform vendor installation
    rm -rf $RPM_BUILD_ROOT
    %{l_make} %{l_mflags} install \
        prefix=$RPM_BUILD_ROOT%{l_prefix} \
        exec_prefix=$RPM_BUILD_ROOT%{l_prefix}

    #   post-adjust vendor installation
    strip $RPM_BUILD_ROOT%{l_prefix}/bin/* >/dev/null 2>&1 || true
    mv  $RPM_BUILD_ROOT%{l_prefix}/man/man5/rsyncd.conf.5 \
        $RPM_BUILD_ROOT%{l_prefix}/man/man5/rsync.conf.5

    #   provide default (deamon) configuration
    %{l_shtool} mkdir -f -p -m 755 \
        $RPM_BUILD_ROOT%{l_prefix}/etc/rsync
    %{l_shtool} install -c -m 644 %{l_value -s -a} \
        -e 's;@version@;%{version};g' \
        %{SOURCE rsync.conf} $RPM_BUILD_ROOT%{l_prefix}/etc/rsync/
    %{l_shtool} install -c -m 644 /dev/null \
        $RPM_BUILD_ROOT%{l_prefix}/etc/rsync/rsync.passwd
    %{l_shtool} mkdir -f -p -m 755 \
        $RPM_BUILD_ROOT%{l_prefix}/var/rsync

    #   provide run-command script
    %{l_shtool} mkdir -f -p -m 755 \
        $RPM_BUILD_ROOT%{l_prefix}/etc/rc.d
    %{l_shtool} install -c -m 755 %{l_value -s -a} \
        %{SOURCE rc.rsync} $RPM_BUILD_ROOT%{l_prefix}/etc/rc.d/

    #   determine binary package files
    %{l_rpmtool} files -v -ofiles -r$RPM_BUILD_ROOT \
        %{l_files_std} '%config %{l_prefix}/etc/rsync/rsync.*'

%files -f files

%clean
    rm -rf $RPM_BUILD_ROOT

%post
    #   after upgrade, restart service
    [ $1 -eq 2 ] || exit 0
    eval `%{l_rc} rsync status 2>/dev/null`
    [ ".$rsync_active" = .yes ] && %{l_rc} rsync restart
    exit 0

%preun
    #   before erase, stop service and remove log files
    [ $1 -eq 0 ] || exit 0
    %{l_rc} rsync stop 2>/dev/null
    rm -f $RPM_INSTALL_PREFIX/var/rsync/rsync.log* >/dev/null 2>&1 || true
    exit 0
------------------------------------------------------------------------------


Listing_2: OpenPKG packaging for rsync (rc.rsync: run-commands)
------------------------------------------------------------------------------
#!@l_prefix@/bin/openpkg rc
##
##  rc.rsync -- Run-Commands
##

%config
    rsync_enable="$openpkg_rc_def"
    rsync_daemon="no"
    rsync_flags=""
    rsync_nicelevel="10"
    rsync_log_prolog="true"
    rsync_log_epilog="true"
    rsync_log_numfiles="10"
    rsync_log_minsize="1M"
    rsync_log_complevel="9"

%common
    rsync_cfgfile="@l_prefix@/etc/rsync/rsync.conf"
    rsync_logfile="@l_prefix@/var/rsync/rsync.log"
    rsync_pidfile="@l_prefix@/var/rsync/rsync.pid"
    rsync_signal () {
        [ -f $rsync_pidfile ] && kill -$1 `cat $rsync_pidfile`
    }

%status -u @l_susr@ -o
    rsync_usable="unknown"
    rsync_active="no"
    rcService rsync enable yes && \
        rsync_signal 0 && rsync_active="yes"
    echo "rsync_enable=\"$rsync_enable\""
    echo "rsync_usable=\"$rsync_usable\""
    echo "rsync_active=\"$rsync_active\""

%start -u @l_susr@
    rcService rsync enable yes || exit 0
    rcService rsync active yes && exit 0
    rcVarIsYes rsync_daemon || exit 0
    nice -n $rsync_nicelevel @l_prefix@/bin/rsync \
         $rsync_flags --daemon --config=$rsync_cfgfile
    sleep 2

%stop -u @l_susr@
    rcService rsync enable yes || exit 0
    rcService rsync active no  && exit 0
    rcVarIsYes rsync_daemon || exit 0
    rsync_signal TERM
    sleep 2

%restart -u @l_susr@
    rcService rsync enable yes || exit 0
    rcService rsync active no  && exit 0
    rcVarIsYes rsync_daemon || exit 0
    rc rsync stop start

%daily -u @l_susr@
    rcService rsync enable yes || exit 0
    rcVarIsYes rsync_daemon || exit 0
    shtool rotate -f \
        -n ${rsync_log_numfiles} -s ${rsync_log_minsize} -d \
        -z ${rsync_log_complevel} -m 644 -o @l_susr@ -g @l_mgrp@ \
        -P "${rsync_log_prolog}" \
        -E "${rsync_log_epilog}; rc rsync restart" \
        $rsync_logfile
------------------------------------------------------------------------------

Listing_3: OpenPKG packaging for rsync (rsync.conf: default configuration)
------------------------------------------------------------------------------
##
##  rsync.conf -- OpenPKG rsync configuration (daemon mode only)
##

secrets file       = @l_prefix@/etc/rsync/rsync.passwd
lock file          = @l_prefix@/var/rsync/rsync.lck
pid file           = @l_prefix@/var/rsync/rsync.pid
log file           = @l_prefix@/var/rsync/rsync.log
    
address            = 127.0.0.1
port               = 873

max connections    = 20
timeout            = 60
strict modes       = yes
use chroot         = yes
read only          = yes
ignore nonreadable = yes
transfer logging   = yes
log format         = "%o %h [%a] %m (%u) %f %l"
dont compress      = *.bz2 *.gz *.zip *.z *.rpm *.deb *.tgz *.iso
list               = no
uid                = @l_nusr@
gid                = @l_ngrp@

[pub]
comment            = Public area of @l_prefix@ OpenPKG instance
path               = @l_prefix@/pub/
exclude            = .*
read only          = yes
list               = yes
------------------------------------------------------------------------------

Listing_4: Bootstrapping OpenPKG instance under /openpkg
------------------------------------------------------------------------------
1. manually download the "openpkg" package

user$ cd /tmp
user$ ftp ftp.openpkg.org
Connected to ftp.openpkg.org.
220 ftp.openpkg.org OpenPKG Anonymous FTP Server ready.
Name (ftp.openpkg.org): anonymous
331 Anonymous login ok, send your email address as your password.
Password: you@example.com
230- [...] Welcome to OpenPKG.org! [...]
230 Anonymous access granted, restrictions apply.
ftp> bin
200 Type set to I.
ftp> cd stable/2.20060622/UPD
ftp> get openpkg-2.20060815-2.20060815.src.sh
ftp> bye
221 Goodbye.

2. build and install "openpkg" package to bootstrap instance

user$ sh openpkg-2.20060615-2.20060615.src.sh \
      --prefix=/openpkg --user=openpkg --group=openpkg
user$ su -
root# sh openpkg-2.20060615-2.20060615.*-*-*.sh
root$ rm -f openpkg-2.20060615-2.20060615.*

3. build, install and start "rsync" package

root# /openpkg/bin/openpkg rpm --rebuild \
      ftp://ftp.openpkg.org/stable/2.20060622/SRC/rsync-2.6.8-2.20060622.src.rpm
root# /openpkg/bin/openpkg rpm -Uvh /openpkg/RPM/PKG/rsync-2.6.8-2.20060622.*-*-*.rpm
root# /openpkg/bin/openpkg rc rsync start
root# exit

4. check that RSYNC service is working

user$ /openpkg/bin/rsync rsync://localhost/
pub             Public area of /openpkg OpenPKG instance

5. cleanup, shutdown and deinstall everything again

user$ su -
root$ rm -f /openpkg/RPM/PKG/*
root# /openpkg/bin/openpkg rc rsync stop
root# /openpkg/bin/rpm -e rsync openpkg
root# exit
------------------------------------------------------------------------------

Listing_5: Installation of non-OpenPKG software
------------------------------------------------------------------------------
user$ su -
root# mkdir -p /openpkg/local/PKG/foo-0.7.42/SRC
root# cd /openpkg/local/PKG/foo-0.7.42/SRC
root# wget ftp://ftp.example.com/foo-0.7.42.tar.gz
root# gunzip <foo-0.7.42.tar.gz | tar xf -
root# cd foo-0.7.42
root# ./configure --prefix=/openpkg/local/PKG/foo-0.7.42
root# make
root# make install
root# cd /openpkg/local/PKG
root# ln -s foo-0.7.42 foo
root# /openpkg/bin/openpkg lsync
------------------------------------------------------------------------------

Table_1: RPM handling of configuration files
------------------------------------------------------------------------------
old in   old on     new in   new on 
package  filesystem package  filesystem approach
======== ========== ======== ========== ===========================================
X        X          X        X          keep old config
X        X'         X'       X'         keep old config
X        X'         X        X'         keep old config
X        X          Y        Y          install new config
X        X'         Y        Y          install new config, preserve X' as .rpmsave
-        X          Y        Y          install new config, preserve X  as .rpmorig
------------------------------------------------------------------------------

Table_2: OpenPKG Administration Commands (Quick Reference)
------------------------------------------------------------------------------
$ <prefix>/bin/openpkg rpm -Uvh <pkg-src>                unpack source package
$ <prefix>/bin/openpkg rpm -bb <pkg-spec>                build binary package from unpacked sources
$ <prefix>/bin/openpkg rpm --clean --rmsource <pkg-spec> cleanup after building binary package
$ <prefix>/bin/openpkg rpm --rebuild <pkg-src>           build binary package from source package

# <prefix>/bin/openpkg rpm -Uvh <pkg-bin>                install/upgrade package
# <prefix>/bin/openpkg rpm -Uvh --nodeps <pkg-bin>       install/upgrade package (ignore dependencies)
# <prefix>/bin/openpkg rpm -Uvh --force <pkg-bin>        install/upgrade package (no checks at all)
# <prefix>/bin/openpkg rpm -Uvh --oldpackage <pkg-bin>   install/downgrade package
# <prefix>/bin/openpkg rpm -Fvh <pkg-bin>                upgrade package (if already installed only)

# <prefix>/bin/openpkg rpm -e <pkg-name>                 uninstall/erase package
# <prefix>/bin/openpkg rpm -e --nodeps <pkg-name>        uninstall/erase package (ignore dependencies)

$ <prefix>/bin/openpkg rpm -qpi <pkg-bin>                list information about binary package
$ <prefix>/bin/openpkg rpm -qplv <pkg-bin>               list all files a binary package will install
$ <prefix>/bin/openpkg rpm -qa                           list all installed packages and their versions
$ <prefix>/bin/openpkg rpm -qi <name>                    list information about an installed package
$ <prefix>/bin/openpkg rpm -qlv <name>                   list all files a package has installed
$ <prefix>/bin/openpkg rpm -qf <prefix>/...              list package owning a particular file
$ <prefix>/bin/openpkg rpm -qc <name>                    list all config files of an installed package

$ <prefix>/bin/openpkg rpm --checksig <pkg-rpm>          verify integrity and origin of a package
$ <prefix>/bin/openpkg rpm -Va                           check the integrity of all packages
$ <prefix>/bin/openpkg rpm -V <name>                     check the integrity of a particular package
$ <prefix>/bin/openpkg rpm -Va --nofiles                 check all package dependencies only

$ <prefix>/bin/openpkg rc --config                       show all %config variables and values
$ <prefix>/bin/openpkg rc --query <variable>             query a particular %config variable
# <prefix>/bin/openpkg rc <pkg_name> start               start a particular package
# <prefix>/bin/openpkg rc <pkg_name> stop                stop  a particular package
# <prefix>/bin/openpkg rc <pkg_name> stop start          restart a particular package
# <prefix>/bin/openpkg rc all stop start                 restart all packages
$ eval `<prefix>/bin/openpkg rc --eval all env`          setup shell environment of all packages 

Legend: #          command has to be executed with     root privileges
        $          command should be executed withough root privileges
        <prefix>   e.g. /openpkg
        <pkg-name> e.g. rsync
        <pkg-spec> e.g. rsync.spec
        <pkg-rpm>  e.g. rsync-2.6.8-2.20060622.*.rpm
        <pkg-src>  e.g. rsync-2.6.8-2.20060622.src.rpm
        <pkg-bin>  e.g. rsync-2.6.8-2.20060622.ix86-freebsd6.1-openpkg.rpm
------------------------------------------------------------------------------

Table_4: OpenPKG summary 
------------------------------------------------------------------------------
- Open source software
- Cross-platform solution for FreeBSD, Linux, Solaris and others
- Subsystem on top of underlying Unix system
- Minimal OS intrusion
- Fully self-contained with minimum OS dependencies
- Primarily targeted at server environments
- Based on a customized RPM 4
- Support for entire package lifecycle
- Build and run-time dependency control
- Builds software from pristine vendor sources
- All OpenPKG RPM package specifications written from scratch
- Clear and concise specifications follow a strict style
- Supports integration of unpackaged software also
- Tricky bootstrapping with no requirement for a preinstalled RPM
- Strict and consistent file system layout
- All packages installed with reasonable preconfiguration
- Users benefits from packager's knowledge
- Cryptographically signed packages
- Package integrity verification
- Support for building packages as unprivileged user
- Installation under arbitrary file system prefix
- Allows multiple instances on the same host
- Fully uninstalls packages
- Powerful package queries
- Flexible run-command facility
- Mature technology in datacenter production use since April 2001
- Variety of nearly 1000 packages available
------------------------------------------------------------------------------