OpenInteract Administrator's Guide

This document goes over issues concerning system administrators, including mod_perl/Apache compilation and configuration, tips for running multiple websites under one mod_perl server, package configuration, logging, error reporting and caching.

While you're reading, it would be handy to have within reach the OpenInteract Glossary to lookup certain terms (like 'website', 'handler', 'package', etc.).

oi_manage

Distributed with OpenInteract is the script oi_manage. If you installed OpenInteract via the normal Perl mechanisms, you should have it in your /usr/bin or /usr/local/bin directory so it's always accessible. (You might also ensure that /usr/local/bin is in your path and your users' paths, or that you create a symlink from /usr/bin/oi_manage to /usr/local/bin/oi_manage.)

This script takes care of many actions for you. Here's a list and a brief description of each:


 install            - Install OpenInteract
 initial_packages   - List packages marked as 'initial'
 list_packages      - List packages installed to app or base
 create_skeleton    - Create a skeleton package for development
 export_package     - Export package(s) to distribution file(s)
 check_package      - Ensure that package passes initial inspection
 install_package    - Install package to the base
 create_website     - Create a new website
 list_actions       - List the actions available in a website
 test_db            - Test database settings in 'server.perl'
 apply_package      - Install a package from base to website
 remove_package     - Remove a package from a website
 upgrade_package    - Upgrade a website package
 install_sql        - Install the SQL for packages
 input_template     - Input package templates to the database
 dump_template      - Dump package templates to the filesystem
 remove_template    - Remove package templates from the database
    

You can read an online text version of the documentation for oi_manage For the most up-to-date information, do a perldoc oi_manage from the command line to view the extensive documentation.

OpenInteract

This section reviews installation, layout and gives a very broad overview of the packaging system.

Shortcuts

There are two environment variables you can use too make your typing life easier:

We encourage you to use these as much as possible. However, we continue to use the long form in the examples below.

Installation

Installation of OpenInteract is very simple:


 cd /path/where/I/unpacked/distribution
 oi_manage --base_dir=/path/to/install install

Easy! Read the INSTALL and INSTALL.website files that were packaged with the OpenInteract distribution for more details.

Layout

OpenInteract is composed of two pieces: the installation and the websites. The installation is what you create when you run the oi_manage command above.

And you refer to it when you create new websites:


 oi_manage --base_dir=/path/to/install --website_dir=/path/to/website \
           --website_name=MySite create_website

Thereafter, the website keeps the information for the base installation directory in its conf/base.conf file described below. So you shouldn't need to refer to the installation directory too often. (Developers need to know it when they are developing packages, but that's a separate issue.)

Packages: Installing and Upgrading

So OpenInteract exists in two parts: the installation and the various websites that use the installation. When users who run the websites want to add new functionality by installing a new package or upgrading an existing one, they must first add the package to the OpenInteract installation and then apply it to the website.

The OpenInteract installation maintains a copy of every version of every package ever installed to the system. This might seem backwards, but it ensures that different websites can use and continue to use different versions of the same packages. This should allow you to accommodate different types of users: bleeding-edge developers who always play with new versions of packages and stay-the-line website maintainers who just want everything to work the same day after day.

Fortunately, the oi_manage script makes this whole process easy for us:

Install a new package to the installation:


 oi_manage --package_file=/path/to/file.tar.gz \
           --base_dir=/path/to/install install_package

Install a new version of an existing package to the installation:


 oi_manage --package_file=/path/to/file.tar.gz \
           --base_dir=/path/to/install install_package

Hey, those two were the same thing! It's true. Since the installation maintains a copy of every version of every package, it doesn't actually 'upgrade' a package in the installation.

This just emphasizes the main point: when you install a package to the OpenInteract installation, you're simply making it available for the websites that are using this installation.

Apply an installed package to a website:


 oi_manage --package=newpackage --website_dir=/path/to/website \
           apply_package

Upgrade to a new version installed package for a website:


 oi_manage --package=newpackage --website_dir=/path/to/website \
           upgrade_package

Ah, now those two commands were different. This shows that a website can only have one version of a particular package installed. If you try to apply_package for a package that has already been applied to a website, you'll get an error and no action will be taken. In this case you need to use upgrade_package.

The docs for oi_manage discuss this next point but it's worth repeating: when you use upgrade_package, you merely update the registry information for the website. You also install new data and structure, object configuration, templates, handlers and everything else.

However, the system does not remove the old files. The system does not know if you want to keep the changes you've made or discard them, so it simply keeps the old directory around and allows you to copy information back and forth as you need. But this old directory is not used by the system anymore, and if you make changes to files in the old package directory they will not be reflected in the website.

Configuration File Formats

We generally use one of three formats. If you create new configuration files, try to stick to these naming schemes.

.conf

Typical configuration file format: information separated into key-value pairs (separated by whitespace), blank lines and lines that begin with a comment (#) are skipped.

Example:


  MyValue              emmet otter
  HerValue   fraggle rock
  TheirValue  jughandle
   # ringo   starr

Parsing this would return a hashref:

 { 
   MyValue    => 'emmet otter',
   HerValue   => 'fraggle rock',
   TheirValue => 'jughandle'
 }

.dat

Very simple: one item per line. Blank lines and lines beginning with a comment (#) are skipped entirely.

Example:

 
 MyClass
 HerClass
 TheirClass
 #RingoClass

Parsing this would return an arrayref:

 
  [ 'MyClass', 'HerClass', 'TheirClass' ]

.perl

This file is a full-fledged perl data structure, dumped to a file using Data::Dumper. It can be any type of structure, but it's normally used to represent a hashref containing all sorts of different types of information. It's also fairly easy to edit such a file using your favorite plain-text file editor.

When reading this type of configuration, we just return the data structure saved in the file -- if the file is an arrayref, we return an arrayref.

When we use this structure to save information for objects (such as the OpenInteract::Config::PerlFile object), we never save class information about the object, just the data. We can always re-bless the structure after it's eval'd in.

Example:


 $data = {
          'db_info' => {
            'db_owner'    => '',
            'username'    => 'test',
            'password'    => '',
            'dsn'         => 'mydb',
            'db_name'     => 'myopeninteract',
            'driver_name' => 'mysql',
          },
          ...
 };

Website Configuration

All files referenced in this section are contained in the home directory of the website. For instance, if you install a website using:


 oi_manage --base_dir=/path/to/install --website_dir=/path/to/website \
           --website_name=MySite create_website

The files described here would be in /path/to/website.

File: conf/base.conf

This is one of the most important configuration files in OpenInteract -- fortunately, nobody should never need to edit it. :-) This file allows OpenInteract::Startup to bootstrap all the configuration information by supplying information for the class that reads in the data for the server configuration (OpenInteract::Config::PerlFile by default), the base directory for this application, the name of the configuration directory and file. You can also specify the Request class (OpenInteract::Request by default) and a Stash class (no default specified -- every application needs to have its own).

Example:


base_dir         /opt/OpenInteract
website_dir      /home/httpd/mysite.com
website_name     MySite
config_class     OpenInteract::Config::PerlFile
config_dir       conf
config_file      server.perl
request_class    OpenInteract::Request
stash_class      MySite::Stash

File: conf/apache.dat

List the classes needed by mod_perl in the Apache:: class. The only ones you might need to change are Apache::StatINC (shouldn't be used on production servers) and Apache::Session::MySQL (you might require a different session backing store).

Website Configuration

File: conf/server.perl

Website configuration is done through the server.perl located in a website's conf/ directory. When you create a new website using oi_manage you get a starter server.perl file with a number of items filled in. (View the sample server.perl file -- note that keys surrounded with '%%' will be replaced when you create a website with oi_manage.

This configuration is always available within the OpenInteract environment (as an OpenInteract::Config object) to developers and is read in anew every time the server starts.

The file itself is formatted in Perl and you can edit it with your favorite plain-text editor. You should be able to check whether it is syntactically valid by doing:


 perl -wc server.perl

This won't tell you whether it's functionally valid, but it's a first step. Note that there is a routine in oi_manage that allows you to check whether the parameters defined in server.perl will allow a database connection to be made:


 oi_manage --website_dir=/path/to/my/website test_db

Apache

NOTE: DO NOT restart the Apache/mod_perl process using the HUP signal. Your modules will not get reloaded properly.

Proxy Setup

OpenInteract depends on a persistent Perl environment within a web server. Currently, the best alternative is mod_perl.

mod_perl is extremely powerful, but this power can come at a price. Embedding Perl into Apache uses more resources (particularly memory) than just using Apache alone. A number of developers have experimented with various ways of minimizing the memory footprint of mod_perl, and one of the easiest and best performing methods is to use a proxy server.

This is described in great detail in the mod_perl guide under the Choosing the Right Strategy heading. But we'll summarize here:

  1. Setup a plain Apache server with mod_proxy and mod_rewrite to listen to port 80 for your website. (We describe the build process below.)
  2. Tell this server to deal with static file requests (images, movies, PDFs, etc.)
  3. Proxy all other requests back to a heavier mod_perl server.
  4. Receive the information back from the mod_perl server and send to the client

The benefits of this are:

  1. Resource-hogging mod_perl processes do not serve static files -- if they did, you'd need more of the processes.
  2. The front-end proxy is able to feed data back to the client at whatever rate it needs without taking up many resources the entire time. For instance, users reaching your website with modems can tie up a web server process for much longer than users who are on some sort of broadband network. If the process is small it's not such a big deal.
  3. Since they are separate, you can make changes to the (heavy) back-end and mask them by the (light) front-end. This is a great help when things are going wrong with the back-end and you don't want users to see nasty error pages.
  4. Also since they are separate, you can very easily move the back-end process to an entirely separate machine (or machines, using some sort of DNS or load-balancing manipulation) if the need arises.

Running OpenInteract in this environment is strongly recommended, and it comes with configuration files that make it easier to do the Right Thing.

Building Apache: Proxy and mod_perl

First, you need to get the mod_proxy_add_forward module make available by Ask Bjoern Hansen. Retrieve it from: http://develooper.com/code/mpaf/mod_proxy_add_forward.c

Once you've retrieved the file, copy it into the src/modules/extra directory of your Apache source code directory. An example of the 'activate-module' and 'enable-module' directives to put this module into your Apache is below in the as well as in the source code for mod_proxy_add_forward itself.

Once you've retrieved the extra module and copied it to the right place, you can create apache and mod_perl with the following steps. Note that this assumes you have not installed apache from source before and that you're installing to the directory /usr/local/apache -- modify as needed.


 1.  >> tar -zxvf apache-1.3.12.tar.gz

 2.  >> tar -zxvf mod_perl-1.24.tar.gz

 3.  >> cd apache-1.3.12

 4.  >> ./configure --prefix=/usr/local/apache \ 
                --enable-module=rewrite --enable_module=proxy \
                --activate-module=src/modules/extra/mod_proxy_add_forward.c \
                --enable-module=proxy_add_forward

 5.  >> make

 6.  >> make install
 (proxy server binary is now installed as /usr/local/apache/bin/httpd)

 7.  >> cd ../mod_perl-1.24

 8.  >> perl Makefile.PL EVERYTHING=1
 # Configure mod_perl with ../apache_1.3.12/src ? [y]

 9.  >> y
 # Shall I build httpd in ../apache_1.3.12/src for you? [y]

 10. >> y

 11. >> make

 12. >> make install
 (mod_perl Perl modules are now installed)

 13. >> cp ../apache-1.3.12/src/httpd /usr/local/apache/bin/httpd_mp
 (mod_perl-enabled Apache is now installed)

This is a very simple method for creating both a lightweight proxy Apache binary and a heavyweight mod_perl-enabled Apache binary. See the mod_perl Guide for many, many more details about building mod_perl.

It is strongly recommended that you do not build mod_perl using DSOs and that you do not use pre-built versions such as those supplied by RedHat with its RPMs. However, using the DSO mechanism probably works fine for the front-end proxy server.

Configuration Overview

Use oi_manage! Use OI_MANAGE! Use OI_MANAGE!

The oi_manage script included with OpenInteract performs a number of tasks for you that make your life much easier. When you run the create_website command along with the appropriate parameters, oi_manage will copy configuration files from the base installation to your website directory and customize them for your website's parameters for you.

For instance, two of the files that are copied to your website's conf/ directory are httpd_static.conf and httpd_modperl.conf. (See httpd_static.conf and httpd_modperl.conf -- the items marked with '%%' are replaced in the customization process.) You will still need to edit a few parameters in them -- oi_manage is pretty smart, but it can't find out which IP address you want your website to listen to! -- but much of it is filled in for you.

Static Apache Configuration

After you've run oi_manage, you will need to modify a few parameters in the static Apache configuration file.

  1. IP address: Do a search-replace for '127.0.0.1' with the IP address you want the website to listen to. Note that if you're using named virtual hosts you will not want to keep the Listen directive. You will also need to specify the NameVirtualHost directive in your main Apache configuration file.
  2. ServerAdmin: Change the value for the 'ServerAdmin' key
  3. ServerName: Change the value for the 'ServerName' key

Proxy configuration is fairly simple. Every rule (starting with RewriteRule) is processed in order. Once a rule is met, no further rules are processed unless the satisfied rule specifies it.

The default proxy configuration assumes that the only static files you will want to serve directly from the proxy server are images. This action is specified by this line:


 RewriteRule ^/images - [L]

If you want to add other locations that will be entirely served by the lightweight server, just add them after this line. For example, if my website had a directory '/forms' where we kept PDF versions of forms for our customers to fill out, I could add:


 RewriteRule ^/forms - [L]

And every URL beginning with /forms will be answered by the front-end lightweight server. The [L] stands for "Local" and means that you want this server (the proxy server) to handle the request.

The only word of warning here is that as an administrator you might need to keep an eye on what the back-end server is using for URLs. For instance, say I entered this /forms configuration directive and later a developer on the back-end server tries to configure OpenInteract to perform a certain action when given the /forms URL. Unless the developer knows that the front-end server is answering all the /forms URLs, she will have a very frustrating time trying to figure out why her handler isn't responding.

mod_perl Configuration

After you've run oi_manage, you will need to modify a few parameters in the mod_perl Apache configuration file.

  1. IP address: Do a search-replace for '127.0.0.1' with the IP address you want the website to listen to.
  2. ServerAdmin: Change the value for the 'ServerAdmin' key
  3. ServerName: Change the value for the 'ServerName' key
  4. Port: (optional) Do a search-replace for the default value of '8080' with whatever port you want to run the mod_perl server on

(Note: You can skip the remainder of this section if you just want to get something up and running. The oi_manage script takes care of all this for you. But if you're curious, read on.)

Four separate items need to be customized in the conf/httpd_modperl.conf file

First, define the library paths for this website. Note that this is applied on a server-wide basis, so be careful of namespace clashes.

Example:


 <Perl>
  use lib qw( /home/httpd/mysite.com );
 </Perl>

Second, define a parameter that allow us to bootstrap our configuration object which contains the rest of everything we need to know. The parameter is 'StashClass' you set it to a value that OpenInteract can use for a website's stash class.

Example:


 PerlSetVar StashClass      MySite::Stash

Third, you need to bring in your startup.pl. (Information on what is done in the startup.pl is found in the OpenInteract Developer's Guide.)


 PerlRequire /home/httpd/mysite.com/conf/startup.pl

Fourth and finally, we need to ensure that every request coming in goes through a single Apache content handler: OpenInteract.pm. (This module is located in the base package.) To enable this, just do:


 <Location /> 
  SetHandler perl-script 
  PerlHandler OpenInteract
 </Location>

This Apache content handler is in the base package since it's part of the base functionality of the framework. We can just say "OpenInteract" in the httpd.conf file because we have already included the library in our startup.pl.

Running Multiple Websites on One Server

mod_perl is a fairly heavyweight server, and the mod_perl Guide spends quite a few pages on different strategies you can use to minimize the memory footprint. While the scenario of running one website per server is appealing because it minimizes interaction between websites, it might not be feasible due to your resources.

Fortunately, OpenInteract has taken pains to ensure that different websites can run together using the same mod_perl-enabled apache process family.

The only trick currently is to ensure that websites do not use the same namespace. It is best to make them as explicit as possible and not use acronyms. This doesn't impose as much of a burden as it might sound: developers rarely use fully qualified class names since OpenInteract provides an aliasing facility for this sort of thing.

To run multiple websites under the same server, Every website must have what is known as a stash class, or a package where variables (such as the configuration object, the database handle, the list of error handlers for website modules, and others) are fetched during a request and kept intact between requests as well.

This allows us to do certain expensive operations once -- connect to a database, establish a cache, create a template processing object, parse the configuration file -- and keep the results for the lifetime of the apache httpd child.

Most of the time you'll never deal with the stash class -- the OpenInteract::Request object saves and retrieves information from it behind the scenes. In addition, when you create a new website the oi_manage script creates a stash class specifically for that website, and you don't need to do anything with it to make it work.

Suggested Readings