reslog Version 3.04

What's New?

2005-05-08: reslog version 3.04 is released. Fixed a typo when loading File::MMagic. Download reslog version 3.04.
2005-05-08: reslog version 3.03 is released. Default installation directory moved back to /sbin. make upgrade is added to remove files from the old versions. flock() policy is redesigned for consistency and less errors. Download reslog version 3.03.
2005-05-02: reslog version 3.02 is released. Resolver is now independent from the main program. Successfully tested and run on MSWin32 and cygwin. Several small bugs are solved. Download reslog version 3.02.
2005-04-26: reslog version 3.01 is released. A serious bug that --suffix and --trim-suffix was obtaining wrong value is fixed. You should upgrade to version 3.01 immediately if you were using 3.00. Download reslog version 3.01.
2005-04-26: reslog version 3.00 is released. I rewrote everything. The code is now object-oriented. ithreads Threading support is added so that it is really fast now. bzip2 compression is supported. Compressing and decompressing using gzip and bzip2 executables without installing Compress::Zlib and Compress::Bzip2 is now possible, although discouraged. Certain file name suffix can be removed before outputing the result file, so that it can be line-up with other log file filters. Download reslog version 3.00.
2002-09-18: reslog.pl version 2.1.0dev2 is released. This release corrects several documentation errors. It also adds SourceForge in the documentations as one of the sources of reslog.pl. You can think of it as a Source Forge Memorial Release. :p You don't have to upgrade to this version in a rush. Download reslog.pl version 2.1.0dev2.
2002-09-17: reslog.pl is hosted at SourceForge now! Congratulations! (Although I'm still trying hard to get it working at this time... ^^; )
2002-08-31: reslog.pl version 2.1.0dev1 is released. This release fixes the problem that it wouldn't run with Perl 5.8.0. I marked it as dev because I wish I can implement Compress::Bzip2 and ExtUtils::MakeMaker in the 2.1.0 stable release. But it is is in fact very stable. Download reslog.pl version 2.1.0dev1.

Description

reslog reverse-resolves IP in the Apache log files. These log files can then be analyzed by another program, like Analog. You can think of it as a replacement of Apache HostNameLookups directive, in the sense that it resolves the client IP altogether once a day.

Caution

Resolving takes long time. This is mainly caused by resolving: Network packets may be filtered by firewalls; DNS servers may not be correctly configured; may not be up working; may sit in slow network sections; may be old slow machines; may have traffic jam... etc. All these reasons are out of our control.
If it stops in the middle of its execution, as when the user hits a Ctrl-Break, it may leave a temporary working file. The next time it runs, it will stop when it sees that temporary working file at the first sight. Please process that file first. You can resolve it again, just like an ordinary log file.
This prorgam needs temporary working space. Between memory and disk space, I choose disk space, since it is cheaper and may be available in more environments. However, this means that it needs free temporary disk space about 2 times of the size of the source log log file (10 times if using memory). Please make sure you have that much free space.

System Requirement

Perl, version 5.8.0 or above. reslog now has threading to speed up resolving, which requires Perl's ithreads threading module support that's only available since 5.8.0. You can still disable threading if it causes troubles to you, but the code itself need it. If you are using a Perl before 5.8.0, please upgrade it. You can run perl -v to see your current Perl version.
Required Perl modules: none.
Optional Perl modules:

File::MMagic

This is used to check the file type. If this is not available, reslog will try to find the file executable instead. If that is not available, too, reslog will try to judge the file type by its name suffix (extension). In that case reslog will not be able to check the file type when reading from STDIN. MSWin32 users can get file.exe from the GnuWin32 home page. Be sure to save it as file.exe somewhere in your PATH.

Compress::Zlib

This is used to support read/write of gzip compressed files. It is only needed when gzip compressed files are encountered. If it is not available when needed, reslog will try to use the gzip executable instead. If that is not available, too, reslog will fail. ActivePerl MSWin32 users can install this using ppm install Compress::Zlib, or get gzip.exe from the gzip home page. Be sure to save it as gzip.exe somewhere in your PATH.

Compress::Bzip2, version 2 or above

This is used to support read/write of bzip2 compressed files. It is only needed when bzip2 compressed files are encountered. If it is not available when needed, reslog will try to use the bzip2 executable instead. If that is not available, too, reslog will fail. Notice that older versions before 2 does not work, since file I/O compression were not implemented yet. ActivePerl MSWin32 does not have Compress::Bzip2 in their PPM deposit yet, as the time I'm writing this. You can get bzip.exe from the bzip home page instead. Be sure to save it as bzip2.exe somewhere in your PATH.

Term::Size

This is used to display the progress bar. Without this reslog won't display the progress bar, but nothing else is different. The progress bar is a good visual representation of what reslog is currently doing. Term::Size is only available for UNIX. For MSWin32, reslog will try to use Win32::Console instead.

You can always search, download and install the missing Perl modules from the the CPAN archieve.

Download

reslog's official websites is at...

You can always download the newest version of reslog from...

imacat's PGP public key is at...

Install

Well, reslog doesn't need installation. :p Just run it.

Okay, okay. In case that you can't get a good sleep without proper installation, here is the instruction. Notice that you can completely ignore this. Just run it.

Starting from version 3, reslog uses standard Perl installation with ExtUtils::MakeMaker. Follow these steps:

% perl Makefile.PL
% make
% make test
% make install

When running make install, make sure you have the priviledge to write to the instalation location. This usually requires the root priviledge.

If you are using ActivePerl under MSWin32, you should use nmake instead of make. nmake can be obtained from the Microsoft FTP site.

If you want to install into another location, you can set the PREFIX. For example, to install into your home when you are not root:

% perl Makefile.PL PREFIX=/home/jessica

Refer to the docuemntation of ExtUtils::MakeMaker for more installation options (by running perldoc ExtUtils::MakeMaker).

If you are upgrading from reslog.pl 3.02 or earlier, please read UPGRADE for some upgrade instruction.

Options

./reslog [options] [logfile...]
./reslog [-h|-v]

logfile

The log file to be resolved. Multiple log files are supported. If not specified, it will read from STDIN and output to STDOUT. You can also specify - to read from STDIN. Result of STDIN will go to STDOUT. Gzip and bzip2 compressed files are supported, too.

-k, --keep=mode

What to keep in the source file. Currently the following modes are supported:

a, all: Keep the source file after records are archived.
r, restart: Restart the source file after records are resolved. This is the default.
d, delete: Delete the source file after records are resolved.

-o, --override=mode

Whether we should overwrite the existing resolved files. Currently the following modes are supported:

o, overwrite: Overwrite existing target files.
a, append: Append the records to existing target files.
f, fail: Stop processing whenever a target file exists, to prevent destroying existing files by accident. This is the default.

-s, --suffix=suf

The suffix to be appended to the output file. If not specified, the default is .resolved.

-t, --trim-suffix=suf

The suffix to be trimmed from the input file name before appending the above suffix. Default is none. If you are running several log file filters, this can help you trim the suffix of the previous one.

-n, --num-threads=num

Number of threads to run simultaneously. The default is 10. Use 0 to disable threading. Your system must support threading itself. This option has no effect for systems that do not support threading.

-c, --stdout

Output the result to STDOUT.

-d, --debug

Show the detailed debugging messages. More -d to be more detailed.

-q, --quiet

Shihhhhhh. Only yell when errors.

-h, --help

Display the help message and exit.

-v, --version

Output version information and exit.

Copyright

This program is free software; you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation; either version 2 of the License, or (at your option) any later version.

This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details.

You should have received a copy of the GNU General Public License along with this program; if not, write to the Free Software Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA.

Release Notes

Please read the Changes for the new functions and bug fixes.

Bugs

reslog has a mailing list reslog-users@lists.sourceforge.net. If you have any question, comment, bug report, patch, please send it to the mailing list. Do not send mails directly to me unless you have private business with me. The mailing list has an archive. People having the same trouble as you may search the archive, find the answer and save a lot of time.