reslog v3.00

What's New?

2005-04-24

reslog v3.00 is released. I rewrite everything. The code is now object-oriented. ithreads Threading support is added so that it is really fast now. Resolving is now by collecting all the IP appeared in all log files, resolving them altogether, and replacing IP with the resolved host names in the log records, instead of resolving line by line. bzip2 compression is supported. Compressing and decompressing using gzip and bzip2 executables without installing Compress::Zlib and Compress::Bzip2 is now possible, although discouraged. Certain file name suffix can be removed before outputing the result file, so that it can be line-up with other log file filters. Download reslog v3.00.

2002-09-18

reslog.pl v2.1.0dev2 is released. This release corrects several documentation errors. It also adds SourceForge in the documentations as one of the sources of reslog.pl. You can think of it as a Source Forge Memorial Release. :p You don't have to upgrade to this version in a rush. Download reslog.pl v2.1.0dev2.

2002-09-17

reslog.pl is hosted at SourceForge now! Congratulations! (Although I'm still trying hard to get it working at this time... ^^; )

2002-08-31

reslog.pl v2.1.0dev1 is released. This release fixes the problem that it wouldn't run with Perl 5.8.0. I marked it as dev because I wish I can implement Compress::Bzip2 and ExtUtils::MakeMaker in the 2.1.0 stable release. But it is is in fact very stable. Download reslog.pl v2.1.0dev1.

Description

reslog reverse-resolves IP in the Apache log files. These log files can then be analyzed by another program, like Analog. You can think of it as a replacement of Apache HostNameLookups directive, in the sense that it resolves the client IP altogether once a day.

Caution

• Resolving takes long time. This is mainly caused by resolving: Network packets may be filtered by firewalls; DNS servers may not be correctly configured; may not be up working; may sit in slow network sections; may be old slow machines; may have traffic jam... etc. All these reasons are out of our control. In order to avoid simultaneously writing to the log file while resolving, that may ruin the log file, it uses flock() to prevent writing from other programs (or another instance of itself) that utilize flock(), too. In order to prevent from those that do not utilize flock(), like httpd, it copies the records to a temporary working file first, to minimize the time occupying the log file.
• If it stops in the middle of its execution, as when the user hits a Ctrl-Break, it may leave a temporary working file. The next time it runs, it will stop when it sees that temporary working file at the first sight. Please process that file first. You can resolve it again, just like an ordinary log file.
• This prorgam needs temporary working space. Between memory and disk space, I choose disk space, since it is cheaper and may be available in more environments. However, this means that it needs free temporary disk space about 2 times of the size of the source log log file (10 times if using memory). Please make sure you have that much free space.

System Requirement

1. Perl, version 5.7.2 or above. This program requires the new ithreads API that is only available from Perl version 5.7.2. Perl is distributed with most Unix now. If not, or if you have an older version of Perl, you can download and install/upgrade it from the Perl website. If you are using MSWin32, you can download and install ActivePerl from ActiveState Corporation.
2. Required Perl modules: none.
3. Optional Perl modules:
   1. File::MMagic: If it is missing, reslog will try to use file or check the file name suffix instead.
   2. Compress::Zlib: It is only needed to process gzip compressed files. If it is missing when needed, reslog will try to use gzip executable instead, and if that is not found, too, reslog stops. Non-UNIX users that don't have gzip need to install Compress::Zlib to process gzip compressed files.
   3. Compress::Bzip2: It is only needed to process bzip2 compressed files. If it is missing when needed, reslog will try to use bzip2 executable instead, and if that is not found, too, reslog stops. Non-UNIX users that don't have bzip2 need to install Compress::Bzip2 to process bzip2 compressed files.
   4. Term-Size: It is needed under UNIX to display the progress bar. If it is missing, no progress bar is displayed. For MSWin32, it will use Win32::Console instead.
   You can always search, download and install the missing Perl modules from the the CPAN archieve.

Download

reslog's official websites is at...

• reslog's Home Page at SourceForge
• reslog's Project Page at SourceForge
• reslog's Page at Tavern IMACAT's

You can always download the newest version of reslog from...

• SourceForge download page
• Tavern IMACAT's FTP directory

imacat's PGP public key is at...

• imacat's PGP key at reslog's Home Page at SourceForge
• imacat's PGP key at Tavern IMACAT's

Install

Well, reslog doesn't need installation. :p Just run it.

Okay, okay. In case that you can't get a good sleep without proper installation, here is the instruction. Notice that you can completely ignore this. Just run it.

Starting from version 3, reslog uses standard Perl installation with ExtUtils::MakeMaker. Follow these steps:

% perl Makefile.PL
% make
% make test
% make install

When running make install, make sure you have the priviledge to write to the instalation location. This usually requires the root priviledge.

If you are using ActivePerl under MSWin32, you should use nmake instead of make. nmake can be obtained from the Microsoft FTP site.

If you want to install into another location, you can set the PREFIX. For example, to install into your home when you are not root:

% perl Makefile.PL PREFIX=/home/jessica

Refer to the docuemntation of ExtUtils::MakeMaker for more installation options (by running perldoc ExtUtils::MakeMaker).

If you are upgrading from reslog.pl version 2, please read UPGRADE for some upgrade instruction.

Options

./reslog [options] [logfile...]
./reslog [-h|-v]

logfile

The log file to be resolved. Multiple log files are supported. If not specified, it will read from STDIN and output to STDOUT. You can also specify - to read from STDIN. Result of STDIN will go to STDOUT. Gzip and bzip2 compressed files are supported, too.

-k, --keep=mode

What to keep in the source file. Currently the following modes are supported:

a, all

Keep the source file after records are archived.

r, restart

Restart the source file after records are resolved. This is the default.

d, delete

Delete the source file after records are resolved.

-o, --override=mode

Whether we should overwrite the existing resolved files. Currently the following modes are supported:

o, overwrite

Overwrite existing target files.

a, append

Append the records to existing target files.

f, fail

Stop processing whenever a target file exists, to prevent destroying existing files by accident. This is the default.

-s, --suffix=suf

The suffix to be appended to the output file. If not specified, the default is .resolved.

-t, --trim-suffix=suf

The suffix to be trimmed from the input file name before appending the above suffix. Default is none. If you are running several log file filters, this can help you trim the suffix of the previous one.

-n, --num-threads=num

Number of threads to run simultaneously. The default is 10. Use 0 to disable threading. This option has no effect on systems that does not support threading.

-c, --stdout

Output the result to STDOUT.

-d, --debug

Show the detailed debugging messages. More -d to be more detailed.

-q, --quiet

Shihhhhhh. Only yell when errors.

-h, --help

Display the help message and exit.

-v, --version

Output version information and exit.

Copyright

Copyright © 2001-2005 imacat. All rights reserved.

This program is free software; you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation; either version 2 of the License, or (at your option) any later version.

This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details.

You should have received a copy of the GNU General Public License along with this program; if not, write to the Free Software Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA.

Release Notes

Please read the Changes for the new functions and bug fixes.

Bugs

reslog has a mailing list reslog-users@lists.sourceforge.net. If you have any question, comment, bug report, patch, please send it to the mailing list. Do not send mails directly to me unless you have private business with me. The mailing list has an archive. People having the same trouble as you may search the archive, find the answer and save a lot of time.