Typical Scenario: You have a web server that serves your domain. You write a simple script to restart apache each night and pipe the logs off to your analyzer.
ISP/Hosting Scenario: Each server hosts many domains. You have load balanced servers (multiple machines) serving each domain. A tool like this is necessary to:
host(s)
FileHandle POSIX Date::Format File::Copy File::Path Date::Parse Compress::Zlib
Most systems have all but Compress::Zlib installed.
vhost should be either a file with all your directives listed (ie, httpd.conf) or a directory (my favorite way) that contains files, each containing the VirtualHost and related directives for that Apache vhost.
While this may not work for everyone, it works very well for me on the several of the web farms that I manage:
The differences to LogFormat are subtle. In fact, that line is identical to it's heir in the httpd.conf-default file except for the %v at the end. That little %v tells Apache to write the canonical servername (vhost) into the logfile. That's how I can reliably parse the logs into vhosts. The CustomLog line is pretty easy too. We pipe our logs to cronolog and it's set to store each days logs into an appropriately named directory. So todays logs are stored on /var/log/apache/2003/03/05/access.log. That makes it very easy for me to grab an interval worth of logs to process.
Set cronolog to ``%Y/%m/%d/%H'', run logmonster with -h, and adjust cron. Get yourself acquainted with webalizer -p and it's limits
Read the Apache docs and all the caveats required to rotate logs, including restarting the server. Then factor that into using several servers in different time zones, etc. and you'll find it's a lot easier to just use cronolog. I've used cronolog for years and have never had a problem with it.
I tried that. One problem is that you end up with lots of open file descriptors (one per vhost) and that only scales so far before you decide it's not such a great idea. You still end up having to collect the files from multiple servers and sort them before feeding them into your log processor so you might as well just start by having them all in one place.
Adjust CustomLog and add the %v to it as show above. If you aren't already using cronolog, start. Wait a day. Test by running ``logmonster -d -n''. It will tell you what it's doing and everything should look reasonable. Correct anything you don't like (like creating $statsdir for domains that should have it, etc) and then create a cron entry running ``logmonster -d'' anytime after midnight. Read the output from logmonster in your mailbox for the next week. When you're confident everything is great, adjust crontab and add a ``-q'' to it so it stops emailing you (unless there's errors).
OK, lets say you shut your server down at 0:55 last night to do some system maintenance. You brought it back up at at 1:05 (10 minutes later) but your cron job that runs logmonster at 1:00am didn't run. Easy enough, you just run it on the command line and all is well.
Now, let's suppose you made an oopsie that's caused logmonster to not run for all of the last week. Your back from vacation and notice the errors in your mailbox because that's where you've configured cron stuff to go, right? Now you set about to fix the problem. The best way to do that is run logmonster with ``-d -b7''. Logmonster will dutifully process the logs from 7 days ago (after confirming the date with you). Then run again with ``-d -b6'', etc until you're current.