Apache::Logmonster - Log Processing Utility
Author: Matt Simerson.
[ Install | Configure | FAQ | ChangeLog | Sample ]
Why did you write this?
Typical Scenario: You have a web server that serves your domain. You write a simple script to restart apache each night and pipe the logs off to your analyzer. It works just fine.
ISP/Hosting Scenario: Each server hosts many domains. You may also have load balanced servers (multiple machines) serving each domain. A tool like this is necessary to:
- 1. collect all the log files from each server
- 2. split the logs based on the virtual host(s)
- 3. sort them into cronological order
- 4. feed logs into analyzer
- 5. do something with the raw logs (compress, drop into vhost dir, etc)
How do I enable log processing for a virtual domain?
- Simply create the directory ("stats" by default) within the DocumentRoot of the virtual host. For example, the docroot for example.com is /usr/home/example.com/html. To enable virtual host processing, create the directory /usr/home/example.com/html/stats. Their statistics will be processed.
How do I process my logs hourly?
Set cronolog to "%Y/%m/%d/%H", run logmonster with -h, and adjust cron. Get yourself acquainted with webalizer -p and it's limits
Why do you use cronolog?
Read the Apache docs and all the caveats required to rotate logs, including restarting the server. Then factor that into using several servers in different time zones, etc. and you'll find it's much more reliable and easier to use cronolog. I've used cronolog for years and have never had a problem with it.
Why not use one file per vhost so you don't have to split them?
I tried that. One problem is that you end up with lots of open file descriptors (one per vhost) and that only scales so far before you decide it's not such a great idea. You still end up having to collect the files from multiple servers and sort them before feeding them into your log processor so you might as well just start by having them all in one place.
What's the recommended way to implement this?
- • Adjust CustomLog and add the %v to it as show above.
- • If you aren't already using cronolog, start. Wait a day.
- • Test by running "logmonster -d -n".
It will tell you what it's doing and everything should look reasonable. Correct anything you don't like (like creating $statsdir for domains that should have it, etc) and then create a cron entry running "logmonster.pl -d" anytime after midnight. Read the output from logmonster in your mailbox for the next week. When you're confident everything is great, adjust crontab and add a "-q" to it so it stops emailing you (unless there's errors).
Can you explain how to use the -b stuff?
Imagine you shut your server down at 0:55 last night to do some system maintenance. You brought it back up at at 1:05 (10 minutes later) but your cron job that runs logmonster at 1:00am didn't run. Easy enough, you just run it on the command line and all is well.
Now, let's suppose you made an oopsie that's caused logmonster to not run for all of the last week. Your back from vacation and notice the errors in your mailbox because that's where you've configured cron stuff to go, right? Now you set about to fix the problem. The best way to do that is run logmonster with "-d -b7". Logmonster will dutifully process the logs from 7 days ago (after confirming the date with you). Then run again with "-d -b6", etc until you're current.
Last modified on 6/18/05.
|
