[NTLUG:Discuss] file size limits - an offshoot thread on log rotation

[David Camm]

<history>
two of the things our company prides itself on is the uptime of our web servers 
and our performance.

linux and apache are great contributors to this.

i've always been bothered about the way apache handles log files and the fact 
that rotation isn't built in.

both 1.3 (which we run) and 2.0 come with log rotation programs which pipe each 
log request to another process. the process then does the logging and decides if 
a rotation is necessary.

the thing that bothers me about this is that every log request gets piped - 
hence another process is kicked off.

my concern is that this puts an additional load on the system and could effect 
performance.

what we've done is to:

(1) not log image references. showing a customer the number of GETs to 
'logoimageslice1.jpg' doesn't provide a lot of meaningful information in a 
traffic report. this really cuts down the size of a site's log.

(2) not log the user-agent information. it can be awfully long, and nobody cares.

here's how we do it:

LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \" \"" short
SetEnvIfNoCase Request_URI \.ico$ unwanted
SetEnvIfNoCase Request_URI \.gif$ unwanted
SetEnvIfNoCase Request_URI \.jpg$ unwanted
SetEnvIfNoCase Request_URI \.png$ unwanted

and in the site definition:

CustomLog logs/adv.log short env=!unwanted

WebTrends has no problem reading this log file.

(3) when the log files get fairly large we do the following from the directory 
above the logs directory:

apachectl stop; mv logs oldlogs; mkdir logs; apachectl start

which takes no time, really, but it irks me.
</history>

so here's the question:

has anyone out there implemented the log rotation and, if so, what was the 
effect on web server performance?

david camm
president
advanced web systems
keller, tx
www.advwebsys.com