How to deal with log storage/rotation

[aseques]

Keeping proper logs is really important when looking for issues on your server. Usually is the only way you can find attacks or broken stuff. Also it's mandatory by law to keep this information for at least six months in Europe.

Here I'll summarize the issues and the best practices about log keeping (in my opinion) and open bugs about what can be improved.
Feel free to report any issues you find or corrections about this.

The most important logfiles are:
.- The logs used for http accounting are the ones /var/log/apache2/*.log (cannot use dateext)
.- The logs used for ftp accounting are the ones in /var/log/proftpd/ftp_traff.log (cannot use dateext)
.- The per user logs from apache are on /var/log/apache2/users/*
.- The mail logs are on /var/log/mail.*

More information about the logs: here

Known issues

Create a script to handle the old log archive
There is a big amount of logs in a used server. There should be a crontab that could handle the catalogation of those logs. There's a ready to use patch in ticket 2577
Ticket 2577


Updates
improvements in logrotate
The logs that are interesting to be keeped should change the name scheme into something more clear (specifying date). Rotation should be done daily to help in high traffic sites. The logs should be keeped for longer time.
Ticket 2570
Fixed on r3838

[aseques]

Included the fixed part to the post...

[kilburn]

Beware that this can impose a performance penalty on servers with a high number of domains. Basically, after running for a full year, such a server will have 365*n_domains files in a single folder. Therefore, anything that tries to scan such a folder is going to impose a huge number of io operations to the disk.

For instance, on a server with 100 domains, this means that performing an "ls -l" inside this directory will generate 36500 io operations. Considering that a typical 7200rpm hdd can perform approximately 90iops, this means that the disk will be saturated for almost 7 full minutes. Using the past configuration, the same problem exists but it is 7 times softer (weekly logs instead of daily logs = 7 times less files).

[aseques]

(05-30-2011 09:58 PM)kilburn Wrote:  Beware that this can impose a performance penalty on servers with a high number of domains. Basically, after running for a full year, such a server will have 365*n_domains files in a single folder. Therefore, anything that tries to scan such a folder is going to impose a huge number of io operations to the disk.

For instance, on a server with 100 domains, this means that performing an "ls -l" inside this directory will generate 36500 io operations. Considering that a typical 7200rpm hdd can perform approximately 90iops, this means that the disk will be saturated for almost 7 full minutes. Using the past configuration, the same problem exists but it is 7 times softer (weekly logs instead of daily logs = 7 times less files).

That's certainly true, maybe this could go together with a script to archive somewhere else the past month logs launched monthly by crontab.
In my experience it's really a pain having to walk around trying to find when something happened and it's a real advantage having the day for the logs.
What do you thing about creating the script, it's quite easy to do.

[kilburn]

I'm okay with a script-based solution to archive the log files (I would create a folder for each day).

If you are going to do it, take into account that log removal also has to be taken care of by the archival (or another) script, since logrotate will not find the logs anymore once archived.

[aseques]

(05-31-2011 06:23 AM)kilburn Wrote:  I'm okay with a script-based solution to archive the log files (I would create a folder for each day).

If you are going to do it, take into account that log removal also has to be taken care of by the archival (or another) script, since logrotate will not find the logs anymore once archived.

Ok, I created a new script based on ispcp-del-logs.sh and some snipped that kilburn sent to me.
I've been testing it on my test machine and works fine to archive the logs from apache users. It archives the logs in archive/YYYYMM/
It tries to guess the logs date from the name (20110606 would mean june 2011), if it doesn't find the name archives the logs to the current month folder.
The maximum number of files that there will be on a folder will be 31 per domain, but it it doesn't need rotation there will be less files.
I attach the script, please have a look and comment what you think.

[aseques]

Just opened ticket 2577 to track the patch with the log archiver. As it is, it should keep the logs in user folder properly archived while having a minimal impact on server performance. I've been testing it on my lab and it works as expected.

[RatS]

Thanks for sharing your thoughts. Now I know what the problem is about: the amount of files in the log folder and the IO operations related to it. this changes my point of view meaning, your script can make a difference