High mem usage by ispcp-vrl-traff

[pgentoo]

How old are your apache *traf.log files, i'm guessing that they aren't being rotated (or at least not often enough)? Run `wc -l /var/log/apache2/*-traf.log` and see how many lines are found in all the traffic files... if you think about it, when it is processing it is adding this many numbers together to get the end result for each domain to put in the db.

I believe /etc/logrotate.d/ispcp defines that all the apache log files be rotated weekly, and keep 52 weeks worth of logs. I don't really see why anyone would ever want a full year of transfer log files archived. Who has space for this?? Even more why we would want to keep the contents at all after we update the traffic into the db?

Maybe one of the developers could chime in as to why this is weekly, and why we keep them... but personally i'd recommend changing it to only rotate the non-traffic related apache logs, and just let the ispcp-vrl-traffic delete the contents of the traf.log files after it reads them.

I think a big reason we don't currently delete these files after each time the script runs, is that we need to restart apache to get it to log to the new file, since it holds a handle to the old one. An alternative here could be to just `cat "" > domain.com-traf.log`, which would wipe the file's contents, but not break apache's traffic logging.... Just some thoughts.

I do something similar to this on my current system (with a control panel i developed years back). I was trying to keep webalizer updated and traffic stats into mysql as often as possible and wound up with very bad performing scripts just like this... I ended up doing just what i suggested above... Running the scripts often, and clearing out the contents of the log file often.

I guess my recommendation is as follows:
1. remove logrotate.d entry for rotation of the traf files (but leave it for the other apache log files). This change might not be required... Are yours located in the /var/log/apache2/users/ folder, or in /var/log/apache2/? Mine appear to be going to /var/log/apache2, but this could be a gentoo specific bug. If they aren't located under ./users/ then no logrotate.d/ispcp changes need to be done.
2. change ispcp-vrl-traff script to copy the log file to ./backup (does this currently), then immediately wipe the contents of the log file (this step would be new). This will keep the log from growing too big and taking forever to process every time.

These recommendations come with no warranty, as I'm new to this system myself. But this seems to me like the issue.

Hope that helps.
Jesse