High mem usage by ispcp-vrl-traff

[achioo]

I'm kinda concerned by the amount of memory being used by ispcp-vrl-traff... exactly what does this script do and why is the mem usage so high?

Code:

PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
8285 root       9   0  809m 655m  64m D  0.3 74.1   0:06.82 ispcp-vrl-traff

[rbtux]

Is this issue reproducable...

When you kill the process and start it again...
Does it use the same amount of memory?

[raphael]

it analyses the traffic log files to generate the per-reseller and epr-domain traffic usage

[achioo]

yes rbtux if I kill it, it does it again because it appears to be a half hour cron job

[achioo]

here it is this morning from top... my load has also gone up to 15/16 because of it....

Code:

PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
20650 root      17   0 1040m 766m 591m R  3.6 86.7   0:32.01 ispcp-vrl-traff

[achioo]

Guys I seriously need help with this... it's causing my CPU to hit the upper 30's

[pgentoo]

I'm not sure exactly what this script is doing, but just some thoughts...

Do you have logrotate up and running on all your log files? If the logs are HUGE because they aren't being rotated properly, the parsing of the files to pulll out recent traffic data could suffer.

Do a "tail -f /var/log/ispcp/ispcp-vrl-traff.log" and see if maybe that gives some more info into what is causing the issues...

You're right though... Parsing log files for traffic data, if done ever 30 minutes, shouldn't take 15minutes to complete. That's not acceptable.

Let me know what you find regarding logrotate and the log file.

-
Jesse

[pgentoo]

Hrm, it looks like the ispcp-vrl-traff script actually rotates the traffic log after it reads it....

Do you have a TON of traffic on your server? How big are all your /var/log/apache2/*-traf.log files when the script runs? How many domains do you have?

[achioo]

The biggest log I have at present is about 26 megs in size... and there are 39 logs

[pgentoo]

How old are your apache *traf.log files, i'm guessing that they aren't being rotated (or at least not often enough)? Run `wc -l /var/log/apache2/*-traf.log` and see how many lines are found in all the traffic files... if you think about it, when it is processing it is adding this many numbers together to get the end result for each domain to put in the db.

I believe /etc/logrotate.d/ispcp defines that all the apache log files be rotated weekly, and keep 52 weeks worth of logs. I don't really see why anyone would ever want a full year of transfer log files archived. Who has space for this?? Even more why we would want to keep the contents at all after we update the traffic into the db?

Maybe one of the developers could chime in as to why this is weekly, and why we keep them... but personally i'd recommend changing it to only rotate the non-traffic related apache logs, and just let the ispcp-vrl-traffic delete the contents of the traf.log files after it reads them.

I think a big reason we don't currently delete these files after each time the script runs, is that we need to restart apache to get it to log to the new file, since it holds a handle to the old one. An alternative here could be to just `cat "" > domain.com-traf.log`, which would wipe the file's contents, but not break apache's traffic logging.... Just some thoughts.

I do something similar to this on my current system (with a control panel i developed years back). I was trying to keep webalizer updated and traffic stats into mysql as often as possible and wound up with very bad performing scripts just like this... I ended up doing just what i suggested above... Running the scripts often, and clearing out the contents of the log file often.

I guess my recommendation is as follows:
1. remove logrotate.d entry for rotation of the traf files (but leave it for the other apache log files). This change might not be required... Are yours located in the /var/log/apache2/users/ folder, or in /var/log/apache2/? Mine appear to be going to /var/log/apache2, but this could be a gentoo specific bug. If they aren't located under ./users/ then no logrotate.d/ispcp changes need to be done.
2. change ispcp-vrl-traff script to copy the log file to ./backup (does this currently), then immediately wipe the contents of the log file (this step would be new). This will keep the log from growing too big and taking forever to process every time.

These recommendations come with no warranty, as I'm new to this system myself. But this seems to me like the issue.

Hope that helps.
Jesse