High mem usage by ispcp-vrl-traff

[achioo]

My traf logs have a total of 9,050,173 lines to go through... the traf logs are located in apache2 with the other logs in users...

[pgentoo]

9 million rows huh? Well, that seems like a lot of work to do every 30 minutes.

-
Jesse

[Sweeny]

Hi there,

I have the same problem ... with VHCS2 and ISPCP i think it's the same script. The Traffic script eats 95% of the memory on a machine with 2GB Ram ... When the Traffic stats are calculating the system is unaccessable via SSH HTTP or anything other ... So this script needs to be speed up and improved ... I deactivatet the traffic calculation on our machines because it always results in highloads ...

Greetings
Sweeny

[pgentoo]

Sw1fty Wrote:Hi there,

I have the same problem ... with VHCS2 and ISPCP i think it's the same script. The Traffic script eats 95% of the memory on a machine with 2GB Ram ... When the Traffic stats are calculating the system is unaccessable via SSH HTTP or anything other ... So this script needs to be speed up and improved ... I deactivatet the traffic calculation on our machines because it always results in highloads ...

Greetings
Sw1fty

Ok, well at this point I would like one of the lead developers to chime in and say whether we need to keep these traff files around, or we can just truncate them each time we generate the traffic stats. If there is no issue doing this, then i think it is an "easy" change to wipe the file after copying it to the ./backup/ (where it does the actual parsing). The only issue i see with this, is what if the database call to update the bandwidth usage fails? We need to persist that somewhere so it can be updated later, but there is no reason we should have to totally recalculate it next time we run...

Anyone have any feedback on the proposed changes?

Thanks,
Jesse

[achioo]

This is actually causing one of my users traffic information to jump drasticly. ispCP is currently reporting that he has had 32Gb of traffic today on his site but in the overall traffic for today there has only been about 1GB (total) of traffic. I think for the time being I'm going to have to disable this because it seems to be looping for some reason which is just killing the system.

[pgentoo]

achioo Wrote:This is actually causing one of my users traffic information to jump drasticly. ispCP is currently reporting that he has had 32Gb of traffic today on his site but in the overall traffic for today there has only been about 1GB (total) of traffic. I think for the time being I'm going to have to disable this because it seems to be looping for some reason which is just killing the system.

It is misreporting after you made the changes, or have you not made them yet? If it is occuring after the changes, we may have to change how this data is stored and the totals are computed.

[achioo]

I didn't change anything on it but it freaked me out so I got mod_cband workin last night and things are normal... teh srv traffic works fine...

[raphael]

Quote:it looks like the ispcp-vrl-traff script actually rotates the traffic log after it reads it....

no, it doesn't

Quote:I don't really see why anyone would ever want a full year of transfer log files archived

When you need to do some tracing you sometimes need old logs.
You can always go ahead and modify the config files. The default ones aren't meant to work the best on all and every situation, that's the reason for their existence.

Quote:Who has space for this??

you don't have 1GB for them?...

Quote:Maybe one of the developers could chime in as to why this is weekl

so you can access current week's logs and so the server doesn't logrotate the logs every day

Quote:An alternative here could be to just `cat "" > domain.com-traf.log`, which would wipe the file's contents, but not break apache's traffic logging.... Just some thoughts.

Not really, AFAIR the file handle would need to be rewinded, and apache won't do it.
Not even a reload seems to work on some setups, that's why restart is used.

[pgentoo]

Raphael, thanks for the reply.

In response to your comments:

I understand keeping log files to some point to troubleshoot issues, etc... But I don't see the point of keeping all the files (error, access, and traffic) for a full year. This just seems like overkill for a default configuration. In my previous post I commented "Who has space" for the traff files, but i meant all the log files. This could add up to quite a bit of space if you keep a whole years worth... This just seems like a lot of wasted space that will pretty much never be used. I agree that if an admin wants they could raise these values, but I just think the default of retaining files for 1 year is overkill.

As per the issue of the file handle... On my current system I generate my webalizer stats (every 15 minutes) and wipe the access logs (with the `echo ""> access_log` method). I don't reload or restart apache and everything works fine for me. I'm not saying this isn't the case with some systems, but I was suggesting this approach based on my own limitted experience. If this doesn't work for all systems then maybe we need to think of another way to make gathering traffic stats more efficient.

I'll take a look into how this data is being stored, and more into how it is being updated, and see if I can come up with a recommendation on how to make it more efficient without wiping the file contents every 30 minutes (only every week).

Thanks again for your comments,
Jesse

[Sweeny]

Is there no solution just to rename the traf files so that they aren't calcualted each time the vrl-traf calculates? Because on high traffic sites with many hits it's just overkill like vrl-traf works at the moment. COnsuming 45-80% Memory for at least an half hour is overkill for every system ... static pages need 30 - 50 seconds to be shown because apache has no memory to deliver them ... mails are bouncing because postfix hasn't enough memory to QUEUE them.

So for high traffic sites you have to deactivate vrl-traf completly but this can't be a soultion for hosters.

Greetings
Sweeny