ispcp-vrl-traff improvement

[Nathariel]

Hey all.

I would like to propose an enhancement for ispcp-vrl-traff script, because it is unable to handle large files and could easily crash heavy loaded servers.

The problem is in the gen_log_file function and the way that it creates the temp .diff files when comparing new and .prev logs. Currently, this is done by the following line:

Code:

diff_command("$main::cfg{'CMD_DIFF'} -a $dest_file $dest_file_prev | $main::cfg{'CMD_EGREP'} '^<' | $main::cfg{'CMD_SED'} 's/^< //' 1> $log_file");

Behind the diff stands the problem with the LCS and no matter how optimized the algorithm is, it needs a big amount of memory. I am not going into details what needs diff to do in order to compare two large files but the memory usage is enormous.

Thus, when we have heavily loaded web servers and log processing, this could crash the log processing server, eating all its memory in a second. (Upon such cases,and if the system is still alive, the kernel starts killing processes with higher memory usage - mysql, apache...are the first to be brought down...)

When we speak about high loaded web and mail servers, the fast log rotation is not enough and such a resource waste should be addressed.

I have written a small enhancement which creates a local file db with the last read positions of all logs. Upon invoking, it checks this position and seeks to it in the new log file. After that it reads to the end and writes the output in the temp .diff file.

The operation is more CPU intensive and has almost no memory overhead.

I am attaching a patch with the differences between the original and modified ispcp-vrl-traff,the look of the new gen_log_function and the FSM logic.
I have also submitted a ticket about this issue: http://www.isp-control.net/ispcp/ticket/1623

The algorithm could be improved a little more, but I have tried to save the old function as much as I can.

Please feel free to test / improve this workaround.

[rkhalil]

Hi!
We trying to apply the patch buts no works,

/var/www/ispcp/engine/traffic# patch -p0 <ispcp-vrl-traff.diff
patching file ispcp-vrl-traff
patch: **** malformed patch at line 58: @@ -153,20 +196,212 @@

We have RC7

do u know whats happen-s ?

thanks
Raul

[Nathariel]

(02-07-2009 11:47 PM)rkhalil Wrote:  Hi!
We trying to apply the patch buts no works,

/var/www/ispcp/engine/traffic# patch -p0 <ispcp-vrl-traff.diff
patching file ispcp-vrl-traff
patch: **** malformed patch at line 58: @@ -153,20 +196,212 @@

We have RC7

do u know whats happen-s ?

thanks
Raul

Yes, it was my fault.
Please try with this patch:
  ispcp-vrl-traff.diff.txt (Size: 5.4 KB / Downloads: 36)

(The diff in the head post is also updated.)

[RatS]

Please update the ticket as well! Thank you for this, will be in 1.0.1.

[rkhalil]

we test this enhancement and is amazing!! no more memory overhead!

Thanks
Raul

[vetch101]

(02-10-2009 03:25 AM)Nathariel Wrote:  

(02-07-2009 11:47 PM)rkhalil Wrote:  Hi!
We trying to apply the patch buts no works,

/var/www/ispcp/engine/traffic# patch -p0 <ispcp-vrl-traff.diff
patching file ispcp-vrl-traff
patch: **** malformed patch at line 58: @@ -153,20 +196,212 @@

We have RC7

do u know whats happen-s ?

thanks
Raul

Yes, it was my fault.
Please try with this patch:

(The diff in the head post is also updated.)

Hi,

I'm having services crash out because of the issues with memory on a VPS.

I'm running a nightly build between RC6 and RC7.
What do I need to do to run the patch?
I intend to upgrade to 1.0 stable at some point in the not-too-distant future.
Will this patch - a) break my capability to do that? b) need to be reapplied? c) work with any RC version?

Many thanks,

Jx

[Zothos]

can someone make the patch working with the current trunk? i realy like to see this in our development trunk soon

[vetch101]

Quote:Hi,

I'm having services crash out because of the issues with memory on a VPS.

I'm running a nightly build between RC6 and RC7.
What do I need to do to run the patch?
I intend to upgrade to 1.0 stable at some point in the not-too-distant future.
Will this patch - a) break my capability to do that? b) need to be reapplied? c) work with any RC version?

Many thanks,

Jx

Hi all,

I ran the command
/var/www/ispcp/engine/traffic# patch -p0 <ispcp-vrl-traff.diff
patching file ispcp-vrl-traff

... and the patch seemed to go through...
Will I need to do anything else, like reapplying the patch when I upgrade to 1.0?

Many thanks,

Jx

[Nathariel]

(03-10-2009 07:59 PM)vetch101 Wrote:  

Quote:Hi,

I'm having services crash out because of the issues with memory on a VPS.

I'm running a nightly build between RC6 and RC7.
What do I need to do to run the patch?
I intend to upgrade to 1.0 stable at some point in the not-too-distant future.
Will this patch - a) break my capability to do that? b) need to be reapplied? c) work with any RC version?

Many thanks,

Jx

Hi all,

I ran the command
/var/www/ispcp/engine/traffic# patch -p0 <ispcp-vrl-traff.diff
patching file ispcp-vrl-traff

... and the patch seemed to go through...
Will I need to do anything else, like reapplying the patch when I upgrade to 1.0?

Many thanks,

Jx

Sorry for the delay, I've been busy in the last weeks.

We are using the patch on 1.00 and there is no need for any changes. It works fine.
You don't need to do something else after you apply the patch, it should be working without problems.

However this patch is not in the trunk yet so if you are intending to upgrade, you should reapply it.

If you have any issues, please post them here.

Thanks

[vetch101]

(03-24-2009 12:00 AM)Nathariel Wrote:  

(03-10-2009 07:59 PM)vetch101 Wrote:  

Quote:Hi,

I'm having services crash out because of the issues with memory on a VPS.

I'm running a nightly build between RC6 and RC7.
What do I need to do to run the patch?
I intend to upgrade to 1.0 stable at some point in the not-too-distant future.
Will this patch - a) break my capability to do that? b) need to be reapplied? c) work with any RC version?

Many thanks,

Jx

Hi all,

I ran the command
/var/www/ispcp/engine/traffic# patch -p0 <ispcp-vrl-traff.diff
patching file ispcp-vrl-traff

... and the patch seemed to go through...
Will I need to do anything else, like reapplying the patch when I upgrade to 1.0?

Many thanks,

Jx

Sorry for the delay, I've been busy in the last weeks.

We are using the patch on 1.00 and there is no need for any changes. It works fine.
You don't need to do something else after you apply the patch, it should be working without problems.

However this patch is not in the trunk yet so if you are intending to upgrade, you should reapply it.

If you have any issues, please post them here.

Thanks

Thanks muchly...

As it goes, I had pretty much assumed the same and applied it previously...

The server has been running happily for 11 days now, so I'm thinking the issue has been resolved!

Thanks so much for your help!

Jx