Current time: 10-21-2018, 02:46 PM Hello There, Guest! (LoginRegister)


Post Reply 
ispcp-vrl-traff improvement
Author Message
Nathariel Offline
Junior Member
*

Posts: 11
Joined: Jan 2009
Reputation: 0
Post: #1
ispcp-vrl-traff improvement
Hey all.

I would like to propose an enhancement for ispcp-vrl-traff script, because it is unable to handle large files and could easily crash heavy loaded servers.

The problem is in the gen_log_file function and the way that it creates the temp .diff files when comparing new and .prev logs. Currently, this is done by the following line:

Code:
diff_command("$main::cfg{'CMD_DIFF'} -a $dest_file $dest_file_prev | $main::cfg{'CMD_EGREP'} '^<' | $main::cfg{'CMD_SED'} 's/^< //' 1> $log_file");

Behind the diff stands the problem with the LCS and no matter how optimized the algorithm is, it needs a big amount of memory. I am not going into details what needs diff to do in order to compare two large files but the memory usage is enormous.

Thus, when we have heavily loaded web servers and log processing, this could crash the log processing server, eating all its memory in a second. (Upon such cases,and if the system is still alive, the kernel starts killing processes with higher memory usage - mysql, apache...are the first to be brought down...)

When we speak about high loaded web and mail servers, the fast log rotation is not enough and such a resource waste should be addressed.

I have written a small enhancement which creates a local file db with the last read positions of all logs. Upon invoking, it checks this position and seeks to it in the new log file. After that it reads to the end and writes the output in the temp .diff file.

The operation is more CPU intensive and has almost no memory overhead.

I am attaching a patch with the differences between the original and modified ispcp-vrl-traff,the look of the new gen_log_function and the FSM logic.
I have also submitted a ticket about this issue: http://www.isp-control.net/ispcp/ticket/1623

The algorithm could be improved a little more, but I have tried to save the old function as much as I can.

Please feel free to test / improve this workaround.


Attached File(s) Thumbnail(s)
   

.txt  ispcp-vrl-traff.diff.txt (Size: 5.4 KB / Downloads: 9)
(This post was last modified: 02-10-2009 03:26 AM by Nathariel.)
01-23-2009 09:07 PM
Visit this user's website Find all posts by this user Quote this message in a reply
rkhalil Offline


Posts: 2
Joined: Feb 2009
Reputation: 0
Post: #2
RE: ispcp-vrl-traff improvement
Hi!
We trying to apply the patch buts no works,

/var/www/ispcp/engine/traffic# patch -p0 <ispcp-vrl-traff.diff
patching file ispcp-vrl-traff
patch: **** malformed patch at line 58: @@ -153,20 +196,212 @@

We have RC7

do u know whats happen-s ?

thanks
Raul
02-07-2009 11:47 PM
Find all posts by this user Quote this message in a reply
Nathariel Offline
Junior Member
*

Posts: 11
Joined: Jan 2009
Reputation: 0
Post: #3
RE: ispcp-vrl-traff improvement
(02-07-2009 11:47 PM)rkhalil Wrote:  Hi!
We trying to apply the patch buts no works,

/var/www/ispcp/engine/traffic# patch -p0 <ispcp-vrl-traff.diff
patching file ispcp-vrl-traff
patch: **** malformed patch at line 58: @@ -153,20 +196,212 @@

We have RC7

do u know whats happen-s ?

thanks
Raul

Yes, it was my fault.
Please try with this patch:
.txt  ispcp-vrl-traff.diff.txt (Size: 5.4 KB / Downloads: 36)

(The diff in the head post is also updated.)
02-10-2009 03:25 AM
Visit this user's website Find all posts by this user Quote this message in a reply
RatS Offline
Project Leader
******

Posts: 1,854
Joined: Oct 2006
Reputation: 17
Post: #4
RE: ispcp-vrl-traff improvement
Please update the ticket as well! Thank you for this, will be in 1.0.1.
02-10-2009 06:43 PM
Visit this user's website Find all posts by this user Quote this message in a reply
rkhalil Offline


Posts: 2
Joined: Feb 2009
Reputation: 0
Post: #5
RE: ispcp-vrl-traff improvement
we test this enhancement and is amazing!! Smile no more memory overhead!

Thanks
Raul
02-11-2009 06:55 AM
Find all posts by this user Quote this message in a reply
vetch101 Offline
Junior Member
*

Posts: 45
Joined: May 2007
Reputation: 0
Post: #6
RE: ispcp-vrl-traff improvement
(02-10-2009 03:25 AM)Nathariel Wrote:  
(02-07-2009 11:47 PM)rkhalil Wrote:  Hi!
We trying to apply the patch buts no works,

/var/www/ispcp/engine/traffic# patch -p0 <ispcp-vrl-traff.diff
patching file ispcp-vrl-traff
patch: **** malformed patch at line 58: @@ -153,20 +196,212 @@

We have RC7

do u know whats happen-s ?

thanks
Raul

Yes, it was my fault.
Please try with this patch:

(The diff in the head post is also updated.)

Hi,

I'm having services crash out because of the issues with memory on a VPS.

I'm running a nightly build between RC6 and RC7.
What do I need to do to run the patch?
I intend to upgrade to 1.0 stable at some point in the not-too-distant future.
Will this patch - a) break my capability to do that? b) need to be reapplied? c) work with any RC version?

Many thanks,

Jx
03-10-2009 06:15 PM
Find all posts by this user Quote this message in a reply
Zothos Offline
Release Manager
*****
Dev Team

Posts: 1,262
Joined: Feb 2007
Reputation: 10
Post: #7
RE: ispcp-vrl-traff improvement
can someone make the patch working with the current trunk? i realy like to see this in our development trunk soon Smile
03-10-2009 06:36 PM
Find all posts by this user Quote this message in a reply
vetch101 Offline
Junior Member
*

Posts: 45
Joined: May 2007
Reputation: 0
Post: #8
RE: ispcp-vrl-traff improvement
Quote:Hi,

I'm having services crash out because of the issues with memory on a VPS.

I'm running a nightly build between RC6 and RC7.
What do I need to do to run the patch?
I intend to upgrade to 1.0 stable at some point in the not-too-distant future.
Will this patch - a) break my capability to do that? b) need to be reapplied? c) work with any RC version?

Many thanks,

Jx

Hi all,

I ran the command
/var/www/ispcp/engine/traffic# patch -p0 <ispcp-vrl-traff.diff
patching file ispcp-vrl-traff

... and the patch seemed to go through...
Will I need to do anything else, like reapplying the patch when I upgrade to 1.0?

Many thanks,

Jx
03-10-2009 07:59 PM
Find all posts by this user Quote this message in a reply
Nathariel Offline
Junior Member
*

Posts: 11
Joined: Jan 2009
Reputation: 0
Post: #9
RE: ispcp-vrl-traff improvement
(03-10-2009 07:59 PM)vetch101 Wrote:  
Quote:Hi,

I'm having services crash out because of the issues with memory on a VPS.

I'm running a nightly build between RC6 and RC7.
What do I need to do to run the patch?
I intend to upgrade to 1.0 stable at some point in the not-too-distant future.
Will this patch - a) break my capability to do that? b) need to be reapplied? c) work with any RC version?

Many thanks,

Jx

Hi all,

I ran the command
/var/www/ispcp/engine/traffic# patch -p0 <ispcp-vrl-traff.diff
patching file ispcp-vrl-traff

... and the patch seemed to go through...
Will I need to do anything else, like reapplying the patch when I upgrade to 1.0?

Many thanks,

Jx

Sorry for the delay, I've been busy in the last weeks.

We are using the patch on 1.00 and there is no need for any changes. It works fine.
You don't need to do something else after you apply the patch, it should be working without problems.

However this patch is not in the trunk yet so if you are intending to upgrade, you should reapply it.

If you have any issues, please post them here.

Thanks
(This post was last modified: 03-24-2009 12:04 AM by Nathariel.)
03-24-2009 12:00 AM
Visit this user's website Find all posts by this user Quote this message in a reply
vetch101 Offline
Junior Member
*

Posts: 45
Joined: May 2007
Reputation: 0
Post: #10
RE: ispcp-vrl-traff improvement
(03-24-2009 12:00 AM)Nathariel Wrote:  
(03-10-2009 07:59 PM)vetch101 Wrote:  
Quote:Hi,

I'm having services crash out because of the issues with memory on a VPS.

I'm running a nightly build between RC6 and RC7.
What do I need to do to run the patch?
I intend to upgrade to 1.0 stable at some point in the not-too-distant future.
Will this patch - a) break my capability to do that? b) need to be reapplied? c) work with any RC version?

Many thanks,

Jx

Hi all,

I ran the command
/var/www/ispcp/engine/traffic# patch -p0 <ispcp-vrl-traff.diff
patching file ispcp-vrl-traff

... and the patch seemed to go through...
Will I need to do anything else, like reapplying the patch when I upgrade to 1.0?

Many thanks,

Jx

Sorry for the delay, I've been busy in the last weeks.

We are using the patch on 1.00 and there is no need for any changes. It works fine.
You don't need to do something else after you apply the patch, it should be working without problems.

However this patch is not in the trunk yet so if you are intending to upgrade, you should reapply it.

If you have any issues, please post them here.

Thanks

Thanks muchly...

As it goes, I had pretty much assumed the same and applied it previously...

The server has been running happily for 11 days now, so I'm thinking the issue has been resolved!

Thanks so much for your help!

Jx
03-24-2009 12:12 AM
Find all posts by this user Quote this message in a reply
Post Reply 


Forum Jump:


User(s) browsing this thread: 1 Guest(s)