ispcp-vrl-traff improvement

[Nathariel]

Hey all.

I would like to propose an enhancement for ispcp-vrl-traff script, because it is unable to handle large files and could easily crash heavy loaded servers.

The problem is in the gen_log_file function and the way that it creates the temp .diff files when comparing new and .prev logs. Currently, this is done by the following line:

Code:

diff_command("$main::cfg{'CMD_DIFF'} -a $dest_file $dest_file_prev | $main::cfg{'CMD_EGREP'} '^<' | $main::cfg{'CMD_SED'} 's/^< //' 1> $log_file");

Behind the diff stands the problem with the LCS and no matter how optimized the algorithm is, it needs a big amount of memory. I am not going into details what needs diff to do in order to compare two large files but the memory usage is enormous.

Thus, when we have heavily loaded web servers and log processing, this could crash the log processing server, eating all its memory in a second. (Upon such cases,and if the system is still alive, the kernel starts killing processes with higher memory usage - mysql, apache...are the first to be brought down...)

When we speak about high loaded web and mail servers, the fast log rotation is not enough and such a resource waste should be addressed.

I have written a small enhancement which creates a local file db with the last read positions of all logs. Upon invoking, it checks this position and seeks to it in the new log file. After that it reads to the end and writes the output in the temp .diff file.

The operation is more CPU intensive and has almost no memory overhead.

I am attaching a patch with the differences between the original and modified ispcp-vrl-traff,the look of the new gen_log_function and the FSM logic.
I have also submitted a ticket about this issue: http://www.isp-control.net/ispcp/ticket/1623

The algorithm could be improved a little more, but I have tried to save the old function as much as I can.

Please feel free to test / improve this workaround.