[Fixed] Traffic accounting issues

[aseques]

I am reopening this topic after we found some abnormal problems in our accouting, basically there are problems in some places and the numbers doesn't match, I'll try to keep it updated with the lastest issues here (also the fixed ones)
engine/traffic/ispcp-vrl-traf:
This part still need some kind or working, because some of the data collected is not totally acurate.

Some previous discussions about this:
http://www.isp-control.net/forum/thread-6555.html
More about this:
http://www.isp-control.net/ispcp/ticket/2186

UPDATE, fixed issues
engine/traffic/ispcp-vrl-traf-correction:
As of rev 2578, traf-correction has been deleted.
If we can fix the accounting for each of the processes on the system (http, ftp, etc ..), then there won't be need to use this unfair process (in the case before, it would distribute the 290 missing MB from the accounting to the rest of the hosting users)
Ticket: http://isp-control.net/ispcp/ticket/2197
fixed in rev2578

engine/traffic/ispcp-srv-traf:
This is the script responsible for the general statistics of the server, there's a rule for iptables for each one of the services.
Ticket was: http://isp-control.net/ispcp/ticket/2096
fixed in rev2546

ispcp-vrl-traf is using /var/log/ispcp/logdb to have a pointer to the current log. logdb have serious problems when the logs are rotated because it doesn't update the content.
There is a open ticket with a patch: http://www.isp-control.net/ispcp/ticket/2180
fixed in rev2552

apache rotation
At least in debian, there is a default logrotate file for apache2. The problem with this file is that doesn't keep the history for the whole year, and also that it doesn't run awstats when rotated.
Doing a postrotate script within the apache logrotate would solve this problem.
Ticket: http://isp-control.net/ispcp/ticket/2097
fixed in rev2614

apache+logio
If I stop downloading a file of 300MB in after having downloaded 10M, the logs register a download of 30MB (wich is totally wrong), to fix this we should use the %I %O variables (wich are part of mod_logio) that report the exact amount of traffic.
With the patch provided, you will have to create a variable in ispcp.conf called APACHE_LOGIO_ACTIVE and set to yes.
Also you have to edit the /etc/apache2/ispcp.conf and change the LogFormat to:
LogFormat "%v %b %h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\" %I %O" ispcplog
Ticket: http://isp-control.net/ispcp/ticket/2187
Fixed in r3201

wrong count for dovecot imap
As seen here http://isp-control.net/forum/thread-1071...l#pid81562
Someone should check that everything is ok with courier too.
Ticket http://isp-control.net/ispcp/ticket/2421
Fixed in r3220

[kilburn]

As I see it, your point is correct (applicable to 1.0.3, trunk, and 1.0.2 which used a different ispcp-network script that is also affected). Just one minor question about the patches though: the ispcp-srv-traff path comments out the "sys_command" calls used to retrieve the iptables counters, and I assume that you did this when testing but they should be there on a release version. Am I right?

[aseques]

(12-10-2009 04:33 AM)kilburn Wrote:  As I see it, your point is correct (applicable to 1.0.3, trunk, and 1.0.2 which used a different ispcp-network script that is also affected). Just one minor question about the patches though: the ispcp-srv-traff path comments out the "sys_command" calls used to retrieve the iptables counters, and I assume that you did this when testing but they should be there on a release version. Am I right?

Oh yes, totally forgot about that. Those comments are just to avoid the log being overwrite.

Cheers

[kilburn]

Ok, I was about to merge this patch when I realized this (and its analogous that I'm not pasting):

Code:

-                       $port = $1 if ($line =~ /dpt\:(\d+) *$/);
+                       $port = $1 if ($line =~ /[dpt,spt]\:(\d+) *$/);

Now, the regular expression that you use seems to be incorrect, because [dpt,spt] means "any of the characters inside brackets, this is, either d or p or t or s or comma", whereas you probably wanted to use (dpt|spt), that means either "dpt" or "spt". Your tests went right because, as you're not matching from the beggining of the line, any "t:xx " (xx being a number) matches your regex

Am I right?

[aseques]

Quote:Now, the regular expression that you use seems to be incorrect, because [dpt,spt] means "any of the characters inside brackets, this is, either d or p or t or s or comma", whereas you probably wanted to use (dpt|spt), that means either "dpt" or "spt". Your tests went right because, as you're not matching from the beggining of the line, any "t:xx " (xx being a number) matches your regex

Am I right?

You are absolutely right, shall I update the patch, or will you?

Cheers

[marchaos]

Something wrong, please look there http://www.isp-control.net/forum/thread-...age-2.html

[aseques]

Updated the post with the already fixed issues, still some stuff is pending.
.- Dovecot imap logs
.- Apache real logs with %I %O
... suggestions ....

[aseques]

Update the main ticket with the proposal to delete the ispcp-vrl-traff-correction (http://www.isp-control.net/ispcp/ticket/2197)

[aseques]

Updated main post to reflect the current state of the traffic problems.
(ispcp-vrl-traf-correction has been deleted)

[aseques]

Updated the thread, corrected a wrong link and added a new link pointing to some problems with awstats.