+- ispCP - Board - Support (http://www.isp-control.net/forum)
+-- Forum: ispCP Omega Development Area (/forum-1.html)
+--- Forum: General discussion (/forum-11.html)
+--- Thread: [Fixed] Traffic accounting issues (/thread-8721.html)
Pages: 1 2
[Fixed] Traffic accounting issues - aseques - 12-10-2009 03:28 AM
I am reopening this topic after we found some abnormal problems in our accouting, basically there are problems in some places and the numbers doesn't match, I'll try to keep it updated with the lastest issues here (also the fixed ones)
engine/traffic/ispcp-vrl-traf:
This part still need some kind or working, because some of the data collected is not totally acurate.
Some previous discussions about this:
http://www.isp-control.net/forum/thread-6555.html
More about this:
http://www.isp-control.net/ispcp/ticket/2186
UPDATE, fixed issues
engine/traffic/ispcp-vrl-traf-correction:
As of rev 2578, traf-correction has been deleted.
If we can fix the accounting for each of the processes on the system (http, ftp, etc ..), then there won't be need to use this unfair process (in the case before, it would distribute the 290 missing MB from the accounting to the rest of the hosting users)
Ticket: http://isp-control.net/ispcp/ticket/2197
fixed in rev2578
engine/traffic/ispcp-srv-traf:
This is the script responsible for the general statistics of the server, there's a rule for iptables for each one of the services.
Ticket was: http://isp-control.net/ispcp/ticket/2096
fixed in rev2546
ispcp-vrl-traf is using /var/log/ispcp/logdb to have a pointer to the current log. logdb have serious problems when the logs are rotated because it doesn't update the content.
There is a open ticket with a patch: http://www.isp-control.net/ispcp/ticket/2180
fixed in rev2552
apache rotation
At least in debian, there is a default logrotate file for apache2. The problem with this file is that doesn't keep the history for the whole year, and also that it doesn't run awstats when rotated.
Doing a postrotate script within the apache logrotate would solve this problem.
Ticket: http://isp-control.net/ispcp/ticket/2097
fixed in rev2614
apache+logio
If I stop downloading a file of 300MB in after having downloaded 10M, the logs register a download of 30MB (wich is totally wrong), to fix this we should use the %I %O variables (wich are part of mod_logio) that report the exact amount of traffic.
With the patch provided, you will have to create a variable in ispcp.conf called APACHE_LOGIO_ACTIVE and set to yes.
Also you have to edit the /etc/apache2/ispcp.conf and change the LogFormat to:
LogFormat "%v %b %h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\" %I %O" ispcplog
Ticket: http://isp-control.net/ispcp/ticket/2187
Fixed in r3201
wrong count for dovecot imap
As seen here http://isp-control.net/forum/thread-10714-post-81562.html#pid81562
Someone should check that everything is ok with courier too.
Ticket http://isp-control.net/ispcp/ticket/2421
Fixed in r3220
RE: Traffic accounting issues - kilburn - 12-10-2009 04:33 AM
As I see it, your point is correct (applicable to 1.0.3, trunk, and 1.0.2 which used a different ispcp-network script that is also affected). Just one minor question about the patches though: the ispcp-srv-traff path comments out the "sys_command" calls used to retrieve the iptables counters, and I assume that you did this when testing but they should be there on a release version. Am I right?
RE: Traffic accounting issues - aseques - 12-10-2009 07:17 AM
(12-10-2009 04:33 AM)kilburn Wrote: As I see it, your point is correct (applicable to 1.0.3, trunk, and 1.0.2 which used a different ispcp-network script that is also affected). Just one minor question about the patches though: the ispcp-srv-traff path comments out the "sys_command" calls used to retrieve the iptables counters, and I assume that you did this when testing but they should be there on a release version. Am I right?Oh yes, totally forgot about that. Those comments are just to avoid the log being overwrite.
Cheers
RE: Traffic accounting issues - kilburn - 12-11-2009 05:16 AM
Ok, I was about to merge this patch when I realized this (and its analogous that I'm not pasting):
Code:
- $port = $1 if ($line =~ /dpt\:(\d+) *$/);
+ $port = $1 if ($line =~ /[dpt,spt]\:(\d+) *$/);Now, the regular expression that you use seems to be incorrect, because [dpt,spt] means "any of the characters inside brackets, this is, either d or p or t or s or comma", whereas you probably wanted to use (dpt|spt), that means either "dpt" or "spt". Your tests went right because, as you're not matching from the beggining of the line, any "t:xx " (xx being a number) matches your regex
Am I right?
RE: Traffic accounting issues - aseques - 12-11-2009 08:36 PM
Quote:Now, the regular expression that you use seems to be incorrect, because [dpt,spt] means "any of the characters inside brackets, this is, either d or p or t or s or comma", whereas you probably wanted to use (dpt|spt), that means either "dpt" or "spt". Your tests went right because, as you're not matching from the beggining of the line, any "t:xx " (xx being a number) matches your regexYou are absolutely right, shall I update the patch, or will you?
Am I right?
Cheers
RE: Traffic accounting issues - marchaos - 01-17-2010 10:20 AM
Something wrong, please look there http://www.isp-control.net/forum/thread-9220-page-2.html
RE: Traffic accounting issues - aseques - 01-26-2010 08:09 PM
Updated the post with the already fixed issues, still some stuff is pending.
.- Dovecot imap logs
.- Apache real logs with %I %O
... suggestions ....
RE: Traffic accounting issues - aseques - 02-04-2010 06:47 PM
Update the main ticket with the proposal to delete the ispcp-vrl-traff-correction (http://www.isp-control.net/ispcp/ticket/2197)
RE: Traffic accounting issues - aseques - 02-09-2010 12:38 AM
Updated main post to reflect the current state of the traffic problems.
(ispcp-vrl-traf-correction has been deleted)
RE: Traffic accounting issues - aseques - 02-19-2010 11:39 PM
Updated the thread, corrected a wrong link and added a new link pointing to some problems with awstats.