About the traffic accounting issues

[sci2tech]

(04-28-2009 05:53 PM)kilburn Wrote:  engine/traffic/ispcp-srv-traf
Noticed problem: On the "sub srv_traff_start_up" function it launches a query equivalent to "SELECT * FROM domain", which results are obviously never used because it doesn't know a thing about domains!

This has the the purpose of instantiate sql connection. But it miss a LIMIT 1, so entire table is selected.

(04-28-2009 05:53 PM)kilburn Wrote:  engine/traffic/ispcp-vrl-traf
Noticed problem: as each traffic type uses separate logfiles, traffic is counted in defferent functions. The problem is that each of this functions pushes the results to mysql, effectively creating 1 insert query and 3 updates for each domain! On a highly loaded table like the domain_traffic one this has a big impact on performance. It shouldn't be difficult to aggregate the data first and then execute a single INSERT

True if in last hour a domain has traffic for http/ftp/mail. There will be execute 2 query(1 Select + 1Update / 1 Insert) for each type of traffic. Multiply with domain number and assume worse case and indeed is a stress for mysql. I`ll try to modify it after I finish what I started already.

(04-28-2009 05:53 PM)kilburn Wrote:  engine/traffic/ispcp-vrl-traf-correction
After reading and re-reading the code, I've concluded that this is trying to make domain_traffic and server_traffic match by evenly distributting the difference (server_traffic - SUM(all domains_traffic)) among all domains.
Noticed "problem": From my point of view this is a nonsense. Obviously, there's traffic that doesn't get caught in logfiles, and the domain and server traffic will never exactly match if you don't perform this artificial "gap removal", but blindly accounting things to the domains is a perfect recipe for annoyed customers, ...:

True, traffic generated by gui is added to costumers traffic, witch is not correct. But imagine following situation -> 1 user offer for download one file of 1GB. Another user hit that file but do not download (cancel the download). In traffic log 1Gb will be added. Here come correction to reflect real traffic. User will not be charged with 1Gb traffic.

(04-28-2009 05:53 PM)kilburn Wrote:  Unsolved things
On earlier ispcp days, I myself proposed to aggregate all the domain_traffic counts per day (we stored record per domain and half an hour), which is the finest scale at which the panel let's you see them. Otherwise, we generate a record for each half hour, up to (24*2*number_of_domains) per day, making the table grow huge on heavily loaded servers and having a noticeable impact on performance. Until today, I thought that the vrl-traf-correction was about this, but now I've realized that...

It`s not true. Traffic is aggregated per day. See line 442 to 458 in ispcp-vrl-traff.
There are other traffic issue like aliases subdomain traffic not been counted.