About the traffic accounting issues

[kilburn]

Ok, I've noticed something very wrong with traffic counting today and decided to take a look to the whole counting arch, with some surprising findings:

engine/traffic/ispcp-srv-traf
This is the responsible of counting server traffic stastistics using iptables information (so it doesn't know which clients to account it for).
Noticed problem: On the "sub srv_traff_start_up" function it launches a query equivalent to "SELECT * FROM domain", which results are obviously never used because it doesn't know a thing about domains!

engine/traffic/ispcp-vrl-traf
This is the responsible of counting domain traffic stastistics, extracting this info from the logfiles of the corresponding services (mail.log for smtp/pop3/imap, apache's traffic logs for web, etc.).
Noticed problem: as each traffic type uses separate logfiles, traffic is counted in defferent functions. The problem is that each of this functions pushes the results to mysql, effectively creating 1 insert query and 3 updates for each domain! On a highly loaded table like the domain_traffic one this has a big impact on performance. It shouldn't be difficult to aggregate the data first and then execute a single INSERT

engine/traffic/ispcp-vrl-traf-correction
After reading and re-reading the code, I've concluded that this is trying to make domain_traffic and server_traffic match by evenly distributting the difference (server_traffic - SUM(all domains_traffic)) among all domains.
Noticed "problem": From my point of view this is a nonsense. Obviously, there's traffic that doesn't get caught in logfiles, and the domain and server traffic will never exactly match if you don't perform this artificial "gap removal", but blindly accounting things to the domains is a perfect recipe for annoyed customers, like in:

client Wrote:Hey! I've my mail service outsourced anywhere else, but I'm consistently accounted for some mail traffic each day! I think you're also miscounting my web traffic, you unskilled morons!

Moreover, up to now if a host got iptables counting wrong because it used some firewall or whatever, the only problem was that server traffic was wrong. Only the admin noticed it and may decide to just ignore it. With this correction thing, server_traffic - domain_traffic would yield negative results and some reall funny corrections...

Unsolved things
On earlier ispcp days, I myself proposed to aggregate all the domain_traffic counts per day (we stored record per domain and half an hour), which is the finest scale at which the panel let's you see them. Otherwise, we generate a record for each half hour, up to (24*2*number_of_domains) per day, making the table grow huge on heavily loaded servers and having a noticeable impact on performance. Until today, I thought that the vrl-traf-correction was about this, but now I've realized that... it hasn't been implemented! (ispcomm posted some code that did the trick, but I've been unable to find it right now).

I don't want to change such things until hearing the opinion of other devs, so that's why I've decided to post here instead of opening tickets and all this