"real time" statistics....

[kilburn]

I'm thinking about trying to integrate real-time traffic statistics into ispcp and would like to share my thoughts and get oppinions on how/why or why not, etc. My main motivations are:

1. - The "batched" traffic accounting has proven fragile and annoying with ispcp (there are a bunch of posts about getting 100% usage when the domain traffic "manager" runs).
2. - Logging traffic usage to a mysql database degrades over time, so when you have 1 year of traffic logs in a 300 domains server the querys become painfully slow (more cpu usage, blah blah...).
3. - This one is childish but... real time statistics rock!
   

Well, after some research here and there I've outlined a plan:

1. 1. Use rrdtool or something similar to store the traffic data in the highly efficient rdd format (this would leave us with 50-200Kb of traffic data per domain&service&year wich is WAY less than the current usage with mysql).
2. 2. Process logs as they're generated:
   
   1. - Apache: Create a piped logging program that (1) accounts traffic usage by vhost and stores them in .rrd databases, (2) splits the single server-wide log stream, storing a single log on per-domain folders (ending with a lot less open file descriptors than now).
   2. - Postfix: Modify mailgraph to split the accounting on a per-host basis and add pop3/imap counters too.
   3. - Others: Use an approach similar to the apache one if their log output can be piped to a parser, or similar to the postfix one otherwise.
      
3. 3. Obviously, modify the gui to display the gathered traffic data (and probably graphs!)
   

Ok so... what do you think?