[SOLVED has patch(sorta)] ispcp-dmn-mngr using 3.8g virt (swap)

[supaplex]

top - 10:56:39 up 34 days, 6:31, 3 users, load average: 11.11, 15.45, 16.15
Tasks: 403 total, 6 running, 392 sleeping, 0 stopped, 5 zombie
Cpu(s): 21.3%us, 34.6%sy, 0.0%ni, 8.0%id, 11.3%wa, 0.3%hi, 24.6%si, 0.0%st
Mem: 3921820k total, 3734976k used, 186844k free, 19504k buffers
Swap: 4883720k total, 3106912k used, 1776808k free, 155616k cached

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
24684 root 18 0 3866m 2.1g 1576 D 0 55.4 1:20.24 ispcp-dmn-mngr

This is running ispcp-omega-1.0.0-rc5 on Debian Etch.

Domains are taking 30 minutes or more to remove/add etc. sometimes I have to add a user manually just so apache will start correctly.

Are these issues fixed in rc7? What changelog or tickets apply from Trac?

Is there something else I can provide to help investigate what's going on?

Thank you,


Scott Edwards

[gilbert]

(01-22-2009 03:07 AM)supaplex Wrote:  top - 10:56:39 up 34 days, 6:31, 3 users, load average: 11.11, 15.45, 16.15
Tasks: 403 total, 6 running, 392 sleeping, 0 stopped, 5 zombie
Cpu(s): 21.3%us, 34.6%sy, 0.0%ni, 8.0%id, 11.3%wa, 0.3%hi, 24.6%si, 0.0%st
Mem: 3921820k total, 3734976k used, 186844k free, 19504k buffers
Swap: 4883720k total, 3106912k used, 1776808k free, 155616k cached

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
24684 root 18 0 3866m 2.1g 1576 D 0 55.4 1:20.24 ispcp-dmn-mngr

This is running ispcp-omega-1.0.0-rc5 on Debian Etch.

Domains are taking 30 minutes or more to remove/add etc. sometimes I have to add a user manually just so apache will start correctly.

Are these issues fixed in rc7? What changelog or tickets apply from Trac?

Is there something else I can provide to help investigate what's going on?

Thank you,


Scott Edwards

Just to add on to this:

deleting a domain was taking very long.
I did kill one of the ispcp-dmn-mngr processes when it was not finished.
Since then we have this problem on not being able to add or delete any domains.
Everything else seems to be working ok.
mail
web
ftp
etc.

This I think caused there to be some inconsistency in the database, conf files, etc.

Is there a script to check the consistency of the system?

If we try and add or delete any domains the memory usage just keeps going up and up.

Consequently the load keeps going up....

We already ave the DEBUG=1 set in the /etc/ispcp/ispcp.conf file.

when we run /var/www/ispcp/engine/ispcp-rqst-mngr

I get

/var/www/ispcp/engine/ispcp-rqst-mngr
DEBUG: push_el() sub_name: mngr_start_up(), msg: Starting...
DEBUG: push_el() sub_name: lock_system(), msg: Starting...
DEBUG: push_el() sub_name: sys_command(), msg: Starting...
DEBUG: push_el() sub_name: sys_command('`which touch` /var/run/ispcp.lock'), msg: Ending...
DEBUG: push_el() sub_name: lock_system(), msg: Ending...
DEBUG: push_el() sub_name: get_conf(), msg: Starting...
DEBUG: push_el() sub_name: get_file(), msg: Starting...
DEBUG: push_el() sub_name: get_file(), msg: Ending...
DEBUG: push_el() sub_name: setup_main_vars(), msg: Starting...
DEBUG: push_el() sub_name: decrypt_db_password(), msg: Starting...
DEBUG: push_el() sub_name: decrypt_db_password(), msg: Ending...
DEBUG: push_el() sub_name: setup_main_vars(), msg: Ending...
DEBUG: push_el() sub_name: get_conf(), msg: Ending...
DEBUG: push_el() sub_name: doSQL(), msg: Starting...
DEBUG: push_el() sub_name: doSQL(), msg: Ending...
DEBUG: push_el() sub_name: mngr_start_up(), msg: Ending...
DEBUG: push_el() sub_name: mngr_engine(), msg: Starting...
DEBUG: push_el() sub_name: doSQL(), msg: Starting...
DEBUG: push_el() sub_name: doSQL(), msg: Ending...
DEBUG: push_el() sub_name: doSQL(), msg: Starting...
DEBUG: push_el() sub_name: doSQL(), msg: Ending...
DEBUG: push_el() sub_name: doSQL(), msg: Starting...
DEBUG: push_el() sub_name: doSQL(), msg: Ending...
DEBUG: push_el() sub_name: doSQL(), msg: Starting...
DEBUG: push_el() sub_name: doSQL(), msg: Ending...
DEBUG: push_el() sub_name: sys_command(), msg: Starting...
DEBUG: push_el() sub_name: sys_command('/var/www/ispcp/engine/ispcp-serv-mngr 2 0 0 0 1>/var/log/ispcp/ispcp-serv-mngr.stdout 2>/var/log/ispcp/ispcp-serv-mngr.stderr'), msg: Ending...
DEBUG: push_el() sub_name: doSQL(), msg: Starting...
DEBUG: push_el() sub_name: doSQL(), msg: Ending...
DEBUG: push_el() sub_name: mngr_engine(), msg: processing 171, adamin.com, delete.
DEBUG: push_el() sub_name: sys_command(), msg: Starting...
DEBUG: push_el() sub_name: sys_command('/var/www/ispcp/engine/ispcp-dmn-mngr 171 1>/var/log/ispcp/ispcp-dmn-mngr.stdout 2>/var/log/ispcp/ispcp-dmn-mngr.stderr'), msg: Ending...
DEBUG: push_el() sub_name: doSQL(), msg: Starting...


at this point the mem usage keeps going up and the command never finishes.

Any help would be appreciated.

Thanks,
Gilbert.

[supaplex]

I think we've solved this issue.
I apologize for no unified diff, but I didn't back up all the scripts :)

joe:/var/www/ispcp/engine# wc /etc/apache2/sites-available/ispcp.conf
16783 51294 745193 /etc/apache2/sites-available/ispcp.conf
joe:/var/www/ispcp/engine# wc last-working-apache2.ispcp.conf
19052031 51294 19780406 last-working-apache2.ispcp.conf

See how we're using the same number of words? Scary...

All in all, if someone runs into this on rc5, or has similar amount of newlines in the config, perhaps they should delete all those extra newlines. :) I'm going to tweak the code to let the normal ones back in, in preparation for upgrade.

Geeks and desperate sysadmins, keep reading. :)

ispcp_common_methods.pl get_file was using

@fdata= <F>;

The ispcp.conf file has a few megabytes worth of newlines only at the tail end.

Now it's using

push_el(\@main::el, 'get_file()', "DEBUG: Opening '$fname' now.");
my $res = open(F, '<', $fname);

if (!defined($res)) {
push_el(\@main::el, 'get_file()', "ERROR: Can't open '$fname' for reading: $!");
return 1;
}

my @fdata;

if ($fname =~ m/ispcp.conf/) {
while (<F>) {
if (length($_) le 1) {
next;
}
# print length($_)." ";
push (@fdata,$_);
}
} else {
@fdata= <F>;
}

push_el(\@main::el, 'get_file()', "DEBUG: Closing '$fname' now.");
close(F);


After that clean up, domains were able to add. However, they were not able to delete. The stdout error message said get_tag() | ERROR: '# httpd [example.com] dmn group entry BEGIN.
' ne '# httpd [example.com] dmn group entry END.

', '# httpd [example.com] dmn group entry BEGIN.
' or '# httpd [example.com] dmn group entry END.

At the first 10 glances, ne means not equal to me. This is the new code I wrote for get_tag. (commented push_el line is the old one)

if ($bt_pos < 0 || $et_pos < 0) {

# code uses 'ne' as if it were perl ... that's just crewel to my graymatter. ouch!
# push_el(\@main::el, 'get_tag()', "ERROR: '$bt' ne '$et', '$bt' or '$et' missing in src !");

push_el(\@main::el, 'get_tag()', "ERROR: No BEGIN marker in ".length($src)." lines of config block: $bt")
if ($bt_pos < 0);
push_el(\@main::el, 'get_tag()', "ERROR: No END marker in ".length($src)." lines of config block: $et")
if ($et_pos < 0);

return (-5, '');

}

When the obvious wasn't comparing, I rolled back in the code (perldoc -f caller is my friend). Eventually I found we were called by dmn_del_httpd_cfg_data.

dmn_del_httpd_cfg_data makes a call to del_tag which needed a newline deleted here for domains to compare.

"$dg_e_val\n", # <<< bad newline! can't delete domains this way. grr

Okay, I don't suggest anyone but the developers tackle the issue on the above line. Naturally, from the first post, we see that gigabytes and gigabytes of ram is a bad thing to waste. :) Had I known what the underlying issue was before I fixed it, this little detail would be of no matter.