Problems with the engine scripts

[fred]

Hi,

It seems, when a more than a few users per hour register, the engine (perl) scripts go haywire. We currently are registering 100 users / hour and on the server you see a lot of serv-mngr and dmn-mngr scripts running making load on the server and, what's worse, corrupting the user installations.

We have seen;

- half installed user dirs; missing a bunch of dirs (like htdocs)
- corrupt ispcp.conf apache file
-- half written
-- missing closing tags
-- missing the include file for a user
- corrupt dns files
- linux user / group not made (but the rest is)
- apache/dns done, but user not in the domain table of the db (???)
and more...

Ofcourse with almost all the above apache and bind will not restart anymore and your host will be down and out.

This with the current version of ispcp and pushing 100+ new users / hour, however it starts even at +/-20 /hour.

I thought I report this as it seems no-one tested it under any load? When you do a serious campaign for your hosting company, 20 / hour is not that weird the pull in. Currently (with the fcgi/php bug) and those scripts it is impossible to do this.

Thanks

[kilburn]

Obviously, this is our responsibility to a certain extent. By the architecture of the panel tough, some of the situations you describe should not happen.

On the one hand, from what I know about the inner workings, it is possible that some parts of the domain account are configured, but not others. This may happen because of a mysql/engine asynchrony (new data added while the engine is running). By possible I mean that *I* am not completely sure that this cannot happen.

On the other hand, the panel uses a locking system to prevent multiple engine scripts running at the same time, and therefore corruption of configuration files should not be possible.

As you observed corruption, it is obviously possible. My only explanation tough, is that for some reason, some processes were killed before they could finish their operations, and therefore left corrupted files behind. My bet is that your server was heavily overloaded, totally consumed all available ram+swap, and the kernel had to begun killing processes to make up some space.

May this be the reason? Which are your server specs and their observed loads?

[fred]

It's a 8-way Xeon with 8gb ; the highest load so far was yesterday evening when there were about 200 users trying to register at the same time and load jumped to 30 after which the server could not be reached and had to be rebooted; at that time there were hundreds of ispcp-serv + ispcp-dmn scripts running in the process list.

But the thing is that we are getting those corruptions (and my perl handywork to detect and copy back backups...) happen actually all the time when there are 20-50 users being created and the load is around 1.

[kilburn]

I've just reviewed the locking mechanism that we have in place to prevent multiple request managers operating at the same time. It's essentially crap (view code lines 1439-1476), vulnerable to race conditions because file check and creation is not handled as an atomic action.

I'll path it tomorrow...

[fred]

Ah, I didn't even check Was in a rush; no OS/FS file locking, so yeah that'll be the problem. Thanks for tracking it down.

[kilburn]

I've commited a fix for the locking bug. If you are not using a trunk version, maybe the attached patch may help.

Cheers