apache error logs

[peman]

Hello,

Today our apache stopped working for some reason. i restarted the services and now it works. where are the ordinary apache error log file? mine is empty under /var/log/apache2/error.log, has ispCP an error log of its own? or where can find what happened to apache?

Also i need to know if there is some kind of program like Nagios or something better that can notice me if apache is down?

[kilburn]

Quote:Today our apache stopped working for some reason. i restarted the services and now it works. where are the ordinary apache error log file? mine is empty under /var/log/apache2/error.log, has ispCP an error log of its own? or where can find what happened to apache?

/var/log/apache2/users/default-error.log

Quote:Also i need to know if there is some kind of program like Nagios or something better that can notice me if apache is down?

As everyone has it's own preferences regarding monitoring, this is left to the user.

[peman]

I dont understand whats going on here, can someone enlight me?

Quote:[Tue Jan 12 00:01:08 2010] [error] [client 35.55.106.186] Premature end of script headers: index.php
[Tue Jan 12 00:01:12 2010] [notice] mod_fcgid: process 19805 going graceful shutdown, sending SIGTERM
[Tue Jan 12 00:02:37 2010] [error] [client 35.55.106.186] Premature end of script headers: index.php
[Tue Jan 12 00:02:41 2010] [notice] mod_fcgid: process 25506 going graceful shutdown, sending SIGTERM
[Tue Jan 12 01:02:58 2010] [error] [client 35.55.106.186] Premature end of script headers: index.php
[Tue Jan 12 01:03:02 2010] [notice] mod_fcgid: process 31503 going graceful shutdown, sending SIGTERM
[Tue Jan 12 03:03:20 2010] [error] [client 35.55.106.186] Premature end of script headers: index.php
[Tue Jan 12 03:03:21 2010] [notice] mod_fcgid: process 846 going graceful shutdown, sending SIGTERM
[Tue Jan 12 03:28:34 2010] [error] [client 200.79.182.245] File does not exist: /var/www/ispcp/gui/roundcube
[Tue Jan 12 03:28:35 2010] [error] [client 200.79.182.245] File does not exist: /var/www/ispcp/gui/mail
[Tue Jan 12 03:28:35 2010] [error] [client 200.79.182.245] File does not exist: /var/www/ispcp/gui/README
[Tue Jan 12 03:28:35 2010] [error] [client 200.79.182.245] File does not exist: /var/www/ispcp/gui/rc
[Tue Jan 12 06:25:36 2010] [notice] caught SIGTERM, shutting down
[Tue Jan 12 06:25:39 2010] [notice] mod_fcgid: process /var/www/ispcp/gui/index.php(27991) exit(shutting down), terminated by calling exit(), return code: 0
[Tue Jan 12 06:25:41 2010] [notice] suEXEC mechanism enabled (wrapper: /usr/lib/apache2/suexec)
[Tue Jan 12 06:25:42 2010] [notice] Apache/2.2.3 (Debian) configured -- resuming normal operations
[Tue Jan 12 06:25:43 2010] [emerg] mod_fcgid: server is restarted, 3954 must exit
[Tue Jan 12 06:25:43 2010] [emerg] mod_fcgid: server is restarted, 3940 must exit
[Tue Jan 12 06:25:43 2010] [emerg] mod_fcgid: server is restarted, 3940 must exit
[Tue Jan 12 06:25:43 2010] [emerg] mod_fcgid: server is restarted, 3954 must exit
[Tue Jan 12 06:25:43 2010] [emerg] mod_fcgid: server is restarted, 3954 must exit
[Tue Jan 12 06:25:43 2010] [emerg] mod_fcgid: server is restarted, 3954 must exit
[Tue Jan 12 06:25:43 2010] [emerg] mod_fcgid: server is restarted, 3954 must exit
[Tue Jan 12 06:25:44 2010] [emerg] mod_fcgid: server is restarted, 3954 must exit
[Tue Jan 12 06:25:46 2010] [warn] child process 3954 still did not exit, sending a SIGTERM
[Tue Jan 12 06:25:48 2010] [warn] child process 3954 still did not exit, sending a SIGTERM
[Tue Jan 12 06:25:50 2010] [warn] child process 3954 still did not exit, sending a SIGTERM
[Tue Jan 12 06:25:52 2010] [error] child process 3954 still did not exit, sending a SIGKILL
[Tue Jan 12 06:25:53 2010] [notice] caught SIGTERM, shutting down
[Tue Jan 12 08:32:01 2010] [notice] suEXEC mechanism enabled (wrapper: /usr/lib/apache2/suexec)
[Tue Jan 12 08:32:02 2010] [notice] Apache/2.2.3 (Debian) configured -- resuming normal operations
[Tue Jan 12 08:32:12 2010] [notice] mod_fcgid: call /var/www/ispcp/gui/index.php with wrapper /var/www/fcgi/master/php5-fcgi-starter
[Tue Jan 12 08:32:12 2010] [notice] mod_fcgid: server /var/www/ispcp/gui/index.php(21656) started
[Tue Jan 12 09:10:33 2010] [notice] SIGHUP received. Attempting to restart
[Tue Jan 12 09:10:33 2010] [notice] mod_fcgid: process /var/www/ispcp/gui/index.php(21656) exit(shutting down), get stop signal 9
[Tue Jan 12 09:10:37 2010] [notice] Apache/2.2.3 (Debian) configured -- resuming normal operations
[Tue Jan 12 09:10:38 2010] [notice] mod_fcgid: call /var/www/ispcp/gui/admin/manage_users.php with wrapper /var/www/fcgi/master/php5-fcgi-starter
[Tue Jan 12 09:10:38 2010] [notice] mod_fcgid: server /var/www/ispcp/gui/admin/manage_users.php(22639) started

---

btw thank you

[kilburn]

During which timespan was the webserver "down for no reason"?

[peman]

I noticed it today at 8.40 so maybe between 6.00 and 9.00.

[peman]

Quote:As everyone has it's own preferences regarding monitoring, this is left to the user.

Maybe you can recommend something?

[kilburn]

Quote:Maybe you can recommend something?

I use a combination of nagios + collectd (without automatic restarting of services) whereas I know some other people who prefer cacti.

About your log, the weirdness is at 06:25, when apache is restarted 2 times in 10 seconds, so the second one never gets to relaunch it. My guess is that two overnight processes tried to restart the service at the same time and it went wrong the hard way...

[peman]

ok, so it could just be a temporary thing maybe. thank you for the tip you gave me.
thanks!

---

maybe a stupid question but where have you installed nagios and so on? not on the ispcp webserver itself? it must be a external webserver, right?

[kilburn]

You can install it on the server itself, but it's obviously more effective when installed outside. In fact, I've a central server monitor installed inhouse (in our office), which surveys multiple remote (datacenter-located) servers.

False "all servers are down" notices are easy to filter if you setup the dependency infrastructure correctly (just make one "internet" router, so if it goes down you know that your office is out of order, not your servers and that's it).