Apache just stopped :O!!

[KristianOellegaard]

Hi there

This morning, a customer of mine, called me and said his site was offline, i connected and i saw that apache was stopped, somehow.

I hope you can help me, cause i think its very weird. The customer says that he was clicking around on his website, the very moment apache stopped. Here is the log:

[Sun Jul 13 06:25:38 2008] [notice] caught SIGWINCH, shutting down gracefully
[Sun Jul 13 06:25:52 2008] [alert] (2)No such file or directory: FastCGI: read() from pipe failed (0)
[Sun Jul 13 06:25:52 2008] [alert] (2)No such file or directory: FastCGI: the PM is shutting down, Apache seems to have disappeared - bye
[Sun Jul 13 09:11:14 2008] [notice] suEXEC mechanism enabled (wrapper: /usr/lib/apache2/suexec)
[Sun Jul 13 09:11:14 2008] [notice] FastCGI: wrapper mechanism enabled (wrapper: /usr/lib/apache2/suexec)
[Sun Jul 13 09:11:14 2008] [notice] FastCGI: process manager initialized (pid 32713)
[Sun Jul 13 09:11:14 2008] [warn] FastCGI: server "/var/www/fcgi/master/php5-fcgi-starter" (uid 2000, gid 2000) started (pid 32714)
[Sun Jul 13 09:11:14 2008] [notice] mod_python: Creating 8 session mutexes based on 6 max processes and 25 max threads.
[Sun Jul 13 09:11:14 2008] [notice] mod_python: using mutex_directory /tmp
[Sun Jul 13 09:11:15 2008] [notice] Apache/2.2.4 (Ubuntu) mod_fastcgi/2.4.2 mod_python/3.3.1 Python/2.5.1 mod_ssl/2.2.4 OpenSSL/0.9.8e configured -- resuming normal operations
[Sun Jul 13 11:03:20 2008] [notice] SIGHUP received. Attempting to restart
[Sun Jul 13 11:03:20 2008] [error] VirtualHost 78.46.36.45:0 -- mixing * ports and non-* ports with a NameVirtualHost address is not supported, proceeding with undefined results
[Sun Jul 13 11:03:20 2008] [notice] FastCGI: wrapper mechanism enabled (wrapper: /usr/lib/apache2/suexec)
[Sun Jul 13 11:03:20 2008] [notice] FastCGI: process manager initialized (pid 995)
[Sun Jul 13 11:03:20 2008] [warn] FastCGI: server "/var/www/fcgi/master/php5-fcgi-starter" (uid 2000, gid 2000) started (pid 996)
[Sun Jul 13 11:03:20 2008] [notice] mod_python: Creating 8 session mutexes based on 3 max processes and 25 max threads.
[Sun Jul 13 11:03:20 2008] [notice] mod_python: using mutex_directory /tmp
[Sun Jul 13 11:03:21 2008] [notice] Apache/2.2.4 (Ubuntu) mod_fastcgi/2.4.2 mod_python/3.3.1 Python/2.5.1 mod_ssl/2.2.4 OpenSSL/0.9.8e configured -- resuming normal operations
[Sun Jul 13 18:06:21 2008] [warn] FastCGI: (dynamic) server "/var/www/fcgi/scandichorse.dk/php5-fcgi-starter" (uid 2004, gid 2004) started (pid 3470)
[Sun Jul 13 18:55:38 2008] [warn] FastCGI: (dynamic) server "/var/www/fcgi/aalestrupoptik.dk/php5-fcgi-starter" (uid 2002, gid 2002) started (pid 3677)


I hope you will help me

regards

Kristian

[rauschr]

change this line in
/etc/init.d/apache2/mods_enable/@fastcgi_iscp.conf

<IfModule mod_fastcgi.c>
FastCgiWrapper On
FastCgiIpcDir /var/lib/apache2/fastcgi2

FastCgiConfig -minProcesses 1 \
-maxProcesses 400 \
-maxClassProcesses 15 \
-multiThreshold 80 \
-killInterval 60 \
-startDelay 5 \
-restart-delay 10 \
-idle-timeout 120 \
-singleThreshold 100
# -autoUpdate \
# -pass-header HTTP_AUTHORIZATION

FastCgiServer /var/www/fcgi/master/php5-fcgi-starter -user vu2000 -group vu2000

#
# PHP5 SUPPORT
#

AddHandler php-fastcgi .php .php5

<Location /php5/php5-fcgi-starter>
SetHandler fastcgi-script
Options +ExecCGI
</Location>

Action php-fastcgi /php5/php5-fcgi-starter
AddType application/x-httpd-php .php .php5
</IfModule>szerver2:/etc/apache2/mods-enabled#

and change in
/etc/init.d/apache2/apache2.conf
KeepAlive=Off

[nicolas]

Hi,

I just got exactly the same problem on a clean Ubuntu 8.04 64bits / ISCP Omega RC6. Here is the error log:

Quote:[Sun Aug 24 06:27:04 2008] [notice] FastCGI: wrapper mechanism enabled (wrapper: /usr/lib/apache2/suexec)
[Sun Aug 24 06:27:04 2008] [notice] FastCGI: process manager initialized (pid 9471)
[Sun Aug 24 06:27:04 2008] [notice] Apache/2.2.8 (Ubuntu) mod_fastcgi/2.4.6 configured -- resuming normal operations
[Sun Aug 24 06:27:04 2008] [warn] FastCGI: server "/var/www/fcgi/master/php5-fcgi-starter" (uid 2000, gid 2000) started (pid 9472)
[Sun Aug 24 06:27:04 2008] [notice] caught SIGWINCH, shutting down gracefully
[Sun Aug 24 06:27:22 2008] [alert] (2)No such file or directory: FastCGI: read() from pipe failed (0)
[Sun Aug 24 06:27:22 2008] [alert] (2)No such file or directory: FastCGI: the PM is shutting down, Apache seems to have disappeared - bye

I am going to try the suggested changes.

[rauschr]

Update the php verson to 5.2.5

and change
..
-restart-delay 10\
-idle-timeout 120
#-singleThreshold 100
# -autoUpdate \
..

[nicolas]

I just noticed that the problem is related to log rotation. Always happens near 6:30AM. Apache stops after log rotation and does not restart.

Error.log:

Quote:[Wed Aug 27 06:33:34 2008] [notice] SIGUSR1 received. Doing graceful restart
[Wed Aug 27 06:33:34 2008] [notice] FastCGI: wrapper mechanism enabled (wrapper: /usr/lib/apache2/suexec)
[Wed Aug 27 06:33:34 2008] [notice] FastCGI: process manager initialized (pid 10839)
[Wed Aug 27 06:33:34 2008] [notice] Apache/2.2.8 (Ubuntu) mod_fastcgi/2.4.6 configured -- resuming normal operations
[Wed Aug 27 06:33:34 2008] [warn] FastCGI: server "/var/www/fcgi/master/php5-fcgi-starter" (uid 2000, gid 2000) started (pid 10840)
[Wed Aug 27 06:33:34 2008] [notice] caught SIGWINCH, shutting down gracefully

[rauschr]

what is your php version?
how many trafic in your server / day ?

what in your apache2.conf?

[nicolas]

rauschr Wrote:what is your php version?
how many trafic in your server / day ?

what in your apache2.conf?

Crash is happening at log rotation time. Logs are rotated but server dies and does not restart... I have tried to replace the "apache2 reload" by a "apache2 restart" in logrotate.d/apache2... I hope it will fix the problem...

There is a low traffic on the server atm because I have not yet (hopefully) migrated all my websites on the server. Server load should not be a problem (Xeon Quad Core with 8Gb RAM).

I am running Ubuntu 8.04LTS 64bits

Server version: Apache/2.2.8 (Ubuntu)

PHP 5.2.6-2ubuntu3 with Suhosin-Patch 0.9.6.2 (cgi-fcgi) (built: Aug 20 2008 15:59:29)

Default apache2.conf :

Quote:#
# Based upon the NCSA server configuration files originally by Rob McCool.
#
# This is the main Apache server configuration file. It contains the
# configuration directives that give the server its instructions.
# See http://httpd.apache.org/docs/2.2/ for detailed information about
# the directives.
#
# Do NOT simply read the instructions in here without understanding
# what they do. They're here only as hints or reminders. If you are unsure
# consult the online docs. You have been warned.
#
# The configuration directives are grouped into three basic sections:
# 1. Directives that control the operation of the Apache server process as a
# whole (the 'global environment').
# 2. Directives that define the parameters of the 'main' or 'default' server,
# which responds to requests that aren't handled by a virtual host.
# These directives also provide default values for the settings
# of all virtual hosts.
# 3. Settings for virtual hosts, which allow Web requests to be sent to
# different IP addresses or hostnames and have them handled by the
# same Apache server process.
#
# Configuration and logfile names: If the filenames you specify for many
# of the server's control files begin with "/" (or "drive:/" for Win32), the
# server will use that explicit path. If the filenames do *not* begin
# with "/", the value of ServerRoot is prepended -- so "/var/log/apache2/foo.log"
# with ServerRoot set to "" will be interpreted by the
# server as "//var/log/apache2/foo.log".
#

### Section 1: Global Environment
#
# The directives in this section affect the overall operation of Apache,
# such as the number of concurrent requests it can handle or where it
# can find its configuration files.
#

#
# ServerRoot: The top of the directory tree under which the server's
# configuration, error, and log files are kept.
#
# NOTE! If you intend to place this on an NFS (or otherwise network)
# mounted filesystem then please read the LockFile documentation (available
# at <URL:http://httpd.apache.org/docs-2.1/mod/mpm_common.html#lockfile>);
# you will save yourself a lot of trouble.
#
# Do NOT add a slash at the end of the directory path.
#
ServerRoot "/etc/apache2"

#
# The accept serialization lock file MUST BE STORED ON A LOCAL DISK.
#
#<IfModule !mpm_winnt.c>
#<IfModule !mpm_netware.c>
LockFile /var/lock/apache2/accept.lock
#</IfModule>
#</IfModule>

#
# PidFile: The file in which the server should record its process
# identification number when it starts.
# This needs to be set in /etc/apache2/envvars
#
PidFile ${APACHE_PID_FILE}

#
# Timeout: The number of seconds before receives and sends time out.
#
Timeout 300

#
# KeepAlive: Whether or not to allow persistent connections (more than
# one request per connection). Set to "Off" to deactivate.
#
KeepAlive On

#
# MaxKeepAliveRequests: The maximum number of requests to allow
# during a persistent connection. Set to 0 to allow an unlimited amount.
# We recommend you leave this number high, for maximum performance.
#
MaxKeepAliveRequests 100

#
# KeepAliveTimeout: Number of seconds to wait for the next request from the
# same client on the same connection.
#
KeepAliveTimeout 15

##
## Server-Pool Size Regulation (MPM specific)
##

# prefork MPM
# StartServers: number of server processes to start
# MinSpareServers: minimum number of server processes which are kept spare
# MaxSpareServers: maximum number of server processes which are kept spare
# MaxClients: maximum number of server processes allowed to start
# MaxRequestsPerChild: maximum number of requests a server process serves
<IfModule mpm_prefork_module>
StartServers 5
MinSpareServers 5
MaxSpareServers 10
MaxClients 150
MaxRequestsPerChild 0
</IfModule>

# worker MPM
# StartServers: initial number of server processes to start
# MaxClients: maximum number of simultaneous client connections
# MinSpareThreads: minimum number of worker threads which are kept spare
# MaxSpareThreads: maximum number of worker threads which are kept spare
# ThreadsPerChild: constant number of worker threads in each server process
# MaxRequestsPerChild: maximum number of requests a server process serves
<IfModule mpm_worker_module>
StartServers 2
MaxClients 150
MinSpareThreads 25
MaxSpareThreads 75
ThreadsPerChild 25
MaxRequestsPerChild 0
</IfModule>

# These need to be set in /etc/apache2/envvars
User ${APACHE_RUN_USER}
Group ${APACHE_RUN_GROUP}

#
# AccessFileName: The name of the file to look for in each directory
# for additional configuration directives. See also the AllowOverride
# directive.
#

AccessFileName .htaccess

#
# The following lines prevent .htaccess and .htpasswd files from being
# viewed by Web clients.
#
<Files ~ "^\.ht">
Order allow,deny
Deny from all
</Files>

#
# DefaultType is the default MIME type the server will use for a document
# if it cannot otherwise determine one, such as from filename extensions.
# If your server contains mostly text or HTML documents, "text/plain" is
# a good value. If most of your content is binary, such as applications
# or images, you may want to use "application/octet-stream" instead to
# keep browsers from trying to display binary files as though they are
# text.
#
DefaultType text/plain


#
# HostnameLookups: Log the names of clients or just their IP addresses
# e.g., http://www.apache.org (on) or 204.62.129.132 (off).
# The default is off because it'd be overall better for the net if people
# had to knowingly turn this feature on, since enabling it means that
# each client request will result in AT LEAST one lookup request to the
# nameserver.
#
HostnameLookups Off

# ErrorLog: The location of the error log file.
# If you do not specify an ErrorLog directive within a <VirtualHost>
# container, error messages relating to that virtual host will be
# logged here. If you *do* define an error logfile for a <VirtualHost>
# container, that host's errors will be logged there and not here.
#
ErrorLog /var/log/apache2/error.log

#
# LogLevel: Control the number of messages logged to the error_log.
# Possible values include: debug, info, notice, warn, error, crit,
# alert, emerg.
#
LogLevel warn

# Include module configuration:
Include /etc/apache2/mods-enabled/*.load
Include /etc/apache2/mods-enabled/*.conf

# Include all the user configurations:
Include /etc/apache2/httpd.conf

# Include ports listing
Include /etc/apache2/ports.conf

#
# The following directives define some format nicknames for use with
# a CustomLog directive (see below).
# If you are behind a reverse proxy, you might want to change %h into %{X-Forwarded-For}i
#
LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" combined
LogFormat "%h %l %u %t \"%r\" %>s %b" common
LogFormat "%{Referer}i -> %U" referer
LogFormat "%{User-agent}i" agent

#
# ServerTokens
# This directive configures what you return as the Server HTTP response
# Header. The default is 'Full' which sends information about the OS-Type
# and compiled in modules.
# Set to one of: Full | OS | Minor | Minimal | Major | Prod
# where Full conveys the most information, and Prod the least.
#
ServerTokens Full

#
# Optionally add a line containing the server version and virtual host
# name to server-generated pages (internal error documents, FTP directory
# listings, mod_status and mod_info output etc., but not CGI generated
# documents or custom error documents).
# Set to "EMail" to also include a mailto: link to the ServerAdmin.
# Set to one of: On | Off | EMail
#
ServerSignature On



#
# Customizable error responses come in three flavors:
# 1) plain text 2) local redirects 3) external redirects
#
# Some examples:
#ErrorDocument 500 "The server made a boo boo."
#ErrorDocument 404 /missing.html
#ErrorDocument 404 "/cgi-bin/missing_handler.pl"
#ErrorDocument 402 http://www.example.com/subscription_info.html
#

#
# Putting this all together, we can internationalize error responses.
#
# We use Alias to redirect any /error/HTTP_<error>.html.var response to
# our collection of by-error message multi-language collections. We use
# includes to substitute the appropriate text.
#
# You can modify the messages' appearance without changing any of the
# default HTTP_<error>.html.var files by adding the line:
#
# Alias /error/include/ "/your/include/path/"
#
# which allows you to create your own set of files by starting with the
# /usr/share/apache2/error/include/ files and copying them to /your/include/path/,
# even on a per-VirtualHost basis. The default include files will display
# your Apache version number and your ServerAdmin email address regardless
# of the setting of ServerSignature.
#
# The internationalized error documents require mod_alias, mod_include
# and mod_negotiation. To activate them, uncomment the following 30 lines.

# Alias /error/ "/usr/share/apache2/error/"
#
# <Directory "/usr/share/apache2/error">
# AllowOverride None
# Options IncludesNoExec
# AddOutputFilter Includes html
# AddHandler type-map var
# Order allow,deny
# Allow from all
# LanguagePriority en cs de es fr it nl sv pt-br ro
# ForceLanguagePriority Prefer Fallback
# </Directory>
#
# ErrorDocument 400 /error/HTTP_BAD_REQUEST.html.var
# ErrorDocument 401 /error/HTTP_UNAUTHORIZED.html.var
# ErrorDocument 403 /error/HTTP_FORBIDDEN.html.var
# ErrorDocument 404 /error/HTTP_NOT_FOUND.html.var
# ErrorDocument 405 /error/HTTP_METHOD_NOT_ALLOWED.html.var
# ErrorDocument 408 /error/HTTP_REQUEST_TIME_OUT.html.var
# ErrorDocument 410 /error/HTTP_GONE.html.var
# ErrorDocument 411 /error/HTTP_LENGTH_REQUIRED.html.var
# ErrorDocument 412 /error/HTTP_PRECONDITION_FAILED.html.var
# ErrorDocument 413 /error/HTTP_REQUEST_ENTITY_TOO_LARGE.html.var
# ErrorDocument 414 /error/HTTP_REQUEST_URI_TOO_LARGE.html.var
# ErrorDocument 415 /error/HTTP_UNSUPPORTED_MEDIA_TYPE.html.var
# ErrorDocument 500 /error/HTTP_INTERNAL_SERVER_ERROR.html.var
# ErrorDocument 501 /error/HTTP_NOT_IMPLEMENTED.html.var
# ErrorDocument 502 /error/HTTP_BAD_GATEWAY.html.var
# ErrorDocument 503 /error/HTTP_SERVICE_UNAVAILABLE.html.var
# ErrorDocument 506 /error/HTTP_VARIANT_ALSO_VARIES.html.var



# Include of directories ignores editors' and dpkg's backup files,
# see README.Debian for details.

# Include generic snippets of statements
Include /etc/apache2/conf.d/

# Include the virtual host configurations:
Include /etc/apache2/sites-enabled/

[rauschr]

you have a find in apache2.conf
uncomment this lines
and add ExtendedStatus On

<ifmodule mod_status.c>
<location /server-status>
SetHandler server-status
</location>
....

after you'll see the apache tread, if you type a webbrowser your ip and /server-status
examp. : 87.229.2.55/server-status
some website make a lot ordering width not too much traffic.
they run out it apache threads
you can find the domain in the server-status list (80%),what many times can be found
that is a advertisement server swich off!

or DOS you are the victim of an attack

[nicolas]

rauschr Wrote:Update the php verson to 5.2.5

and change
..
-restart-delay 10\
-idle-timeout 120
#-singleThreshold 100
# -autoUpdate \
..

Does not work. Apache still dying instead of restarting.

[nicolas]

rauschr Wrote:you have a find in apache2.conf
uncomment this lines
and add ExtendedStatus On

<ifmodule mod_status.c>
<location /server-status>
SetHandler server-status
</location>
....

after you'll see the apache tread, if you type a webbrowser your ip and /server-status
examp. : 87.229.2.55/server-status
some website make a lot ordering width not too much traffic.
they run out it apache threads
you can find the domain in the server-status list (80%),what many times can be found
that is a advertisement server swich off!

or DOS you are the victim of an attack

Server-status is up, and Apache server is running fine. No overload, or other problem.

Anyway I just noticed ownerships and rights in /var/log/apache2 were a mess. I am trying to investigate any possibilities. Given the fact apache is dying after log rotation that may be a path to investigate. Could anyone post which ownerships and mode should be used for /var/log/apache2 directory and files inside?

Currently I have:
/var/log
drwxr-x--- 4 root adm 16384 2008-09-01 07:29 apache2

/var/log/apache
drwxr-xr-x 2 root root 4096 2008-08-21 21:42 backup
drwxr-xr-x 2 www-data www-data 20480 2008-09-01 07:29 users
-rw-r----- 1 root adm 5328 2008-09-01 13:13 suexec.log
-rw-r----- 1 root adm 99968 2008-08-30 19:15 suexec.log.1
-rw-r--r-- 1 root www-data 313 2008-08-22 17:39 suexec.log.2.gz
-rw-r----- 1 root adm 14696714 2008-09-01 13:31 site1.com-combined.log
-rw-r----- 1 root adm 67363979 2008-08-31 07:31 site1.com-combined.log.1
-rw-r--r-- 1 root root 265528 2008-08-27 06:38 site1.com-combined.log.2.gz
-rw-r----- 1 root adm 278791 2008-09-01 13:31 site1.com-traf.log
-rw-r----- 1 root adm 1316532 2008-08-31 07:31 site1.com-traf.log.1
-rw-r--r-- 1 root root 21991 2008-08-27 06:38 site1.com-traf.log.2.gz
... etc

/var/log/apache2/users
-rw-r----- 1 root adm 5766735 2008-09-01 13:32 site1.com-access.log
-rw-r----- 1 root adm 25967717 2008-08-31 07:31 site1.com-access.log.1
-rw-r--r-- 1 root root 167339 2008-08-27 06:38 site1.com-access.log.2.gz
-rw-r----- 1 root adm 34015 2008-09-01 11:41 site1.com-error.log
-rw-r----- 1 root adm 57657 2008-08-31 02:28 site1.com-error.log.1
-rw-r--r-- 1 root root 296 2008-08-22 14:56 site1.com-error.log.2.gz
... etc