I am new fairly new to LAMP. My problem is that my server will continue to consume memory, and never seems to release it. If I leave it alone for several days, it will have consumed all the physical RAM and started swapping. It does this pretty slowly - maybe 50MB an hour?
I can't seem to figure out where the memory goes. I don't see any big consumers, but I notice that the amount of used memory never goes down, only up. If I restart apache (apache2ctl restart), it will only free up a small amount of memory. Likewise for MySQL. The only thing that frees up the memory counter is if I reboot the box.
This is a website running on a dedicated Ubuntu 8.1 server. 3.0Ghz P4, 1GB RAM, hardware mirrored 500GB hard drives.
It is running ISPCP and everything that comes with it:
-Bind
-MySQL
-PHP
-ProFTPD
-Postfix
-Courier
As well as Fail2Ban.
Everything has its default config, except for MySQL, where I changed the key_buffer_pool to 48MB and increased table cache size to 128. Everything else in the MySQL config is unchanged. I have not changed any PHP, mail, or Apache settings.
The server is running 5 virtual hosts. 4 of those virtual hosts are basically unused. The remaining one gets approximately 4000 page views per day. There are never more than 25 concurrent sessions.
CPU utilization is extremely low, typically 95+% idle.
Here is top, sorted by memory:
top - 15:53:19 up 10:50, 1 user, load average: 0.00, 0.00, 0.00
Tasks: 101 total, 1 running, 100 sleeping, 0 stopped, 0 zombie
Cpu(s): 0.0%us, 0.2%sy, 0.0%ni, 99.8%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Mem: 1018156k total, 748576k used, 269580k free, 81292k buffers
Swap: 3036276k total, 0k used, 3036276k free, 444640k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
4469 mysql 20 0 161m 39m 5332 S 0 4.0 3:27.42 mysqld
26057 vu2002 20 0 29804 15m 3264 S 0 1.6 0:54.48 php5-cgi
26055 vu2002 20 0 29832 15m 3264 S 0 1.6 0:23.51 php5-cgi
26058 vu2002 20 0 27784 13m 3264 S 0 1.4 0:23.53 php5-cgi
5144 vu2001 20 0 25584 11m 3280 S 0 1.2 0:02.79 php5-cgi
26059 vu2002 20 0 25188 11m 3264 S 0 1.1 0:26.48 php5-cgi
4778 www-data 20 0 234m 10m 1644 S 0 1.1 0:04.78 apache2
5143 vu2001 20 0 24844 10m 3252 S 0 1.1 0:03.33 php5-cgi
4775 www-data 20 0 233m 10m 1640 S 0 1.0 0:04.85 apache2
4763 root 20 0 13480 9868 2040 S 0 1.0 0:03.45 ispcp-apache-lo
4762 root 20 0 13340 9800 2040 S 0 1.0 0:00.42 ispcp-apache-lo
4348 bind 20 0 44620 8836 2032 S 0 0.9 0:00.79 named
31403 www-data 20 0 232m 8420 1648 S 0 0.8 0:01.77 apache2
4533 postgrey 20 0 11136 8260 2680 S 0 0.8 0:00.18 postgrey
I was thinking it was just a lot of apache php5-cgi or apache processes, but they don't seem to be using that much. MySQL is the #1 consumer right now, but sometimes it will be a php5 process.
Here is ps aux at almost 11 hours of up time. In another 24 hours or so, it will be swapping:
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
root 1 0.0 0.1 2844 1692 ? Ss 05:03 0:01 /sbin/init
root 2 0.0 0.0 0 0 ? S< 05:03 0:00 [kthreadd]
root 3 0.0 0.0 0 0 ? S< 05:03 0:00 [migration/0]
root 4 0.0 0.0 0 0 ? S< 05:03 0:00 [ksoftirqd/0]
root 5 0.0 0.0 0 0 ? S< 05:03 0:00 [watchdog/0]
root 6 0.0 0.0 0 0 ? S< 05:03 0:00 [migration/1]
root 7 0.0 0.0 0 0 ? S< 05:03 0:00 [ksoftirqd/1]
root 8 0.0 0.0 0 0 ? S< 05:03 0:00 [watchdog/1]
root 9 0.0 0.0 0 0 ? S< 05:03 0:00 [events/0]
root 10 0.0 0.0 0 0 ? S< 05:03 0:00 [events/1]
root 11 0.0 0.0 0 0 ? S< 05:03 0:00 [khelper]
root 46 0.0 0.0 0 0 ? S< 05:03 0:00 [kblockd/0]
root 47 0.0 0.0 0 0 ? S< 05:03 0:00 [kblockd/1]
root 50 0.0 0.0 0 0 ? S< 05:03 0:00 [kacpid]
root 51 0.0 0.0 0 0 ? S< 05:03 0:00 [kacpi_notify]
root 123 0.0 0.0 0 0 ? S< 05:03 0:00 [kseriod]
root 163 0.0 0.0 0 0 ? S 05:03 0:00 [pdflush]
root 164 0.0 0.0 0 0 ? S 05:03 0:00 [pdflush]
root 165 0.0 0.0 0 0 ? S< 05:03 0:00 [kswapd0]
root 207 0.0 0.0 0 0 ? S< 05:03 0:00 [aio/0]
root 208 0.0 0.0 0 0 ? S< 05:03 0:00 [aio/1]
root 1275 0.0 0.0 0 0 ? S< 05:03 0:00 [ksuspend_usbd]
root 1282 0.0 0.0 0 0 ? S< 05:03 0:00 [khubd]
root 1434 0.0 0.0 0 0 ? S< 05:03 0:00 [scsi_eh_0]
root 1468 0.0 0.0 0 0 ? S< 05:03 0:00 [ata/0]
root 1469 0.0 0.0 0 0 ? S< 05:03 0:00 [ata/1]
root 1471 0.0 0.0 0 0 ? S< 05:03 0:00 [ata_aux]
root 2322 0.0 0.0 0 0 ? S< 05:03 0:00 [scsi_eh_1]
root 2324 0.0 0.0 0 0 ? S< 05:03 0:00 [scsi_eh_2]
root 2381 0.0 0.0 0 0 ? S< 05:03 0:00 [scsi_eh_3]
root 2382 0.0 0.0 0 0 ? S< 05:03 0:00 [scsi_eh_4]
root 2483 0.0 0.0 0 0 ? S< 05:03 0:00 [kjournald]
root 2640 0.0 0.0 2224 672 ? S<s 05:03 0:00 /sbin/udevd --daemon
root 2967 0.0 0.0 0 0 ? S< 05:03 0:00 [kpsmoused]
root 4255 0.0 0.0 1716 512 tty4 Ss+ 05:03 0:00 /sbin/getty 38400 tty4
root 4256 0.0 0.0 1716 516 tty5 Ss+ 05:03 0:00 /sbin/getty 38400 tty5
root 4258 0.0 0.0 1716 516 tty2 Ss+ 05:03 0:00 /sbin/getty 38400 tty2
root 4260 0.0 0.0 1716 512 tty3 Ss+ 05:03 0:00 /sbin/getty 38400 tty3
root 4262 0.0 0.0 1716 512 tty6 Ss+ 05:03 0:00 /sbin/getty 38400 tty6
syslog 4304 0.0 0.0 1936 684 ? Ss 05:03 0:00 /sbin/syslogd -u syslog
root 4323 0.0 0.0 1872 544 ? S 05:03 0:00 /bin/dd bs 1 if /proc/kmsg of /var/run/klogd/kmsg
klog 4325 0.0 0.1 3152 2008 ? Ss 05:03 0:00 /sbin/klogd -P /var/run/klogd/kmsg
bind 4348 0.0 0.8 44620 8836 ? Ssl 05:03 0:00 /usr/sbin/named -u bind
root 4371 0.0 0.0 5316 1012 ? Ss 05:03 0:00 /usr/sbin/sshd
root 4427 0.0 0.0 1772 520 ? S 05:03 0:00 /bin/sh /usr/bin/mysqld_safe
mysql 4469 0.5 4.0 164892 40840 ? Sl 05:03 3:28 /usr/sbin/mysqld --basedir=/usr --datadir=/var/lib/mysql -
root 4470 0.0 0.0 1700 556 ? S 05:03 0:00 logger -p daemon.err -t mysqld_safe -i -t mysqld
polw 4529 0.0 0.5 8140 5948 ? Ss 05:03 0:00 policyd-weight (master)
polw 4530 0.0 0.5 8272 5584 ? Ss 05:03 0:00 policyd-weight (cache)
postgrey 4533 0.0 0.8 11136 8260 ? Ss 05:03 0:00 /usr/sbin/postgrey --pidfile=/var/run/postgrey.pid --daemo
root 4557 0.0 0.0 1904 452 ? S 05:03 0:00 /usr/sbin/courierlogger -pid=/var/run/courier/authdaemon/p
root 4558 0.0 0.0 2112 672 ? S 05:03 0:00 /usr/lib/courier/courier-authlib/authdaemond
root 4573 0.0 0.0 1904 364 ? S 05:03 0:00 /usr/sbin/courierlogger -pid=/var/run/courier/imapd.pid -s
root 4574 0.0 0.0 2008 600 ? S 05:03 0:00 /usr/sbin/couriertcpd -address=0 -maxprocs=40 -maxperip=20
root 4587 0.0 0.0 1904 452 ? S 05:03 0:00 /usr/sbin/courierlogger -pid=/var/run/courier/pop3d.pid -s
root 4588 0.0 0.0 2008 612 ? S 05:03 0:00 /usr/sbin/couriertcpd -maxprocs=40 -maxperip=4 -nodnslooku
root 4591 0.0 0.1 2912 1284 ? S 05:03 0:00 /usr/lib/courier/courier-authlib/authdaemond
root 4592 0.0 0.0 2116 588 ? S 05:03 0:00 /usr/lib/courier/courier-authlib/authdaemond
root 4593 0.0 0.0 2116 588 ? S 05:03 0:00 /usr/lib/courier/courier-authlib/authdaemond
root 4594 0.0 0.0 2116 588 ? S 05:03 0:00 /usr/lib/courier/courier-authlib/authdaemond
root 4595 0.0 0.0 2116 588 ? S 05:03 0:00 /usr/lib/courier/courier-authlib/authdaemond
root 4660 0.0 0.1 5396 1784 ? Ss 05:03 0:00 /usr/lib/postfix/master
postfix 4664 0.0 0.1 5444 1832 ? S 05:03 0:00 qmgr -l -t fifo -u
nobody 4708 0.0 0.1 5992 1920 ? Ss 05:03 0:00 proftpd: (accepting connections)
daemon 4722 0.0 0.0 1984 424 ? Ss 05:03 0:00 /usr/sbin/atd
root 4733 0.0 0.0 2104 888 ? Ss 05:03 0:00 /usr/sbin/cron
root 4759 0.0 0.3 11476 3340 ? Ss 05:03 0:01 /usr/sbin/apache2 -k start
root 4760 0.0 0.0 1772 488 ? S 05:03 0:00 /bin/sh -c /var/www/ispcp/engine/ispcp-apache-logger -e
root 4761 0.0 0.0 1772 484 ? S 05:03 0:00 /bin/sh -c /var/www/ispcp/engine/ispcp-apache-logger
root 4762 0.0 0.9 13340 9800 ? S 05:03 0:00 /usr/bin/perl /var/www/ispcp/engine/ispcp-apache-logger -e
root 4763 0.0 0.9 13480 9868 ? S 05:03 0:03 /usr/bin/perl /var/www/ispcp/engine/ispcp-apache-logger
www-data 4765 0.0 0.2 10928 2148 ? S 05:03 0:00 /usr/sbin/apache2 -k start
www-data 4772 0.0 0.2 11248 2240 ? S 05:03 0:00 /usr/sbin/apache2 -k start
www-data 4775 0.0 1.0 239028 10272 ? Sl 05:03 0:04 /usr/sbin/apache2 -k start
www-data 4778 0.0 1.0 240052 11036 ? Sl 05:03 0:04 /usr/sbin/apache2 -k start
root 4837 0.0 0.4 40228 4616 ? Sl 05:03 0:12 /usr/bin/python /usr/bin/fail2ban-server -b -s /var/run/fa
root 4860 0.0 0.0 1700 448 ? S 05:03 0:00 /var/www/ispcp/daemon/ispcp_daemon -p /var/run/ispcp_daemo
root 4933 0.0 0.0 1716 512 tty1 Ss+ 05:03 0:00 /sbin/getty 38400 tty1
polw 4978 0.0 0.6 8400 6264 ? S 05:09 0:00 policyd-weight (child)
vu2001 5142 0.0 0.5 19240 5484 ? Ss 05:39 0:00 /usr/bin/php5-cgi
vu2001 5143 0.0 1.0 24844 11028 ? S 05:39 0:03 /usr/bin/php5-cgi
vu2001 5144 0.0 1.1 25584 11820 ? S 05:39 0:02 /usr/bin/php5-cgi
postfix 25912 0.0 0.1 5404 1672 ? S 14:36 0:00 pickup -l -t fifo -u -c
vu2002 26054 0.0 0.5 19240 5488 ? Ss 15:00 0:00 /usr/bin/php5-cgi
vu2002 26055 0.7 1.5 29832 15964 ? S 15:00 0:24 /usr/bin/php5-cgi
vu2002 26056 0.0 0.5 19240 5488 ? Ss 15:00 0:00 /usr/bin/php5-cgi
vu2002 26057 1.6 1.5 29804 15992 ? S 15:00 0:55 /usr/bin/php5-cgi
vu2002 26058 0.7 1.3 27784 13872 ? S 15:00 0:24 /usr/bin/php5-cgi
vu2002 26059 0.8 1.1 25444 11664 ? S 15:00 0:27 /usr/bin/php5-cgi
root 26241 0.0 0.2 8056 2520 ? Ss 15:42 0:00 sshd: pragupia6060 [priv]
1000 26243 0.0 0.1 8056 1552 ? S 15:42 0:00 sshd: pragupia6060@pts/0
1000 26244 0.0 0.2 5560 2980 pts/0 Ss 15:42 0:00 -bash
postfix 26266 0.0 0.3 6612 3188 ? S 15:46 0:00 smtpd -n smtp -t inet -u -c -o stress
postfix 26267 0.0 0.1 5408 1732 ? S 15:46 0:00 anvil -l -t unix -u -c
1000 26294 0.0 0.0 2644 1008 pts/0 R+ 15:57 0:00 ps aux
polw 31044 0.0 0.6 8404 6300 ? S 09:38 0:00 policyd-weight (child)
www-data 31403 0.0 0.8 237696 8420 ? Sl 10:51 0:01 /usr/sbin/apache2 -k start
Now, here is top again, about 2.5 hours later. Notice that I've used another 100MB of memory.
Code:
top - 18:19:28 up 13:16, 1 user, load average: 0.00, 0.02, 0.00
Tasks: 100 total, 3 running, 97 sleeping, 0 stopped, 0 zombie
Cpu(s): 1.0%us, 0.0%sy, 0.0%ni, 99.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Mem: 1018156k total, 851868k used, 166288k free, 88116k buffers
Swap: 3036276k total, 0k used, 3036276k free, 451840k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
26058 vu2002 20 0 129m 83m 3268 S 0 8.4 0:56.54 php5-cgi
4469 mysql 20 0 161m 41m 5332 S 0 4.1 3:53.50 mysqld
26057 vu2002 20 0 30132 16m 3312 R 1 1.6 2:05.74 php5-cgi
5144 vu2001 20 0 25584 11m 3308 S 0 1.2 0:03.37 php5-cgi
26574 vu2002 20 0 25504 11m 3264 S 0 1.2 0:21.35 php5-cgi
26059 vu2002 20 0 25456 11m 3268 S 0 1.1 1:10.12 php5-cgi
26573 vu2002 20 0 25468 11m 3240 S 0 1.1 0:23.85 php5-cgi
26397 vu2002 20 0 25296 11m 3264 S 0 1.1 1:08.82 php5-cgi
26396 vu2002 20 0 25292 11m 3272 S 0 1.1 0:37.31 php5-cgi
4778 www-data 20 0 234m 10m 1644 S 0 1.1 0:05.20 apache2
5143 vu2001 20 0 24844 10m 3252 S 0 1.1 0:04.18 php5-cgi
4775 www-data 20 0 233m 10m 1640 S 0 1.0 0:05.31 apache2
4763 root 20 0 13480 9868 2040 S 0 1.0 0:03.85 ispcp-apache-lo
4762 root 20 0 13340 9800 2040 S 0 1.0 0:00.42 ispcp-apache-lo
4348 bind 20 0 44620 8836 2032 S 0 0.9 0:00.80 named
31403 www-data 20 0 232m 8528 1648 S 0 0.8 0:02.31 apache2
4533 postgrey 20 0 11136 8260 2680 S 0 0.8 0:00.20 postgrey
31044 polw 20 0 8404 6300 1456 S 0 0.6 0:00.42 policyd-weight
4978 polw 20 0 8400 6264 1424 S 0 0.6 0:00.85 policyd-weight
4529 polw 20 0 8140 5948 1244 S 0 0.6 0:00.14 policyd-weight
4530 polw 20 0 8272 5596 876 S 0 0.5 0:00.13 policyd-weight
26056 vu2002 20 0 19240 5488 3700 S 0 0.5 0:00.03 php5-cgi
26395 vu2002 20 0 19240 5488 3700 S 0 0.5 0:00.02 php5-cgi
26572 vu2002 20 0 19240 5488 3700 S 0 0.5 0:00.04 php5-cgi
5142 vu2001 20 0 19240 5484 3700 S 0 0.5 0:00.02 php5-cgi
4837 root 20 0 40228 4616 1848 S 0 0.5 0:14.25 fail2ban-server
4759 root 20 0 11476 3340 1744 S 0 0.3 0:01.45 apache2
27287 pragupia 20 0 5560 2940 1416 S 0 0.3 0:00.10 bash
27284 root 20 0 8056 2524 2068 S 0 0.2 0:00.01 sshd
4772 www-data 20 0 11248 2240 624 S 0 0.2 0:00.45 apache2
4765 www-data 20 0 10928 2148 556 S 0 0.2 0:00.00 apache2
4325 klog 20 0 3152 2008 424 S 0 0.2 0:00.11 klogd
4708 nobody 20 0 5992 1920 568 S 0 0.2 0:00.38 proftpd
4664 postfix 20 0 5444 1832 1488 S 0 0.2 0:00.08 qmgr
4660 root 20 0 5396 1784 1452 S 0 0.2 0:00.27 master
26890 postfix 20 0 5404 1740 1412 S 0 0.2 0:00.00 pickup
27027 postfix 20 0 5408 1732 1412 S 0 0.2 0:00.00 anvil
And here is vmstat -s run at the same time as this top. I wish I had it for when I grabbed the stats 2 hours ago, but I don't.
Code:
1018156 K total memory
854916 K used memory
383396 K active memory
428436 K inactive memory
163240 K free memory
87856 K buffer memory
451784 K swap cache
3036276 K total swap
0 K used swap
3036276 K free swap
219036 non-nice user cpu ticks
17 nice user cpu ticks
44155 system cpu ticks
9243817 idle cpu ticks
5510 IO-wait cpu ticks
537 IRQ cpu ticks
1753 softirq cpu ticks
0 stolen cpu ticks
455927 pages paged in
950492 pages paged out
0 pages swapped in
0 pages swapped out
12198813 interrupts
6801841 CPU context switches
1234011795 boot time
124200 forks
Search
Member List
Calendar
Help
