Current time: 10-05-2022, 07:53 PM Hello There, Guest! (LoginRegister)


Post Reply 
Memory Usage Problem
Author Message
Keirnan Offline
Newbie
*

Posts: 6
Joined: Feb 2009
Reputation: 0
Post: #1
Memory Usage Problem
I am new fairly new to LAMP. My problem is that my server will continue to consume memory, and never seems to release it. If I leave it alone for several days, it will have consumed all the physical RAM and started swapping. It does this pretty slowly - maybe 50MB an hour?

I can't seem to figure out where the memory goes. I don't see any big consumers, but I notice that the amount of used memory never goes down, only up. If I restart apache (apache2ctl restart), it will only free up a small amount of memory. Likewise for MySQL. The only thing that frees up the memory counter is if I reboot the box.

This is a website running on a dedicated Ubuntu 8.1 server. 3.0Ghz P4, 1GB RAM, hardware mirrored 500GB hard drives.

It is running ISPCP and everything that comes with it:
-Bind
-MySQL
-PHP
-ProFTPD
-Postfix
-Courier

As well as Fail2Ban.

Everything has its default config, except for MySQL, where I changed the key_buffer_pool to 48MB and increased table cache size to 128. Everything else in the MySQL config is unchanged. I have not changed any PHP, mail, or Apache settings.

The server is running 5 virtual hosts. 4 of those virtual hosts are basically unused. The remaining one gets approximately 4000 page views per day. There are never more than 25 concurrent sessions.

CPU utilization is extremely low, typically 95+% idle.

Here is top, sorted by memory:

top - 15:53:19 up 10:50, 1 user, load average: 0.00, 0.00, 0.00
Tasks: 101 total, 1 running, 100 sleeping, 0 stopped, 0 zombie
Cpu(s): 0.0%us, 0.2%sy, 0.0%ni, 99.8%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Mem: 1018156k total, 748576k used, 269580k free, 81292k buffers
Swap: 3036276k total, 0k used, 3036276k free, 444640k cached

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
4469 mysql 20 0 161m 39m 5332 S 0 4.0 3:27.42 mysqld
26057 vu2002 20 0 29804 15m 3264 S 0 1.6 0:54.48 php5-cgi
26055 vu2002 20 0 29832 15m 3264 S 0 1.6 0:23.51 php5-cgi
26058 vu2002 20 0 27784 13m 3264 S 0 1.4 0:23.53 php5-cgi
5144 vu2001 20 0 25584 11m 3280 S 0 1.2 0:02.79 php5-cgi
26059 vu2002 20 0 25188 11m 3264 S 0 1.1 0:26.48 php5-cgi
4778 www-data 20 0 234m 10m 1644 S 0 1.1 0:04.78 apache2
5143 vu2001 20 0 24844 10m 3252 S 0 1.1 0:03.33 php5-cgi
4775 www-data 20 0 233m 10m 1640 S 0 1.0 0:04.85 apache2
4763 root 20 0 13480 9868 2040 S 0 1.0 0:03.45 ispcp-apache-lo
4762 root 20 0 13340 9800 2040 S 0 1.0 0:00.42 ispcp-apache-lo
4348 bind 20 0 44620 8836 2032 S 0 0.9 0:00.79 named
31403 www-data 20 0 232m 8420 1648 S 0 0.8 0:01.77 apache2
4533 postgrey 20 0 11136 8260 2680 S 0 0.8 0:00.18 postgrey

I was thinking it was just a lot of apache php5-cgi or apache processes, but they don't seem to be using that much. MySQL is the #1 consumer right now, but sometimes it will be a php5 process.

Here is ps aux at almost 11 hours of up time. In another 24 hours or so, it will be swapping:

USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
root 1 0.0 0.1 2844 1692 ? Ss 05:03 0:01 /sbin/init
root 2 0.0 0.0 0 0 ? S< 05:03 0:00 [kthreadd]
root 3 0.0 0.0 0 0 ? S< 05:03 0:00 [migration/0]
root 4 0.0 0.0 0 0 ? S< 05:03 0:00 [ksoftirqd/0]
root 5 0.0 0.0 0 0 ? S< 05:03 0:00 [watchdog/0]
root 6 0.0 0.0 0 0 ? S< 05:03 0:00 [migration/1]
root 7 0.0 0.0 0 0 ? S< 05:03 0:00 [ksoftirqd/1]
root 8 0.0 0.0 0 0 ? S< 05:03 0:00 [watchdog/1]
root 9 0.0 0.0 0 0 ? S< 05:03 0:00 [events/0]
root 10 0.0 0.0 0 0 ? S< 05:03 0:00 [events/1]
root 11 0.0 0.0 0 0 ? S< 05:03 0:00 [khelper]
root 46 0.0 0.0 0 0 ? S< 05:03 0:00 [kblockd/0]
root 47 0.0 0.0 0 0 ? S< 05:03 0:00 [kblockd/1]
root 50 0.0 0.0 0 0 ? S< 05:03 0:00 [kacpid]
root 51 0.0 0.0 0 0 ? S< 05:03 0:00 [kacpi_notify]
root 123 0.0 0.0 0 0 ? S< 05:03 0:00 [kseriod]
root 163 0.0 0.0 0 0 ? S 05:03 0:00 [pdflush]
root 164 0.0 0.0 0 0 ? S 05:03 0:00 [pdflush]
root 165 0.0 0.0 0 0 ? S< 05:03 0:00 [kswapd0]
root 207 0.0 0.0 0 0 ? S< 05:03 0:00 [aio/0]
root 208 0.0 0.0 0 0 ? S< 05:03 0:00 [aio/1]
root 1275 0.0 0.0 0 0 ? S< 05:03 0:00 [ksuspend_usbd]
root 1282 0.0 0.0 0 0 ? S< 05:03 0:00 [khubd]
root 1434 0.0 0.0 0 0 ? S< 05:03 0:00 [scsi_eh_0]
root 1468 0.0 0.0 0 0 ? S< 05:03 0:00 [ata/0]
root 1469 0.0 0.0 0 0 ? S< 05:03 0:00 [ata/1]
root 1471 0.0 0.0 0 0 ? S< 05:03 0:00 [ata_aux]
root 2322 0.0 0.0 0 0 ? S< 05:03 0:00 [scsi_eh_1]
root 2324 0.0 0.0 0 0 ? S< 05:03 0:00 [scsi_eh_2]
root 2381 0.0 0.0 0 0 ? S< 05:03 0:00 [scsi_eh_3]
root 2382 0.0 0.0 0 0 ? S< 05:03 0:00 [scsi_eh_4]
root 2483 0.0 0.0 0 0 ? S< 05:03 0:00 [kjournald]
root 2640 0.0 0.0 2224 672 ? S<s 05:03 0:00 /sbin/udevd --daemon
root 2967 0.0 0.0 0 0 ? S< 05:03 0:00 [kpsmoused]
root 4255 0.0 0.0 1716 512 tty4 Ss+ 05:03 0:00 /sbin/getty 38400 tty4
root 4256 0.0 0.0 1716 516 tty5 Ss+ 05:03 0:00 /sbin/getty 38400 tty5
root 4258 0.0 0.0 1716 516 tty2 Ss+ 05:03 0:00 /sbin/getty 38400 tty2
root 4260 0.0 0.0 1716 512 tty3 Ss+ 05:03 0:00 /sbin/getty 38400 tty3
root 4262 0.0 0.0 1716 512 tty6 Ss+ 05:03 0:00 /sbin/getty 38400 tty6
syslog 4304 0.0 0.0 1936 684 ? Ss 05:03 0:00 /sbin/syslogd -u syslog
root 4323 0.0 0.0 1872 544 ? S 05:03 0:00 /bin/dd bs 1 if /proc/kmsg of /var/run/klogd/kmsg
klog 4325 0.0 0.1 3152 2008 ? Ss 05:03 0:00 /sbin/klogd -P /var/run/klogd/kmsg
bind 4348 0.0 0.8 44620 8836 ? Ssl 05:03 0:00 /usr/sbin/named -u bind
root 4371 0.0 0.0 5316 1012 ? Ss 05:03 0:00 /usr/sbin/sshd
root 4427 0.0 0.0 1772 520 ? S 05:03 0:00 /bin/sh /usr/bin/mysqld_safe
mysql 4469 0.5 4.0 164892 40840 ? Sl 05:03 3:28 /usr/sbin/mysqld --basedir=/usr --datadir=/var/lib/mysql -
root 4470 0.0 0.0 1700 556 ? S 05:03 0:00 logger -p daemon.err -t mysqld_safe -i -t mysqld
polw 4529 0.0 0.5 8140 5948 ? Ss 05:03 0:00 policyd-weight (master)
polw 4530 0.0 0.5 8272 5584 ? Ss 05:03 0:00 policyd-weight (cache)
postgrey 4533 0.0 0.8 11136 8260 ? Ss 05:03 0:00 /usr/sbin/postgrey --pidfile=/var/run/postgrey.pid --daemo
root 4557 0.0 0.0 1904 452 ? S 05:03 0:00 /usr/sbin/courierlogger -pid=/var/run/courier/authdaemon/p
root 4558 0.0 0.0 2112 672 ? S 05:03 0:00 /usr/lib/courier/courier-authlib/authdaemond
root 4573 0.0 0.0 1904 364 ? S 05:03 0:00 /usr/sbin/courierlogger -pid=/var/run/courier/imapd.pid -s
root 4574 0.0 0.0 2008 600 ? S 05:03 0:00 /usr/sbin/couriertcpd -address=0 -maxprocs=40 -maxperip=20
root 4587 0.0 0.0 1904 452 ? S 05:03 0:00 /usr/sbin/courierlogger -pid=/var/run/courier/pop3d.pid -s
root 4588 0.0 0.0 2008 612 ? S 05:03 0:00 /usr/sbin/couriertcpd -maxprocs=40 -maxperip=4 -nodnslooku
root 4591 0.0 0.1 2912 1284 ? S 05:03 0:00 /usr/lib/courier/courier-authlib/authdaemond
root 4592 0.0 0.0 2116 588 ? S 05:03 0:00 /usr/lib/courier/courier-authlib/authdaemond
root 4593 0.0 0.0 2116 588 ? S 05:03 0:00 /usr/lib/courier/courier-authlib/authdaemond
root 4594 0.0 0.0 2116 588 ? S 05:03 0:00 /usr/lib/courier/courier-authlib/authdaemond
root 4595 0.0 0.0 2116 588 ? S 05:03 0:00 /usr/lib/courier/courier-authlib/authdaemond
root 4660 0.0 0.1 5396 1784 ? Ss 05:03 0:00 /usr/lib/postfix/master
postfix 4664 0.0 0.1 5444 1832 ? S 05:03 0:00 qmgr -l -t fifo -u
nobody 4708 0.0 0.1 5992 1920 ? Ss 05:03 0:00 proftpd: (accepting connections)
daemon 4722 0.0 0.0 1984 424 ? Ss 05:03 0:00 /usr/sbin/atd
root 4733 0.0 0.0 2104 888 ? Ss 05:03 0:00 /usr/sbin/cron
root 4759 0.0 0.3 11476 3340 ? Ss 05:03 0:01 /usr/sbin/apache2 -k start
root 4760 0.0 0.0 1772 488 ? S 05:03 0:00 /bin/sh -c /var/www/ispcp/engine/ispcp-apache-logger -e
root 4761 0.0 0.0 1772 484 ? S 05:03 0:00 /bin/sh -c /var/www/ispcp/engine/ispcp-apache-logger
root 4762 0.0 0.9 13340 9800 ? S 05:03 0:00 /usr/bin/perl /var/www/ispcp/engine/ispcp-apache-logger -e
root 4763 0.0 0.9 13480 9868 ? S 05:03 0:03 /usr/bin/perl /var/www/ispcp/engine/ispcp-apache-logger
www-data 4765 0.0 0.2 10928 2148 ? S 05:03 0:00 /usr/sbin/apache2 -k start
www-data 4772 0.0 0.2 11248 2240 ? S 05:03 0:00 /usr/sbin/apache2 -k start
www-data 4775 0.0 1.0 239028 10272 ? Sl 05:03 0:04 /usr/sbin/apache2 -k start
www-data 4778 0.0 1.0 240052 11036 ? Sl 05:03 0:04 /usr/sbin/apache2 -k start
root 4837 0.0 0.4 40228 4616 ? Sl 05:03 0:12 /usr/bin/python /usr/bin/fail2ban-server -b -s /var/run/fa
root 4860 0.0 0.0 1700 448 ? S 05:03 0:00 /var/www/ispcp/daemon/ispcp_daemon -p /var/run/ispcp_daemo
root 4933 0.0 0.0 1716 512 tty1 Ss+ 05:03 0:00 /sbin/getty 38400 tty1
polw 4978 0.0 0.6 8400 6264 ? S 05:09 0:00 policyd-weight (child)
vu2001 5142 0.0 0.5 19240 5484 ? Ss 05:39 0:00 /usr/bin/php5-cgi
vu2001 5143 0.0 1.0 24844 11028 ? S 05:39 0:03 /usr/bin/php5-cgi
vu2001 5144 0.0 1.1 25584 11820 ? S 05:39 0:02 /usr/bin/php5-cgi
postfix 25912 0.0 0.1 5404 1672 ? S 14:36 0:00 pickup -l -t fifo -u -c
vu2002 26054 0.0 0.5 19240 5488 ? Ss 15:00 0:00 /usr/bin/php5-cgi
vu2002 26055 0.7 1.5 29832 15964 ? S 15:00 0:24 /usr/bin/php5-cgi
vu2002 26056 0.0 0.5 19240 5488 ? Ss 15:00 0:00 /usr/bin/php5-cgi
vu2002 26057 1.6 1.5 29804 15992 ? S 15:00 0:55 /usr/bin/php5-cgi
vu2002 26058 0.7 1.3 27784 13872 ? S 15:00 0:24 /usr/bin/php5-cgi
vu2002 26059 0.8 1.1 25444 11664 ? S 15:00 0:27 /usr/bin/php5-cgi
root 26241 0.0 0.2 8056 2520 ? Ss 15:42 0:00 sshd: pragupia6060 [priv]
1000 26243 0.0 0.1 8056 1552 ? S 15:42 0:00 sshd: pragupia6060@pts/0
1000 26244 0.0 0.2 5560 2980 pts/0 Ss 15:42 0:00 -bash
postfix 26266 0.0 0.3 6612 3188 ? S 15:46 0:00 smtpd -n smtp -t inet -u -c -o stress
postfix 26267 0.0 0.1 5408 1732 ? S 15:46 0:00 anvil -l -t unix -u -c
1000 26294 0.0 0.0 2644 1008 pts/0 R+ 15:57 0:00 ps aux
polw 31044 0.0 0.6 8404 6300 ? S 09:38 0:00 policyd-weight (child)
www-data 31403 0.0 0.8 237696 8420 ? Sl 10:51 0:01 /usr/sbin/apache2 -k start
Now, here is top again, about 2.5 hours later. Notice that I've used another 100MB of memory.

Code:
top - 18:19:28 up 13:16,  1 user,  load average: 0.00, 0.02, 0.00
Tasks: 100 total,   3 running,  97 sleeping,   0 stopped,   0 zombie
Cpu(s):  1.0%us,  0.0%sy,  0.0%ni, 99.0%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Mem:   1018156k total,   851868k used,   166288k free,    88116k buffers
Swap:  3036276k total,        0k used,  3036276k free,   451840k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
26058 vu2002    20   0  129m  83m 3268 S    0  8.4   0:56.54 php5-cgi
4469 mysql     20   0  161m  41m 5332 S    0  4.1   3:53.50 mysqld
26057 vu2002    20   0 30132  16m 3312 R    1  1.6   2:05.74 php5-cgi
5144 vu2001    20   0 25584  11m 3308 S    0  1.2   0:03.37 php5-cgi
26574 vu2002    20   0 25504  11m 3264 S    0  1.2   0:21.35 php5-cgi
26059 vu2002    20   0 25456  11m 3268 S    0  1.1   1:10.12 php5-cgi
26573 vu2002    20   0 25468  11m 3240 S    0  1.1   0:23.85 php5-cgi
26397 vu2002    20   0 25296  11m 3264 S    0  1.1   1:08.82 php5-cgi
26396 vu2002    20   0 25292  11m 3272 S    0  1.1   0:37.31 php5-cgi
4778 www-data  20   0  234m  10m 1644 S    0  1.1   0:05.20 apache2
5143 vu2001    20   0 24844  10m 3252 S    0  1.1   0:04.18 php5-cgi
4775 www-data  20   0  233m  10m 1640 S    0  1.0   0:05.31 apache2
4763 root      20   0 13480 9868 2040 S    0  1.0   0:03.85 ispcp-apache-lo
4762 root      20   0 13340 9800 2040 S    0  1.0   0:00.42 ispcp-apache-lo
4348 bind      20   0 44620 8836 2032 S    0  0.9   0:00.80 named
31403 www-data  20   0  232m 8528 1648 S    0  0.8   0:02.31 apache2
4533 postgrey  20   0 11136 8260 2680 S    0  0.8   0:00.20 postgrey
31044 polw      20   0  8404 6300 1456 S    0  0.6   0:00.42 policyd-weight
4978 polw      20   0  8400 6264 1424 S    0  0.6   0:00.85 policyd-weight
4529 polw      20   0  8140 5948 1244 S    0  0.6   0:00.14 policyd-weight
4530 polw      20   0  8272 5596  876 S    0  0.5   0:00.13 policyd-weight
26056 vu2002    20   0 19240 5488 3700 S    0  0.5   0:00.03 php5-cgi
26395 vu2002    20   0 19240 5488 3700 S    0  0.5   0:00.02 php5-cgi
26572 vu2002    20   0 19240 5488 3700 S    0  0.5   0:00.04 php5-cgi
5142 vu2001    20   0 19240 5484 3700 S    0  0.5   0:00.02 php5-cgi
4837 root      20   0 40228 4616 1848 S    0  0.5   0:14.25 fail2ban-server
4759 root      20   0 11476 3340 1744 S    0  0.3   0:01.45 apache2
27287 pragupia  20   0  5560 2940 1416 S    0  0.3   0:00.10 bash
27284 root      20   0  8056 2524 2068 S    0  0.2   0:00.01 sshd
4772 www-data  20   0 11248 2240  624 S    0  0.2   0:00.45 apache2
4765 www-data  20   0 10928 2148  556 S    0  0.2   0:00.00 apache2
4325 klog      20   0  3152 2008  424 S    0  0.2   0:00.11 klogd
4708 nobody    20   0  5992 1920  568 S    0  0.2   0:00.38 proftpd
4664 postfix   20   0  5444 1832 1488 S    0  0.2   0:00.08 qmgr
4660 root      20   0  5396 1784 1452 S    0  0.2   0:00.27 master
26890 postfix   20   0  5404 1740 1412 S    0  0.2   0:00.00 pickup
27027 postfix   20   0  5408 1732 1412 S    0  0.2   0:00.00 anvil

And here is vmstat -s run at the same time as this top. I wish I had it for when I grabbed the stats 2 hours ago, but I don't.

Code:
1018156 K total memory
       854916 K used memory
       383396 K active memory
       428436 K inactive memory
       163240 K free memory
        87856 K buffer memory
       451784 K swap cache
      3036276 K total swap
            0 K used swap
      3036276 K free swap
       219036 non-nice user cpu ticks
           17 nice user cpu ticks
        44155 system cpu ticks
      9243817 idle cpu ticks
         5510 IO-wait cpu ticks
          537 IRQ cpu ticks
         1753 softirq cpu ticks
            0 stolen cpu ticks
       455927 pages paged in
       950492 pages paged out
            0 pages swapped in
            0 pages swapped out
     12198813 interrupts
      6801841 CPU context switches
   1234011795 boot time
       124200 forks
(This post was last modified: 02-08-2009 01:11 PM by Keirnan.)
02-08-2009 01:05 PM
Find all posts by this user Quote this message in a reply
Post Reply 


Forum Jump:


User(s) browsing this thread: 1 Guest(s)