[How-to] Configure MONIT to monitorize your server

[GaRCieLD]

MONITORING YOUR SERVER WITH MONIT

We are going to install MONIT to be able to monitorize our server, and be able to restart the processes that crashes or bypass permited memory or CPU.

This code are optimized for Debian Etch 4.0.

1. Installation

Code:

apt-get install monit

2. Configure MONIT, making a backup of the default config (you can look inside for other options)

Code:

cp /etc/monit/monitrc /etc/monit/monitrc_default
vi /etc/monit/monitrc

I put the config to monitorize: PROFTPD, SSHD, DNS SERVER (bind), MYSQLD, APACHE and POSTFIX.

You have to change the mail-format and put your domain instead of hosting.example.com.

Code:

set daemon  60
set logfile syslog facility log_daemon
set mailserver localhost
set mail-format { from: monit@hosting.example.com }
set alert root@localhost
set httpd port 2812 and
     SSL ENABLE
     PEMFILE  /etc/monit/monit.pem
     allow admin:ispcp

# PROFTPD
check process proftpd with pidfile /var/run/proftpd.pid
   group services
   start program = "/etc/init.d/proftpd start"
   stop program  = "/etc/init.d/proftpd stop"
   if failed port 21 protocol ftp then restart
   if 5 restarts within 5 cycles then timeout

# SSHD
check process sshd with pidfile /var/run/sshd.pid
   group system
   start program  "/etc/init.d/ssh start"
   stop program  "/etc/init.d/ssh stop"
   if failed port 22 protocol ssh then restart
   if 5 restarts within 5 cycles then timeout

# DNS SERVER
check process named with pidfile /var/run/bind/run/named.pid
   group services
   start program  "/etc/init.d/bind9 start"
   stop program  "/etc/init.d/bind9 stop"
   if failed host 127.0.0.1 port 53 type tcp protocol dns then alert
   if failed host 127.0.0.1 port 53 type udp protocol dns then alert
   if 5 restarts within 5 cycles then timeout

# MYSQL
check process mysql with pidfile /var/run/mysqld/mysqld.pid
   group services
   start program = "/etc/init.d/mysql start"
   stop program = "/etc/init.d/mysql stop"
   if failed host 127.0.0.1 port 3306 then restart
   if 5 restarts within 5 cycles then timeout

# APACHE
check process apache with pidfile /var/run/apache2.pid
   group services
   start program = "/etc/init.d/apache2 start"
   stop program  = "/etc/init.d/apache2 stop"
   if failed host admin.hosting.example.com port 80 protocol http
      and request "/tools/monin_test_file" then restart
   if cpu is greater than 60% for 2 cycles then alert
   if cpu > 90% for 5 cycles then restart
   if totalmem > 500 MB for 5 cycles then restart
   if children > 250 then restart
   if loadavg(5min) greater than 10 for 8 cycles then stop
   if 3 restarts within 5 cycles then timeout

# POSTFIX
check process postfix with pidfile /var/spool/postfix/pid/master.pid
   group services
   start program = "/etc/init.d/postfix start"
   stop  program = "/etc/init.d/postfix stop"
   if failed port 25 protocol smtp then restart
   if 5 restarts within 5 cycles then timeout

# ISPCP_DAEMON
check process ispcp_daemon with pidfile /var/run/ispcp-daemon.pid
   group services
   start program = "/etc/init.d/ispcp_daemon start"
   stop  program = "/etc/init.d/ispcp_daemon stop"
# If you active this, your /var/log/daemon will grow every minute,
# so it's not recomended unless you don't want to trust only in .pid
#   if failed port 9876 then restart
   if 5 restarts within 5 cycles then timeout

3. Configure MONIN to start

First we make the "test file" to test if apache is running (see the moninrc config file to personalize for you). You can put anywhere this file, but then change moninrc.

Code:

echo "OK" > /var/www/ispcp/gui/tools/monin_test_file

and then we can edit /etc/default/monit to enable the monit daemon, and change startup to "1", and if you want change also the CHECK_INTERVALS to the seconds that you want (normaly 60, default 180)

Code:

vi /etc/default/monit

----- BEGIN:/etc/default/monit -----
# Defaults for monit initscript
# sourced by /etc/init.d/monit
# installed at /etc/default/monit by maintainer scripts
# Fredrik Steen <stone@debian.org>

# You must set this variable to for monit to start
startup=1

# To change the intervals which monit should run uncomment
# and change this variable.
CHECK_INTERVALS=60
----- END:/etc/default/monit -----

4. Configure SSL with MONIN

When you generates the certificate, you can change this values, so you can put this example file without changes.

Code:

vi /etc/monit/monit.cnf

----- BEGIN:monit.cnf -----
# create RSA certs - Server

RANDFILE = ./openssl.rnd

[ req ]
default_bits = 1024
encrypt_key = yes
distinguished_name = req_dn
x509_extensions = cert_type

[ req_dn ]
countryName = Country Name (2 letter code)
countryName_default = MO

stateOrProvinceName             = State or Province Name (full name)
stateOrProvinceName_default     = Monitoria

localityName                    = Locality Name (eg, city)
localityName_default            = Monittown

organizationName                = Organization Name (eg, company)
organizationName_default        = Monit Inc.

organizationalUnitName          = Organizational Unit Name (eg, section)
organizationalUnitName_default  = Dept. of Monitoring Technologies

commonName                      = Common Name (FQDN of your server)
commonName_default              = server.monit.mo

emailAddress                    = Email Address
emailAddress_default            = root@monit.mo

[ cert_type ]
nsCertType = server
----- END:monit.cnf -----

and then generate the certificate

Code:

openssl req -new -x509 -days 365 -nodes -config ./monit.cnf -out /etc/monit/monit.pem -keyout /etc/monit/monit.pem

openssl gendh 512 >> /etc/monit/monit.pem

openssl x509 -subject -dates -fingerprint -noout -in /etc/monit/monit.pem

chmod 700 /etc/monit/monit.pem

You also can configure a client certificate to bypass the login/pwd (http://www.tildeslash.com/monit/doc/ssl.php)

6. Start MONIT

Finally start monit:

Code:

/etc/init.d/monit start

and watch /var/log/syslog for errors in monit configuration, and try to stop proftpd or other service to see if it works correctly.

then point your browser to https://www.example.com:2812/ to see the web interface

Hope this helps to increase your uptime

Albert

[Illidan]

is that not the same like the server status in ispCP?

ok, sry, it have more information

http://www.tildeslash.com/monit/doc/shoo..._httpd.gif

nice manual, i will install it at the weekend thx

[GaRCieLD]

Illidan Wrote:is that not the same like the server status in ispCP?

ok, sry, it have more information

http://www.tildeslash.com/monit/doc/shoo..._httpd.gif

nice manual, i will install it at the weekend thx

not only more information (cpu for process, memory for each process, ...), also a daemon to restart your process if it crashes, and if can't restart it send you and email...and all very configurable...

Maybe serverstatus can have the same info in 1.1

When I can install maia & amavis &... I will update the config file with this info.

thanks,
Albert

[xister]

whats with the ispcp deamon?

[GaRCieLD]

xister Wrote:whats with the ispcp deamon?

good point. I've just edited the original post to include it. Thanks!

[tioan]

GaRCieLD Wrote:

Illidan Wrote:is that not the same like the server status in ispCP?

ok, sry, it have more information

http://www.tildeslash.com/monit/doc/shoo..._httpd.gif

nice manual, i will install it at the weekend thx

not only more information (cpu for process, memory for each process, ...), also a daemon to restart your process if it crashes, and if can't restart it send you and email...and all very configurable...

Maybe serverstatus can have the same info in 1.1

When I can install maia & amavis &... I will update the config file with this info.

thanks,
Albert

monit also provide this info from shell with monit status, so it is maby a good idee to use monit als backend for the ispcp server stats so isoco can show the infos from monit status on an own ispcp page.

[BioALIEN]

Nice guide, I'll be sure to try this out once I am confident enough to use ispCP on a production server.

[Miguel]

Thanks for the howto! it works great for me!
I needed to do alittle modification when it says in the /etc/monit/monitrc file at the apache section:

Code:

and request "/monin_test_file" then restart

it should say

Code:

and request "/tools/monin_test_file" then restart

or where ever you want but it has to be the same place where you put

Code:

echo "OK" > /var/www/ispcp/gui/tools/monin_test_file

[GaRCieLD]

Thanks Miguel, I've just edited the post to correct this.

[GaRCieLD]

another change...

In monitrc, if you leave this line,

# ISPCP_DAEMON
check process ispcp_daemon with pidfile /var/run/ispcp-daemon.pid
group services
start program = "/etc/init.d/ispcp_daemon start"
stop program = "/etc/init.d/ispcp_daemon stop"
if failed port 9876 then restart
if 5 restarts within 5 cycles then timeout

in /var/log/daemon you will receive every minute something like this, so I've commented it in the first post.

Code:

Jun 21 12:43:12 test_ispcp ispcp_daemon[23642]: child 23642 started !
Jun 21 12:43:12 test_ispcp ispcp_daemon[23642]: Aeee! SIG_PIPE was received ! Will we survive ?
Jun 21 12:43:12 test_ispcp ispcp_daemon[23642]: send_line(): socket write error: Broken pipe
Jun 21 12:43:12 test_ispcp ispcp_daemon[23642]: read_line(): socket EOF ! other end closed the connection !

Jun 21 12:44:14 test_ispcp ispcp_daemon[23733]: child 23733 started !
Jun 21 12:44:14 test_ispcp ispcp_daemon[23733]: Aeee! SIG_PIPE was received ! Will we survive ?
Jun 21 12:44:14 test_ispcp ispcp_daemon[23733]: send_line(): socket write error: Broken pipe
Jun 21 12:44:14 test_ispcp ispcp_daemon[23733]: read_line(): socket EOF ! other end closed the connection !

Jun 21 12:45:16 test_ispcp ispcp_daemon[24168]: child 24168 started !
Jun 21 12:45:16 test_ispcp ispcp_daemon[24168]: Aeee! SIG_PIPE was received ! Will we survive ?
Jun 21 12:45:16 test_ispcp ispcp_daemon[24168]: send_line(): socket write error: Broken pipe
Jun 21 12:45:16 test_ispcp ispcp_daemon[24168]: read_line(): socket EOF ! other end closed the connection !