Real HA cluster implementation

[alexskynet]

Hi everyone

I've my cluster up and running in HA mode.

Actually it works in active/passive configuration, but i plan to have it soon working with active/active load balanced conf.

I'm using heartbeat to have the things done and a NAS to hold all the share points I need (it may be done using DRBD too)

The main goal was to have users in sync in the two members, so I've setup nsslib-mysql to have users from ISPCP stored in the database.

It works very nice.

All the changes I needed to apply were in ispcp_common_methods.pl to have the users stored in the database if I activate the NSS patch in ispcp.conf.

It has been as simple as adding very few lines and a couple of queries.

I'll now focus on a ISNATTED configuration, where a number of servers (and the DNS) will be behind load balancers.

The only important thing will be to have only one ispcp daemon running at a time to ensure correct operation.

Here is a very litte howto about setting up and running the nss patch (I am on fedora 12 so adjust your package manager commands - all commands are as root):
a) yum install nsslib-mysql
Now create the required tables: log in in your mysql and type the following:
use ispcp;
CREATE TABLE groups (
name varchar(16) NOT NULL default '',
password varchar(34) NOT NULL default 'x',
gid int(11) NOT NULL auto_increment,
PRIMARY KEY (gid)
) TYPE=MyISAM AUTO_INCREMENT=5000;
CREATE TABLE users (
username varchar(16) NOT NULL default '',
uid int(11) NOT NULL auto_increment,
gid int(11) NOT NULL default '5000',
gecos varchar(128) NOT NULL default '',
homedir varchar(255) NOT NULL default '',
shell varchar(64) NOT NULL default '/bin/bash',
password varchar(34) NOT NULL default 'x',
lstchg bigint(20) NOT NULL default '1',
min bigint(20) NOT NULL default '0',
max bigint(20) NOT NULL default '99999',
warn bigint(20) NOT NULL default '0',
inact bigint(20) NOT NULL default '0',
expire bigint(20) NOT NULL default '-1',
flag bigint(20) unsigned NOT NULL default '0',
PRIMARY KEY (uid),
UNIQUE KEY username (username),
KEY uid (uid)
) TYPE=MyISAM AUTO_INCREMENT=5000;

CREATE TABLE grouplist (
rowid int(11) NOT NULL auto_increment,
gid int(11) NOT NULL default '0',
username char(16) NOT NULL default '',
PRIMARY KEY (rowid)
) TYPE=MyISAM;
GRANT USAGE ON *.* TO `nss-root`@`localhost` IDENTIFIED BY 'yourpass'; # <----- change password
GRANT USAGE ON *.* TO `nss-user`@`localhost` IDENTIFIED BY 'yourpass'; # <----- change password

GRANT Select (`username`, `uid`, `gid`, `gecos`, `homedir`, `shell`, `password`,
`lstchg`, `min`, `max`, `warn`, `inact`, `expire`, `flag`)
ON `ispcp`.`users`
TO 'nss-root'@'localhost';
GRANT Select (`name`, `password`, `gid`)
ON `ispcp`.`groups`
TO 'nss-root'@'localhost';

GRANT Select (`username`, `uid`, `gid`, `gecos`, `homedir`, `shell`)
ON `ispcp`.`users`
TO 'nss-user'@'localhost';
GRANT Select (`name`, `password`, `gid`)
ON `ispcp`.`groups`
TO 'nss-user'@'localhost';

GRANT Select (`username`, `gid`)
ON `ispcp`.`grouplist`
TO 'nss-user'@'localhost';
GRANT Select (`username`, `gid`)
ON `ispcp`.`grouplist`
TO 'nss-root'@'localhost';

quit;

Now we set up NSS auth on the system.

Edit /etc/nsswitch.conf

Look for the lines:

passwd: files
shadow: files
group: files

And change them to look as:

passwd: files mysql
shadow: files mysql
group: files mysql

Save the file.

Edit /etc/nss.mysql.conf and /etc/mss-mysql-root.conf and place the correct username/password/database on the lines.

From now on NSS first looks into shadow/passwd and the queries the database for user data. (you need to reboot)

Now let's apply ISPCP the required changes

Edit /etc/ispcp/ispcp.con and add a line as follows (I place mine at the end of the file)
USENSSMYSQL = 1

If you set this to

USENSSMYSQL = 0

the patch is completely ignored and ISPCP works with original code

Now replace /var/www/ispcp/engine/ispcp_common_methods.pl with the attached one (you'll need to rename it .pl and make it chmod 777) and you have users stored in the database.

now you are ready to install/configure heartbeat, move your relevant directories to share points and run ispcp in HA evironment.

The code in not very clean but it works.

Any idea or suggestion is welcome.

Best regards

Alessandro Bianchi

[pgentoo]

Alessandro,

Is this patch based on the current stable release?

Did you use a script to import all your existing ispcp system users into the new tables and then remove them from the shadow/passwd/group files? If so, can you share?

[alexskynet]

And now multi active server environment!

Restarting daemons (bind-httpd) is a clue in multi active server environment.

I plan to use iNotify to have some "flag files" written in a given share point

Cron jobs will check for those files on every node in the cluster and if present, will restart/reload daemons and then remove the flag files.

Only one ispcp_daemon will run at a time.

IPs will be natted, and for now I'll simply change the relevant templates for bind and httpd

The balancers will provide automatic cluster addiction/subtraction of nodes in a completey automated way (new nodes will have to be added to a configuration file, but stopping a node will cause traffic migration on superstit nodes)

See you soon

---

(05-23-2010 01:26 PM)pgentoo Wrote:  Alessandro,

Is this patch based on the current stable release?

Did you use a script to import all your existing ispcp system users into the new tables and then remove them from the shadow/passwd/group files? If so, can you share?

Hi

yes the patch is on 1.0.5

No: to me it was a completely new installation so I didn't have to import users.

I guess it can be done using a script in PHP or Perl

Best regards

[securitywonks]

lvs in nat mode works nice. If lvs-DR (Direct Routing method) works, then it will reduce load on load balancer much further.

using nfs is nice, having option for nfs failover (active/passive setup) can be more helpful

for web servers, if nginx http server (with php-fpm and fast-cgi to process php), it will be really a big load reducing feature,

coming to mysql servers, organising different databases on different mysql servers, can help make a simple start, and once above components of the multi-server layer 4 load balanced setup are well tuned, it will already show a more scalable picture, implementing mysql clusters etc other mysql server high available concepts can go as next step,

just some thoughts, thank you

[alexskynet]

Here is the same patch against 1.0.7

Change the extension from txt to pl and set proper permissions to use it

Best regards

[R1zbear]

I have some setups worink in the real world where no "real users" exists:
dovecot, postfix, httpd and pure-ftpd are able to use mysql users with absolutely no connection with real users

[alexskynet]

I have the same setups too!
Did you use the patch or followed a different ay?