[HowTo] Rsync backup&restore + remote backup

[HowTo] Rsync backup&restore + remote backup - Printable Version

+- ispCP - Board - Support (http://www.isp-control.net/forum)
+-- Forum: ispCP Omega Contributions Area (/forum-40.html)
+--- Forum: Howtos (/forum-41.html)
+--- Thread: [HowTo] Rsync backup&restore + remote backup (/thread-5156.html)

Pages: 1 2 3 4

[HowTo] Rsync backup&restore + remote backup - Kika - 12-16-2008 09:14 AM

UPDATED for ispCP 1.0.3-1

I did the rsync backup modification in the backup script, because i had too many data for packing everything day by day The old method use lot of resources for a long time (and maybe decrease the lifetime of the HDDs). With this modification the first backup was 2 hours on my sytem (the gzip backup was much longer), but the second was only 7 minutes, because this script backup only the databases and the modificated files.

This script create a folder in the {$DOMAIN_DIR}/backups and syncronizing every file to it without the log and the phptmp.

For database backup this script will use gzip (just because i like this, not the bzip2), but you can change this.

BACKUP

open /etc/ispcp/ispcp.conf

search for (about line 146)

Code:

ZIP =

replace this line with

Code:

ZIP = rsync

open /var/www/ispcp/engine/backup/ispcp-backup-all

search for (about line 161)

Code:

} elsif ($zip eq "gzip") {

                        $db_filename = "$db_backup_file.gz";

                        $db_compresscmd = "$main::cfg{'CMD_GZIP'} --force '$db_backup_file'";

replace with

Code:

} elsif ($zip eq "gzip"  || $zip eq "rsync") {

                        $db_filename = "$db_backup_file.gz";

                        $db_compresscmd = "$main::cfg{'CMD_GZIP'} --force '$db_backup_file'";

search for (about line 267)

Code:

if( $zip eq 'bzip2' || $zip eq 'gzip' || $zip eq 'lzma') {

                                $backup_filename = "$dmn_name-backup-$date.tar.".($zip eq 'bzip2' ? 'bz2' : ($zip eq 'gzip' ? 'gz' : $zip));

                                if ($allowbackup eq "full" || $allowbackup eq "dmn") {

                                        $backup_cmd = "$cmd_tar --create --directory=$dmn_dir --$zip --file=$dmn_dir/$backup_filename --exclude=logs --exclude=phptmp --exclude=backups --exclude=$backup_filen

                                }

                                if ( $allowbackup eq "full" || $allowbackup eq "sql" ) {

                                        $rs = backup_sql($dmn_id, $dmn_name, $domain_uid, $domain_gid);

                                }

insert after

Code:

} elsif( $zip eq 'rsync') {

            $backup_filename = "";

                                if ($allowbackup eq "full" || $allowbackup eq "dmn") {

            $backup_cmd = "rsync -azv --delete-excluded  --exclude=logs --exclude=phptmp --exclude=backups $dmn_dir $dmn_dir/backups";

                                }

                                if ( $allowbackup eq "full" || $allowbackup eq "sql" ) {

                                        $rs = backup_sql($dmn_id, $dmn_name, $domain_uid, $domain_gid);

                                }

searching for (about line 292)

Code:

        if ($rs == 0) { # everything ok

insert before

Code:

        if ($zip ne "rsync") {

searching for (about line 319)

Code:

send_error_mail('backup_all_engine()', "Domain $dmn_name: Error while removing file $dmn_dir/$backup_filename!");

                                        }

                                }

insert after

Code:

}

open /var/www/ispcp/engine/backup/ispcp-backup-ispcp

search for (about line 211)

Code:

my ($db_filename, $db_compresscmd) = (undef, undef);

insert after

Code:

if ($zip eq "rsync") {

            $zip = "gzip";

        }

search for (about line 274)

Code:

my $zip = $main::cfg{'ZIP'};

insert after

Code:

if ($zip eq "rsync") {

            $zip = "gzip";

        }

open /var/www/ispcp/engine/quota/ispcp-dsk-quota

search for

Code:

my $s = `$cmd_du --exclude="backups/$domain_name-*.lzma" --exclude="backups/$domain_name-*.bz2" --exclude="backups/$domain_name-*.gz" --exclude="backups/*.sql.lzma" --exclude="backups/*.sql.bz2" --exclude="backups/*.sql.gz" --exclude="logs/*.log" -s -B1 $main::cfg{APACHE_WWW_DIR}/$domain_name`;

replace with

Code:

            my $s = `$cmd_du --exclude="backups/$domain_name" --exclude="backups/$domain_name-*.lzma" --exclude="backups/$domain_name-*.bz2" --exclude="backups/$domain_name-*.gz" --exclude="backups/*.sql.lzma" --exclude="backups/*.sql.bz2" --exclude="backups/*.sql.gz" --exclude="logs/*.log" -s -B1 $main::cfg{APACHE_WWW_DIR}/$domain_name`;

RESTORE

open /var/www/ispcp/engine/ispcp-dmn-mngr

search for (about line 3567)

Code:

################################################################################​

##

## Restore Backups Data

##

sub dmn_restore_data {

  my ($dmn_data) = @_;

  push_el(\@main::el, 'dmn_restore_data()', 'Starting...');

  if (!defined($dmn_data) || $dmn_data eq '') {

    push_el(\@main::el, 'dmn_restore_data()', 'ERROR: Undefined Input Data...');

    return -1;

  }

  my ($rs, $rdata) = ('', '');

  my ($rs0, $rdata0) = ('','');

  my $dmn_id       = @$dmn_data[0];

  my $dmn_name     = @$dmn_data[1];

  my $dmn_uid<---> = @$dmn_data[3];

  my $www_dir      = $main::cfg{'APACHE_WWW_DIR'};

  my $dmn_dir      = "$www_dir/$dmn_name";

  my $dmn_bk_dir   = "$www_dir/$dmn_name/backups";

  my $cmd_tar      = $main::cfg{'CMD_TAR'};

  my $cmd_lzma     = $main::cfg{'CMD_LZMA'};

  my $cmd_bzcat    = $main::cfg{'CMD_BZCAT'};

  my $cmd_gzcat    = $main::cfg{'CMD_GZCAT'};

  my $cmd_mysql    = $main::cfg{'CMD_MYSQL'};

  my $cmd_chown<-> = $main::cfg{'CMD_CHOWN'};

insert after

Code:

my $zip = $main::cfg{ZIP};

  my $restore_cmd = undef;

  # Restore rsync

  if ($zip eq "rsync") {

    $rs = opendir(DIR, "$dmn_bk_dir/$dmn_name");

    if (!$rs) {

      push_el(\@main::el, 'dmn_restore_data()', "ERROR: Can't open '$dmn_bk_dir/$dmn_name' directory.");

      return -1;

    }

    my @bk_files = readdir(DIR);

    closedir(DIR);

    return 0 if (scalar(@bk_files) == 0);

    foreach(@bk_files) {

      my ($fname) = ($_);

      if ($fname ne "." && $fname ne "..") {

        if ( -d "$dmn_bk_dir/$dmn_name/$fname" ) {

          $restore_cmd = "rsync -azv --delete-excluded $dmn_bk_dir/$dmn_name/$fname/ $dmn_dir/$fname/";

          $rs = sys_command($restore_cmd);

        } else {

          $restore_cmd = "rsync -azv --delete-excluded $dmn_bk_dir/$dmn_name/$fname $dmn_dir/$fname";

          $rs = sys_command($restore_cmd);

        }

      }

    }

  }

REMOTE BACKUP

If you want to create a remote backup you can do that with this simple steps below. (This is not need for the local rsync backup)

1, Generate the rsa key

Code:

ispCP server: 

ssh-keygen -N '' -C backup1 -t rsa -f ~/.ssh/backup

scp ~/.ssh/backup.pub server:.ssh

Remote backup server: 

cd ~/.ssh

cat backup.pub >> authorized_keys

rm backup.pub

2, Create backup.sh file on your ispCP server:

Code:

#!/bin/sh

for i in /var/www/virtual/*; do

  domain=${i:17:${#i}};

  rsync -v --delete --delete-excluded --timeout=999 -az -e 'ssh -c blowfish -i /root/.ssh/backup -ax -o ClearAllForwardings=yes' /var/www/virtual/$domain/backups server.tld:backup_dir/$domain/

done

Don't forget to replace the "server.tld:backup_dir" to your backup server.

3, Add this script to your crontab after the ispCP backup

4, Run backup.sh from command line at the first time (because the new key)

RE: [HowTo] Rsync backup + remote backup - Mouv - 12-16-2008 04:57 PM

Excellent... I test this during day...

Thank you

RE: [HowTo] Rsync backup + remote backup - mydebians - 12-16-2008 06:58 PM

Yes very nice, i'll test too today or tomorrow
bests regards

RE: [HowTo] Rsync backup + remote backup - Quemeros - 12-16-2008 10:38 PM

Hi Kika,
...It's for RC7, right?
Very good mod Smile

... in this and next week, i will try it and think new ideas.

RE: [HowTo] Rsync backup + remote backup - Kika - 12-16-2008 10:43 PM

Yes, this version was tested with RC7, but works with RC6 too (maybe the line numbers not equal).

RE: [HowTo] Rsync backup + remote backup - itanium - 12-24-2008 07:46 AM

Thanks for this mod! Smile

but now, how does the restore backup script work? Sad

RE: [HowTo] Rsync backup + remote backup - Kika - 12-24-2008 08:55 AM

The users could restore that files what he need via FTP, because if somebody want to restore something than usualy he need only some little file what he can download and upload to the good place.

If i will have more time than i will create a restore script to. (i never used the restore script, but i will try soon)

RE: [HowTo] Rsync backup + remote backup - itanium - 12-25-2008 12:03 PM

Thank you for your answer. I will look to change the restoration script

RE: [HowTo] Rsync backup + remote backup - Arturo - 05-03-2009 12:20 AM

someone continue using this tutorial?

RE: [HowTo] Rsync backup + remote backup - Kika - 05-03-2009 01:01 AM

I using this with the 1.0 too, but i had't time to see the automatic restore script. If somebody want to restore anything then he could use ftp Wink