ispcp needs backup engine overhaul...

[pgentoo]

I'm having some issues with backups my boxes. Mostly around performance when backups are running, pretty much bringing my websites to a halt due to high i/o wait %'s.

First, it seems that backups settings (full, db only, etc) can only be configured when adding a user, and not after the fact?

More importantly, currently backups are handled to the same logical volume as the websites (/var/www/virtual, for example) just under a backups folder, which means that backups thrash heavily reading and writing from the same volume. In a setup that isn't backed by very fast spindles (and a lot of them), this causes a lot of load on the system, and can slow down sites to the point where they stop responding. Further, it would be better use of resources to say put backups on a not so fault tolerant set of spindles (say RAID0 instead of RAID10) to further speed up the backup operations. Ideally we would natively have support for putting backups off on another server via rsync over ssh or something to get them off the box should something really bad happen.

I also have some sites that write data into large files (think sqlite, application specific log files, etc) and if files are modified while the tar operation is in process, the backup breaks entirely...

I'd like to propose the idea of moving to an (optional?) mirrored type backup, where files can be rsync'd to an identical copy on a specified folder (or even backup server in the future). Once rsync'd over, they could be tar'd, and optionally zipped (lzma, bzip2, gzip, etc) for a daily copy. I used this approach on a server/control panel that i developed myself back in the day, and it worked out very well, removing the i/o wait issues from the "production" volumes, and of to other spindles where load would not affect runtime performance (aside from cpu, but that can be mitigated with 'nice').

Backups could be off on the other volume, and ispcp could create symlinks out to the users folder under the backups volume, so that retrieval via ftp and recovery could still work the same.

I think backups are a big weakness in ispcp right now once you start adding more than few sites, especially if the sites have a lot of files (think large photogalleries, etc). I know that personally backups are not running (completely) for me on several boxes due to the issues I mentioned, which means that i have to run my own rsync's and such to get data to a safe place.

I'm really looking for feedback on this, to see what people think, and to see if others are having the same issues. If its a wide spread thing, I really think some effort should be focused in this area to make ispcp a more robust product.

Thoughts?

[Lucan]

Hey,

i agree with you, also we may should add the possibility for inkremental backups -> that would reduce the backup time also

[Kika]

from the 1.0.7 i will continue to develop the rsync backup mod.

http://isp-control.net/forum/thread-5156.html

[kassah]

Incremental backups would be nice. Although restoring from those seems like it would be a pain.

[Nuxwin]

Hello ;

I agreed with you too. Since the begin, i know that this feature is very.... :

1. Coded like my ass...
2. Can kill all servers (same big servers)..

For now, I should eat but later, I'll try to answer point per point...

[pgentoo]

Nuxwin, I'm interested to hear your thoughts on this and how we can rework it into future ispcp versions.

---

Nuxwin, I'm interested to hear your thoughts on this and how we can rework it into future ispcp versions.

[kilburn]

The worst thing about backups is that there's a myriad of ways to handle them, and trying to code something that adapts to everyone's needs is nearly impossible.

For example, I have set up my own backups solution, where:

- My servers are backed up 6 times a day (every 4 hours) to an external backup machine. Copies are time-rolling incremental (using rsnapshot), meaning that, for each server, I have: (1) A copy of its state 4,8,12,16 and 20 hours ago; (2) A copy of its state 1, 2, 3, 4, 5 and 6 days ago; (3) a copy of its state 1, 2, and 3 weeks ago; (4) a copy of its state 1, 2, 3, 4, 5, and 6 moths ago; and (5) a copy of its state 1 year ago. Without storing duplicated data twice, of course.
- The backup server is heavily protected, so it is the one initiating copies (this is, the servers can *not* connect to the backup box for security reasons).
- Backups are made over an lvm snapshot, to avoid the "files being modified" problem.
- Additionally, the backup process runs at a low I/O priority to avoid impacting the server while running. This is, users hardly notice that backups are being made during their execution.

What I'm trying to say with this is that IMHO the best option we have is to provide some *basic* backup functionality, along with some howtos describing alternative backup schemes (such as kurgans' how to transfer an ispcp server using rsync).