Redirect to offsite server during downtime

[pgentoo]

I just had a bit of downtime due to issues out of my control at my datacenter and it got me thinking...

In this case, all clients simply get server timeouts. DNS could easily be switched for all the websites, to point to some other IP address, which has a "Server under maintenance" page. This would at least give some feedback to the user that the site does exist, something is temporarily wrong, and it will be fixed soon.

I have some scripts to update all the serials on my standalone nameserver, but I didn't have scripts written to update all the ip's in the bind files, to point to another common IP. I was thinking this could just be done on a copy of the files, and then the named.conf adjusted to include the modified zones instead of the normal ones. Once the server is backup, then named could be switched back to the original configuration and the temporary zones discarded.

I want to put together some scripts to do this, but I figured others must already be doing similar stuff, and maybe I could just leverage something already written.

It would be even nicer, if when ispcp generates its bind conf, it could generate two sets of zone files, one for normal operation, and one for a disaster scenario like mentioned above. All these zones could point to some offsite server that should never be down at the same time as the core servers. Then the only change would be to tell bind to load the "maintenance" set of zones, instead of the normal ones.

Thoughts? Ideas? Someone want to write it?

Thanks,
pgentoo

[kilburn]

Your approach assumes that bind will keep working in this "disaster" scenario, so I'm not sure about which kind of "disaster" you are speaking about. Two options for me:

-) Total machine disaster, panel upgrade failed and everything is severly fucked up: your isp should allow you to move the IP to another physical server (that was previously acting as secondary nameserver), where you just would have to add an apache catch-all virtual host with the error message for all requests to this IP.

-) Partial machine disaster: you just need apache running on the local machine, and you can simply add the "mainteinance" catch-all virtual host to it (even if bind is down, the secondary nameserver should keep resolving as usual).

Your solution would only handle the case where, on the one hand, bind is not fucked up but apache is and, on the other hand, your ISP doesn't support "moving" IP's. I don't think this is a usual case...

Greets

[pgentoo]

killburn,

In my scenario, the whole power at my facility was down for some time. Another example would be long network connectivity interuption.

What i was suggesting is that if you have your secondary nameserver and maintenance page web server in a different physical location (different provider, different building, preferably geographically separate), so there is still something connected and ready to serve DNS/Web requests.

In this case, you could just go to the secondary nameserver, and flip the bind config, so it starts resolving to your maintenance pages (itself for example) until the issues have passed, then flip back to normal.

[rbtux]

well this will not solve the problem because the dns need some time for replication...

[ephigenie]

In this Cases i'd use the advanced features of my cisco / juniper and just redirect all incoming http traffic offsite.
Thus if your entire facility is down - all your panel-server are down as well - so you can't even switch to another facility / backup server if you'd like to - because its too late.

If you have external dns servers available it should be easy to just return only 1 ip for every request - redirecting your traffic to your backup site. Of course with very low ttl (180s i.e.)
But as rbtux already pointed out - if your ttl was actually 86400 before - then it'll take up to 1 day until everyone gets redirected - and then hopefully your already up & running at your main facility.

[pgentoo]

In my recent case, I couldn't redirect any traffic since all the power was down in the facility, ie, no connectivity. I know this is a huge edge case, but i figured i'd throw the idea out there...

I run an offsite DNS server, and run all my TTL's very low, so I can make changes quickly. I find that my DNS load is next to nothing, so I maintain this. If i get higher DNS load I would consider raising the TTL's. So in my case, changing the IP's in DNS would work, but i guess if TTL was higher, it wouldn't make much sense.

So noone sees much value in a setup like this? Just too low of frequency that this type of scenario would happen?

-
pgentoo

[ephigenie]

The question is ... you got your 2nd Dns outside ... ok.
And where's your ispCP installation ?

I guess inside the currently down facility.. so you cannot even use ispCP to switch something.
You will need a 2nd ispCP installation outside your facility.
Else this switch doesn't help - thats what i mean.

What may help is a complete HA - ispCP installation with heartbeat, drbd and so on. But even then you got a problem with your ip routing (cross facility routing) - so you only gather your data.

[rbtux]

depends on your setup. Assuming you use bgp4 (as every isp does) you could route a /32 IP to your ispcp box and use it as virtual address. In case one facility goes down, the ip is rerouted to the second box. (For that purpose you configure the prefix at the backupsite with a lower localpref, so when the ibgp session to the router of your mainfacility goes down, the backup route becomes the best valid route...)

[ephigenie]

ok - in this kind of setup is pretty simple ...
just create a hot-standby clone with drbd + Heartbeat and you're done.

and if there's more then one server (whose dns is to change) it gets pretty ugly without a multi-server-panel right now.

And then if you actually get to reroute your traffic into another facility for sure you're kind enough to tell your firewall to forward every packet for a server with unexpected downtime onto a highspeed single-threaded nginx/lighttpd/ tux machine with a static "out of order" page or similar (apache2.2/event_mpm with mod_mem_cache may apply as well)

That way - you don't have the hazzle of possible dns inconsistencies and : when you're first facility is up again your preferred route is up again and upstream routers will prefer that route - so your online again without throwing a bomb at your firewall again

[pgentoo]

(05-20-2009 05:24 PM)ephigenie Wrote:  The question is ... you got your 2nd Dns outside ... ok.
And where's your ispCP installation ?

I guess inside the currently down facility.. so you cannot even use ispCP to switch something.
You will need a 2nd ispCP installation outside your facility.
Else this switch doesn't help - thats what i mean.

What may help is a complete HA - ispCP installation with heartbeat, drbd and so on. But even then you got a problem with your ip routing (cross facility routing) - so you only gather your data.

Yes, my dns is offsite, and my ispcp is in the down facility (in this case). I wasn't asking for some automated wait to fail the sites over to the "down for maintenance" page... I simply was suggesting having the configs already created (another set of zone files), and then just be able to manually go into named on the offsite dns server, and change to point to the other set of zone files, should something horrible go down in the main datacenter. I wasn't asking for the ability to make this switch via ispcp panel.

---

(05-20-2009 06:50 PM)rbtux Wrote:  depends on your setup. Assuming you use bgp4 (as every isp does) you could route a /32 IP to your ispcp box and use it as virtual address. In case one facility goes down, the ip is rerouted to the second box. (For that purpose you configure the prefix at the backupsite with a lower localpref, so when the ibgp session to the router of your mainfacility goes down, the backup route becomes the best valid route...)

I'm not running my own BGP, but I suppose this is something i could get setup with my provider. I'm just not really to this point yet...

At the time i get a full hot standby setup, this would make sense for sure.