Re: replicating backup servers offsite

On 02/27 03:22 , Craig Barratt wrote:
> But it would be great to discuss the architecture of such a mechanism.
> Could it be below the application level, eg: invisible to BackupPC?

Just throwing out some brainstorm ideas... some of these are pretty
half-baked.


it may be possible to do the replication on the block device, using
something like DRBD. only way to find out if that works is to try it and see
if it breaks.

what occurred to me was something like Squid's cache replication protocol.
(Which I know nothing about either). It would be easier and more portable to
do this at the application layer... but you're right, if this is something
that can be done at a different layer, it's better to follow the Unix
philosophy of a lot of small tools working together.

what I was wondering, was if the pool replication for redundancy, could be
integrated with a heirarchichal management scheme... so if you have multiple
small servers (possibly at remote locations), you can manage them all with
one central console.

The central management console may or may not be the second-level replicant
machine... which leads me to think that the two functions are largely
orthogonal to each other. hmmm...

What first occurred to me was that if we put the central management on the
replicant server, accessing it one of the machines listed as being backed up
there, would check for a flag to see if this was a replicated share. if
indeed this was a replicant; it would attempt to contact the remote machine
to perform certain operations (like start a backup when you clicked the
button to do so), and if those operations failed, you could prompt the user
to see if they wanted to do them locally. (if the replicated server is down
and you wanted to back up a host, then run the backup from the replicant
server).

or if you wanted to restore a backup, it might first try the replicated
server to get the backup information, and failing that, try the replicant
copy. (or check to see if they're out of sync, and offer the newer one).

the complication arises when you consider that these replicated servers
might:
- be on the same network, in which case it doesn't matter which one you get
  information from; going from the server to the user is roughly equal cost
  for each of them.
- be running over a WAN link, where it's really important for performance to
  choose one server over the other. However the up-to-date information might be
  slow, but the old information might be good enough for what the user
  wants, and far faster to get to.

I think some sensible defaults will cover 75% of cases... but can really
irritate people that remaining 25%.

The important thing is to only show people as much complexity as needed. (if
the option doesn't make sense, don't show it... so if no replication is set
up, some widgets and options shouldn't be displayed).


as for the actual mechanics of syncing the pools... can we just check for
the remote list of backups for a given host, compare that to the local list
of backup numbers for that same host, and tar+compress+ssh those backups
from one machine to another and let the nightly job deal with the hardlink
issues?
or will we get better results from a full rsync operation?
can we mix the two... doing tar transfers most of the time, since that'll be
more efficient (we already know what files have changed because they're in a
given backup tree), with occasional checks from rsync operations over the
whole tree? (and watch out for rsync breaking from insufficient memory,
giving the user a helpful error message in that case).


-- 
Carl Soderstrom
Systems Administrator
Real-Time Enterprises
www.real-time.com


-------------------------------------------------------
SF email is sponsored by - The IT Product Guide
Read honest & candid reviews on hundreds of IT Products from real users.
Discover which products truly live up to the hype. Start reading now.
http://ads.osdn.com/?ad_id=6595&alloc_id=14396&op=click
_______________________________________________
BackupPC-users mailing list
BackupPC-users@xxxxxxxxxxxxxxxxxxxxx
https://lists.sourceforge.net/lists/listinfo/backuppc-users
http://backuppc.sourceforge.net/