Rsnapshot resuming

On Tue, Aug 22, 2006 at 08:46:25AM -0700, Edward Glen wrote:

> I looked through the site but didn't see a location for feature
> requests or features that are in progress so I wanted to pass this one
> your way.

The best place for that is the rsnapshot-discuss mailing list, to which
I have CCed this reply.  I very strongly recommend joining at:
  http://lists.sourceforge.net/mailman/listinfo/rsnapshot-discuss

> I use rsnapshot for backing up a number of computers through ssh...
> the problem is that one of these computers (a mac) frequently gets a
> large amount of data (photos) all at one time.  When this happens, it
> seems rsync is unable to get all the data from it before the
> connection gets dropped or rsync otherwise fails.  The backup then
> gets rolled back and starts over from the beginning again the next
> time.
> 
> There are three ways I see to approach the problem.
> 1) Make sure the connection doesn't get dropped (I've tried and can't
> seem to find the root source of the problem)

This is a fairly common problem with rsync especially across WAN links.
I'm not aware of any nice way to fix it.

If the problem is specifically that a network device somewhere is going
down - eg a cable modem logging out if it sees no traffic for a certain
period of time (HATE HATE HATE) - then just sending occasional pings
alongside your backup might help.

> 2) Have a retry count for rsnapshot so that if rsync returns an error,
> it will reconnect and resume the transfer for a specified number of
> times before finally giving up and rolling back.

IME, once rsync has failed, it will fail again if you try the same
command immediately afterwards.  Waiting a few hours can help, but
obviously isn't practical in this case.  I suppose it depends on the
nature of the problem, and the problem I usually see is dead network
connections.

> 3) Have rsnapshot download all the new backups into a temporary
> directory and only replace the *.0 backup once that temporary
> directory matches the client you're backing up.  This way it would be
> able to resume every night (for daily) and hopefully eventually catch
> up to the client and get a full backup without having to restart from
> scratch every time.

This smells funny to me :-)

> I think if I were to work on this, I would adopt the 2nd approach.
> Would you be willing to accept a patch for that if I were to write it?

Certainly!  We'd want the retries to be off by default, and to be
controllable by a setting in the config file.  I suggest calling the
parameter rsync_retries, defaulting to 0.  With this turned on, I
suggest also using the --partial option to rsync, and spitting a
suitable warning to the logs.

-- 
David Cantrell | Official London Perl Mongers Bad Influence

    If I could read only one thing it would be the future, in the
    entrails of the bastard denying me access to anything else.

-------------------------------------------------------------------------
Using Tomcat but need to do more? Need to support web services, security?
Get stuff done quickly with pre-integrated technology to make your job easier
Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642