osdir.com

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [PROPOSE] Backup and Recovery Framework


Thanks for the details Nicolas. I do think the picture is clear(er?) now.
At least, now I understand what you are doing.

BTW: It is a very interesting feature ;)

On Thu, Jun 7, 2018 at 8:30 PM, Will Stevens <wstevens@xxxxxxxxxxxx> wrote:

> Thanks for clarifying Nicolas.  I just wanted to make sure that there
> wasn't any confusion in this conversation as a 'snapshot' has different
> meanings to different people/vendors, so I wanted to make sure we were all
> on the same page.
>
> BTW, for what it is worth, I love this initiative.  This is an important
> feature integration which I know a LOT of organizations will appreciate.
>
> Cheers,
>
> *Will Stevens*
> Chief Technology Officer
> c 514.826.0190
>
> <https://goo.gl/NYZ8KK>
>
>
> On Thu, Jun 7, 2018 at 12:57 PM Nicolas Vazquez <
> Nicolas.Vazquez@xxxxxxxxxxxxx> wrote:
>
> > First of all, the framework is agnostic of any backup vendor/provider. It
> > just provides the way to communicate with the provider to allow users to
> > backup their VMs.
> >
> >
> > Will is right, backups are not the same as snapshots, I got confused with
> > the Veeam backup which starts from taking a snapshot to generate the
> > backup. The way they are generated are up to each vendor, and it is not
> > relevant to the framework. Sorry for the confusion, we are also
> introducing
> > the Veeam plugin in top of this framework.
> >
> >
> > So, in terms of this framework, it does not matter what a backup is, we
> > just keep a reference to an existing backup on each vendor.
> >
> >
> > I would also like to state that this framework is not using a snapshot
> > mechanism for backups in CloudStack. It is just providing interfaces and
> > mechanisms for vendors/providers plugins in CloudStack. Implementation of
> > each plugin should be relevant to the vendor and any backup handling will
> > be external to CloudStack.
> >
> >
> > I think that the diagram in the FS could explain what the task of the
> > framework is:
> > https://cwiki.apache.org/confluence/download/
> attachments/85476500/Backup%20and%20Recovery%20UML.png?
> version=1&modificationDate=1528367734183&api=v2.
> > The framework simply allows management server to communicate with backup
> > and recovery vendors/providers. To allow CloudStack communication with an
> > specific vendor, a plugin should be implemented for it (as we are doing
> > with Veeam for example).
> >
> >
> > Please let me know if it is still confusing and I'll try to explain
> better
> >
> > ________________________________
> > From: Todd Pigram <todd@xxxxxxxxxxxxxx>
> > Sent: Thursday, June 7, 2018 12:11:18 PM
> > To: dev@xxxxxxxxxxxxxxxxxxxxx
> > Subject: Re: [PROPOSE] Backup and Recovery Framework
> >
> > Now outside of our ACS, I use Veeam for vCloud and vSphere and use Alike
> > for our XS pools. Inside ACS, the VMs use MaxBackup
> >
> >
> > Nicolas.Vazquez@xxxxxxxxxxxxx
> > www.shapeblue.com
> > ,
> > @shapeblue
> >
> >
> >
> > On Thu, Jun 7, 2018 at 11:09 AM, Todd Pigram <todd@xxxxxxxxxxxxxx>
> wrote:
> >
> > > I agree completely. That is why I use inside VM backup products
> > currently.
> > >
> > > On Thu, Jun 7, 2018 at 11:07 AM, Will Stevens <wstevens@xxxxxxxxxxxx>
> > > wrote:
> > >
> > >> I have not been following this thread entirely, but I would not count
> on
> > >> our snapshot mechanism for backups.  If something like Veeam is used,
> > then
> > >> an agent is in play and that agent will quiesce the running processes,
> > >> such
> > >> as MySQL.  This is important in order to do backup and DR as you will
> > lose
> > >> transactions in memory and can potentially have a corrupted backup.
> > >>
> > >> Snapshots are not backups!!!
> > >>
> > >> *Will Stevens*
> > >> Chief Technology Officer
> > >> c 514.826.0190
> > >>
> > >> <https://goo.gl/NYZ8KK>
> > >>
> > >>
> > >> On Thu, Jun 7, 2018 at 10:13 AM Rafael Weingärtner <
> > >> rafaelweingartner@xxxxxxxxx> wrote:
> > >>
> > >> > Ok, now I start to understand.  So, we will be handling snapshots
> and
> > >> they
> > >> > will be sent to a third-party system.
> > >> >
> > >> > Now let’s proceed. If I configure this “feature” with
> external=false.
> > >> What
> > >> > does it do? Nothing? I mean, what is the purpose of this “external”
> > >> > parameter?
> > >> >
> > >> >
> > >> > On Thu, Jun 7, 2018 at 4:09 PM, Nicolas Vazquez <
> > >> > Nicolas.Vazquez@xxxxxxxxxxxxx> wrote:
> > >> >
> > >> > > Its format may depend on the provider, but under the hood it is a
> > >> > snapshot
> > >> > > which can be full or incremental and it is copied and stored in a
> > >> backup
> > >> > > repository in one of the backup servers. For example, if you use
> > >> Veeam,
> > >> > you
> > >> > > will find your backups stored as .vbk (for full backups) and .vib
> > (for
> > >> > > incremental backups). It is the task of the backup provider to
> > create
> > >> and
> > >> > > move these files to the backup repository.
> > >> > >
> > >> > > ________________________________
> > >> > > From: Rafael Weingärtner <rafaelweingartner@xxxxxxxxx>
> > >> > > Sent: Thursday, June 7, 2018 10:39:35 AM
> > >> > > To: dev
> > >> > > Subject: Re: [PROPOSE] Backup and Recovery Framework
> > >> > >
> > >> > > Ok, there are a lot of things, which are kind of blurry to me. So,
> > >> let’s
> > >> > > try to go step by step.
> > >> > >
> > >> > > Let me ask again. What is a backup of the VM? You mention
> > >> restore-point.
> > >> > > What is it? Is it a snapshot in the hypervisor (complete/partial
> > >> > snapshot)?
> > >> > > If it is not a snapshot,  what is it then? Are you installing some
> > >> agent
> > >> > in
> > >> > > the VM and then copying/mirroring the files in the VM’s file
> system?
> > >> > >
> > >> > > On Thu, Jun 7, 2018 at 3:26 PM, Nicolas Vazquez <
> > >> > > Nicolas.Vazquez@xxxxxxxxxxxxx> wrote:
> > >> > >
> > >> > > > Hi Daniel and Rafael,
> > >> > > >
> > >> > > >
> > >> > > > I have updated the diagram picture on the FS, please let me know
> > if
> > >> the
> > >> > > > problem persists.
> > >> > > >
> > >> > > >
> > >> > > > @<mailto:daniel.herrmann@xxxxxxxxxxxxxxxx>Daniel you are right
> > >> about
> > >> > > > restoring VMs, please let me update the FS. Regarding your
> > >> question, we
> > >> > > > have considered the backups as a first class citizens, meaning
> > that
> > >> a
> > >> > > > backup would continue existing unless its originating VM has
> been
> > >> > > > removed/expunged. This way, listBackups API method would list
> them
> > >> in
> > >> > > order
> > >> > > > to allow a user to restore a deleted VM from a backup.
> > >> > > >
> > >> > > >
> > >> > > > @Rafael the idea of the Dummy provider is simply returning the
> > >> expected
> > >> > > > entities from a backup provider. For example, the BackupManager
> > >> could
> > >> > ask
> > >> > > > the BackupProvider to list its backups and would expect a list
> of
> > >> > > backups.
> > >> > > > As the Dummy Provider implements the BackupProvider interface,
> it
> > >> has
> > >> > its
> > >> > > > own implementation of listBackups() method and simply returns
> some
> > >> mock
> > >> > > > entities to the Backup Manager. It does not store any backup, it
> > >> simply
> > >> > > > creates them in execution time and returns them.
> > >> > > >
> > >> > > >
> > >> > > > From the CloudStack point of view, Backups are stored
> externally,
> > on
> > >> > the
> > >> > > > Backup and Recovery provider servers and are handled by them. We
> > >> keep
> > >> > > track
> > >> > > > of the existing backups in the Backup provider. Implementation
> may
> > >> > depend
> > >> > > > on the Backup and Recovery provider, would consist on snapshots,
> > but
> > >> > are
> > >> > > > handled by the provider and not by CloudStack.
> > >> > > >
> > >> > > >
> > >> > > > Backup policies are created on the Backup and Recovery provider
> > and
> > >> > > > provide retention options and periods as well as backup times.
> For
> > >> > > example,
> > >> > > > the provider may expose Golden, Silver and Bronze policies and
> > have
> > >> > > > different options for each of them.
> > >> > > >
> > >> > > >
> > >> > > > The 'external' parameter on the listBackupPolicies API method
> > would
> > >> > only
> > >> > > > ask the Backup and Recovery provider to list their policies if
> it
> > is
> > >> > set
> > >> > > to
> > >> > > > true. If it is not set, or set to false, it just simply returns
> > the
> > >> > > > existing mapped policies in CloudStack. Initially, CloudStack
> does
> > >> not
> > >> > > > store any backup policy, those should be imported/mapped by the
> > >> > > > 'importBackupPolicy' API method.
> > >> > > >
> > >> > > >
> > >> > > > Finally, we consider backups as restore points. As mentioned
> > above,
> > >> > > > backups are stored on the Backup and Recovery provider servers,
> so
> > >> we
> > >> > > keep
> > >> > > > a mapping between the CloudStack entity and the backup, which
> can
> > be
> > >> > used
> > >> > > > to restore VMs.
> > >> > > >
> > >> > > >
> > >> > > > Please let me know if anything.
> > >> > > >
> > >> > > >
> > >> > > > Regards,
> > >> > > >
> > >> > > > Nicolas Vazquez
> > >> > > >
> > >> > > > ________________________________
> > >> > > > From: Rafael Weingärtner <rafaelweingartner@xxxxxxxxx>
> > >> > > > Sent: Thursday, June 7, 2018 6:55:57 AM
> > >> > > > To: dev
> > >> > > > Subject: Re: [PROPOSE] Backup and Recovery Framework
> > >> > > >
> > >> > > > Hi Nicolas,
> > >> > > >
> > >> > > > The idea is great. I have some questions though.
> > >> > > >
> > >> > > > Before getting to the questions, your pictures are broken, or at
> > >> least
> > >> > > they
> > >> > > > seem broken to me.
> > >> > > >
> > >> > > > How would the “dummy” backup provider work? I mean, what does it
> > do?
> > >> > > where
> > >> > > > does it storage the "backup"?
> > >> > > > What is the difference from this Backup to a Snapshot? Is it
> > simply
> > >> > > storing
> > >> > > > a snapshot outside the cloud structure?
> > >> > > > How would these “backup policies” look like? I mean, what can I
> > >> > > configure?
> > >> > > > Regarding that parameter “external” of your backup policy, how
> > >> would it
> > >> > > > work if we set it to false?
> > >> > > > What are you defining as a backup (what do you mean by backup in
> > the
> > >> > > > context of this feature?)? I mean, is it simply a snapshot file
> > that
> > >> > you
> > >> > > > store somewhere else?
> > >> > > >
> > >> > > >
> > >> > > > Nicolas.Vazquez@xxxxxxxxxxxxx
> > >> > > > www.shapeblue.com<http://www.shapeblue.com>
> > >> > > > ,
> > >> > > > @shapeblue
> > >> > > >
> > >> > > >
> > >> > > >
> > >> > >
> > >> > > Nicolas.Vazquez@xxxxxxxxxxxxx
> > >> > > www.shapeblue.com<http://www.shapeblue.com>
> > >> > > ,
> > >> > > @shapeblue
> > >> > >
> > >> > >
> > >> > >
> > >> > > > On Thu, Jun 7, 2018 at 6:05 AM, <daniel.herrmann@zv.
> fraunhofer.de
> > >
> > >> > > wrote:
> > >> > > >
> > >> > > > > Hi all,
> > >> > > > >
> > >> > > > > +1 for this feature request, thank you for bringing this to
> > life!
> > >> > > Volume
> > >> > > > > snapshots are what our users currently use as backup solution,
> > >> which
> > >> > > has
> > >> > > > > some inherent problems such as handling large volumes and the
> > >> > handling
> > >> > > of
> > >> > > > > multiple volumes (consistency).
> > >> > > > >
> > >> > > > > One thought: when restoring a VM, I think there are two cases
> to
> > >> > > > consider:
> > >> > > > >
> > >> > > > > - The VM is still in CS and the user just wants to restore an
> > old
> > >> > > version
> > >> > > > > of the VM
> > >> > > > > - The VM has already been deleted and the user wants to
> restore
> > >> the
> > >> > > > > deleted VM
> > >> > > > >
> > >> > > > > As listBackups is described as " listBackups API method. List
> > >> > existing
> > >> > > > > backups for a VM". Is the second use case supported at all?
> > >> > > > >
> > >> > > > > I like the approach not to define the backup policies in CS
> but
> > in
> > >> > > > > whatever tool is used in the end, as otherwise a lot of
> features
> > >> > might
> > >> > > be
> > >> > > > > lost. We as services providers need to create them in the
> first
> > >> > place,
> > >> > > > but
> > >> > > > > I think this is the better solution.
> > >> > > > >
> > >> > > > > Regards
> > >> > > > > Daniel
> > >> > > > >
> > >> > > > > On 06.06.18, 14:43, "Nicolas Vazquez" <
> > >> > Nicolas.Vazquez@xxxxxxxxxxxxx>
> > >> > > > > wrote:
> > >> > > > >
> > >> > > > >     Hi all,
> > >> > > > >
> > >> > > > >
> > >> > > > >     We would like to introduce a new framework into CloudStack
> > >> that
> > >> > > will
> > >> > > > > allow adding Backup and Recovery providers as plugins. This
> > >> framework
> > >> > > > goal
> > >> > > > > is allowing users to backup their guest VMs for recovery
> > purposes.
> > >> > > > >
> > >> > > > >
> > >> > > > >     Please find the FS on this link:
> https://cwiki.apache.org/
> > >> > > > > confluence/display/CLOUDSTACK/Backup+and+Recovery+Framework
> > >> > > > >
> > >> > > > >
> > >> > > > >     Regards,
> > >> > > > >
> > >> > > > >     Nicolas Vazquez
> > >> > > > >
> > >> > > > >     Nicolas.Vazquez@xxxxxxxxxxxxx
> > >> > > > >     www.shapeblue.com<http://www.shapeblue.com>
> > >> > > > >     ,
> > >> > > > >     @shapeblue
> > >> > > > >
> > >> > > > >
> > >> > > > >
> > >> > > > >
> > >> > > > >
> > >> > > > >
> > >> > > >
> > >> > > >
> > >> > > > --
> > >> > > > Rafael Weingärtner
> > >> > > >
> > >> > >
> > >> > >
> > >> > >
> > >> > > --
> > >> > > Rafael Weingärtner
> > >> > >
> > >> >
> > >> >
> > >> >
> > >> > --
> > >> > Rafael Weingärtner
> > >> >
> > >>
> > >
> > >
> > >
> > > --
> > >
> > >
> > > Todd Pigram
> > > http://about.me/ToddPigram
> > > www.linkedin.com/in/toddpigram/<http://www.linkedin.com/in/toddpigram/
> >
> > > @pigram86 on twitter
> > > https://plus.google.com/+ToddPigram86
> > > Mobile - 216-224-5769
> > >
> > > PGP Public Key
> > > <http://pgp.mit.edu/pks/lookup?op=get&search=0x96B7B0F0C55933BB>
> > >
> > >
> >
> >
> > --
> >
> >
> > Todd Pigram
> > http://about.me/ToddPigram
> > www.linkedin.com/in/toddpigram/<http://www.linkedin.com/in/toddpigram/>
> > @pigram86 on twitter
> > https://plus.google.com/+ToddPigram86
> > Mobile - 216-224-5769
> >
> > PGP Public Key
> > <http://pgp.mit.edu/pks/lookup?op=get&search=0x96B7B0F0C55933BB>
> >
>



-- 
Rafael Weingärtner