OSDir


[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: 4.11 without Host-HA framework


Thanks Jon. I've been following that thread too, have now much clear
understanding of how VM HA behaves in cloudstack. Thanks for finding a bug
and saving time of others :)

Regards,
Parth Patel

On Mon 11 Jun, 2018, 16:30 Jon Marshall, <jms.123@xxxxxxxxxxxxx> wrote:

> Hi Parth
>
>
> Just in case you have not seen my other thread, it turns out that all this
> time it has been a bug.
>
>
> Using multiple NICs with basic networking and using zone wide NFS VM HA
> just does not work. If you change to cluster wide NFS then it works fine
> (and quite quickly as well :))
>
>
> I am now going to setup Host HA and see make sure that all works as well
> using cluster NFS.
>
>
> Got there in the end :)
>
>
> Jon
>
>
>
>
>
> ________________________________
> From: Parth Patel <parthpatel2597@xxxxxxxxx>
> Sent: 24 May 2018 06:52
> To: users@xxxxxxxxxxxxxxxxxxxxx
> Subject: Re: 4.11 without Host-HA framework
>
> Hi Jon and Angus,
>
> I did not shutdown the VMs as Yiping Zhang said, but I have confirmed this
> and discussed earlier in the users list that my HA-enabled VMs got started
> on another suitable available host in the cluster even when I didn't have
> IPMI-enabled hardware and did no configuration for OOBM and Host-HA. I
> simply pulled the ethernet cable connecting the host to entire network (I
> did use just one NIC) and according to the value set in ping timeout event,
> the HA-enabled VMs were restarted on another available host. I tested the
> scenario using both the scenarios: the echo command as well as good old
> plugging out the NIC from the host. My VMs were successfully started on
> another available host after CS manager confirmed they were not reachable.
>
> I too want to understand how the failover mechanism in CloudStack actually
> works. I used ACS 4.11 packages available here:
> http://cloudstack.apt-get.eu/centos/7/4.11/
>
> Regards,
> Parth Patel
>
>
> On Thu, 24 May 2018 at 10:53 Paul Angus <paul.angus@xxxxxxxxxxxxx> wrote:
>
> > I'm afraid that is not a host crash.  When shutting down the guest OS,
> the
> > CloudStack agent on the host is still able to report to the management
> > server that the VM has stopped.
> >
> > This is my point. VM-HA relies on the management sever communication with
> > the host agent.
> >
> > Kind regards,
> >
> > Paul Angus
> >
> > paul.angus@xxxxxxxxxxxxx
> > www.shapeblue.com<http://www.shapeblue.com>
> > 53 Chandos Place, Covent Garden, London  WC2N 4HSUK
> > @shapeblue
> >
> >
> >
> >
> > -----Original Message-----
> > From: Yiping Zhang <yzhang@xxxxxxxxxxx>
> > Sent: 24 May 2018 00:44
> > To: users@xxxxxxxxxxxxxxxxxxxxx
> > Subject: Re: 4.11 without Host-HA framework
> >
> > I can say for fact that VM's using a HA enabled service offering will be
> > restarted by CS on another host, assuming there are enough
> > capacity/resources in the cluster, when their original host crashes,
> > regardless that host comes back or not.
> >
> > The simplest way to test VM HA feature with a VM instance using HA
> enabled
> > service offering is to issue shutdown command in guest OS, and watching
> it
> > gets restarted by CS manager.
> >
> > On 5/23/18, 1:23 PM, "Paul Angus" <paul.angus@xxxxxxxxxxxxx> wrote:
> >
> >     Hi Jon,
> >
> >     Don't worry, TBH I'm dubious about those claiming to have VM-HA
> > working when a host crashes (but doesn't restart).
> >     I'll check in with the guys that set values for host-ha when testing,
> > to see which ones they change and what they set them to.
> >
> >     paul.angus@xxxxxxxxxxxxx
> >     www.shapeblue.com<http://www.shapeblue.com>
> >     53 Chandos Place, Covent Garden, London  WC2N 4HSUK
> >     @shapeblue
> >
> >
> >
> >
> >     -----Original Message-----
> >     From: Jon Marshall <jms.123@xxxxxxxxxxxxx>
> >     Sent: 23 May 2018 21:10
> >     To: users@xxxxxxxxxxxxxxxxxxxxx
> >     Subject: Re: 4.11 without Host-HA framework
> >
> >     Rohit / Paul
> >
> >
> >     Thanks again for answering.
> >
> >
> >     I am a Cisco guy with an ex Unix background but no virtualisation
> > experience and I can honestly say I have never felt this stupid before 😊
> >
> >
> >     I have Cloudstack working but failover is killing me.
> >
> >
> >     When you say VM HA relies on the host telling CS the VM is down how
> > does that work because if you crash the host how does it tell CS
> anything ?
> > And when you say tell CS do you mean the CS manager  ?
> >
> >
> >     I guess I am just not understanding all the moving parts. I have had
> > HOST HA working (to an extent) although it takes a long time to failover
> > even after tweaking the timers but the fact that I keep finding
> references
> > to people saying even without HOST HA it should failover (and mine
> doesn't)
> > makes me think I have configured it incorrectly somewhere along the line.
> >
> >
> >     I have configured a compute offering with HA and I am crashing the
> > host with the echo command as suggested but still nothing.
> >
> >
> >     I understand what you are saying Paul about it not being a good idea
> > to rely on VM HA so I will go back to Host HA and try to speed up
> failover
> > times.
> >
> >
> >     Can I ask, from your experiences, what is a realistic fail over time
> > for CS ie. if a host fails for example ?
> >
> >
> >     Jon
> >
> >
> >
> >
> >     ________________________________
> >     From: Paul Angus <paul.angus@xxxxxxxxxxxxx>
> >     Sent: 23 May 2018 19:55
> >     To: users@xxxxxxxxxxxxxxxxxxxxx
> >     Subject: RE: 4.11 without Host-HA framework
> >
> >     Jon,
> >
> >     As Rohit says, it is very important to understand the difference
> > between VM HA and host HA.
> >     VM HA relies on the HOST telling CloudStack that the VM is down on
> > order for CloudStack start it again (wherever that ends up being).
> >     Any sequence of events that ends up with VM HA restarting the VM when
> > CloudStack can't contact the host is luck/fluke/unreliable/bad(tm)
> >
> >     The purpose of Host HA was to create a reliable mechanism to
> determine
> > that a host has 'crashed' and that the VMs within it are inoperative.
> Then
> > take appropriate action, including ultimately telling VM HA to restart
> the
> > VM elsewhere.
> >
> >
> >
> >
> >
> >     paul.angus@xxxxxxxxxxxxx
> >     www.shapeblue.com<http://www.shapeblue.com>
> >     Shapeblue - The CloudStack Company<http://www.shapeblue.com/>
> >     www.shapeblue.com<http://www.shapeblue.com>
> >     ShapeBlue are the largest independent integrator of CloudStack
> > technologies globally and are specialists in the design and
> implementation
> > of IaaS cloud infrastructures for both private and public cloud
> > implementations.
> >
> >
> >
> >     53 Chandos Place, Covent Garden, London  WC2N 4HSUK @shapeblue
> >
> >
> >
> >
> >     -----Original Message-----
> >     From: Rohit Yadav <rohit.yadav@xxxxxxxxxxxxx>
> >     Sent: 23 May 2018 10:45
> >     To: users@xxxxxxxxxxxxxxxxxxxxx
> >     Subject: Re: 4.11 without Host-HA framework
> >
> >     Jon,
> >
> >
> >     In the VM's compute offering, make sure that HA is ticked/enabled.
> > Then use that HA-enabled VM offering while deploying a VM. Around
> testing -
> > it depends how you're crashing. In case of KVM, you can try to cause host
> > crash (example: echo c > /proc/sysrq-trigger) and see if HA-enabled VMs
> > gets started on a different host.
> >
> >
> >     - Rohit
> >
> >     <https://cloudstack.apache.org>
> >
> >
> >
> >     ________________________________
> >     From: Jon Marshall <jms.123@xxxxxxxxxxxxx>
> >     Sent: Tuesday, May 22, 2018 8:28:06 PM
> >     To: users@xxxxxxxxxxxxxxxxxxxxx
> >     Subject: Re: 4.11 without Host-HA framework
> >
> >     Hi Rohit
> >
> >
> >     Thanks for responding.
> >
> >
> >     I have not had much luck with HA at all.  I crash a server and
> nothing
> > happens  in terms of VMs migrating to another host. Monitoring the
> > management log file it seems the management server recognises the host
> has
> > stopped responding to pings but doesn't think it has to do anything.
> >
> >
> >     I am currently running v4.11 with basic network but 3 separate NICs,
> > one for management, one for storage and one for VMs themselves.
> >
> >
> >     Should it make it any difference ie. would it be worth trying to run
> > management and storage over the same NIC ?
> >
> >
> >     I am just lost as to why I see no failover at all whereas others are
> > reporting it works fine.
> >
> >
> >     Jon
> >
> >
> >     ________________________________
> >     From: Rohit Yadav <rohit.yadav@xxxxxxxxxxxxx>
> >     Sent: 22 May 2018 12:12
> >     To: users@xxxxxxxxxxxxxxxxxxxxx
> >     Subject: Re: 4.11 without Host-HA framework
> >
> >     Hi Jon,
> >
> >
> >     Yes, Host-HA is different from VM-HA and without Host HA enabled a HA
> > enabled VM should be recovered/run on a different host when it crashes.
> > Historically the term 'HA' in CloudStack is used around high availability
> > of a VM.
> >
> >
> >     Host HA as the name tries to imply is around HA of a physical
> > hypervisor host by means of out-of-band management technologies such as
> > ipmi and currently supporting ipmi as OOBM and KVM hosts with NFS
> storage.
> >
> >
> >     - Rohit
> >
> >     <https://cloudstack.apache.org>
> >     [https://cloudstack.apache.org/images/monkey-144.png]<
> > https://cloudstack.apache.org/>
> >
> >     Apache CloudStack: Open Source Cloud Computing<
> > https://cloudstack.apache.org/>
> >     cloudstack.apache.org
> >     CloudStack is open source cloud computing software for creating,
> > managing, and deploying infrastructure cloud services
> >
> >
> >
> >
> >
> >
> >     ________________________________
> >     From: Jon Marshall <jms.123@xxxxxxxxxxxxx>
> >     Sent: Monday, May 21, 2018 8:36:04 PM
> >     To: users@xxxxxxxxxxxxxxxxxxxxx
> >     Subject: 4.11 without Host-HA framework
> >
> >     I keep seeing conflicting information about this in the mailing lists
> > and in blogs etc.
> >
> >     If I run 4.11 without enabling Host HA framework should HA still work
> > if I crash a compute node because my understanding was the new framework
> > was added for certain cases only.
> >
> >     It doesn't work for me but I can find a number of people saying you
> > don't need to enable the new framework for it to work.
> >
> >     Thanks
> >
> >     Jon
> >
> >     rohit.yadav@xxxxxxxxxxxxx
> >     www.shapeblue.com<http://www.shapeblue.com>
> >     53 Chandos Place, Covent Garden, London  WC2N 4HSUK @shapeblue
> >
> >
> >
> >
> >     rohit.yadav@xxxxxxxxxxxxx
> >     www.shapeblue.com<http://www.shapeblue.com>
> >     53 Chandos Place, Covent Garden, London  WC2N 4HSUK @shapeblue
> >
> >
> >
> >
> >
> >
>