OSDir


[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: 4.11 without Host-HA framework


Hi Parth


I remember you saying this worked for you in a previous thread.


I am beginning to wonder if it is the fact I have used 3 separate NICs, one for management, one for the VM traffic and the third for storage that I am not seeing the behaviour you saw.


That is why, I too would like to understand exactly what is talking to what and doing checks for both non Host-HA and Host-HA.


I did get failover working in some scenarios with Host-HA and OOBM using IPMI but it was slow even after tweaking the timers eg. for a crashed host the best time i got was around 8 minutes which seems a long time but perhaps that is an acceptable time for CS, I just don't know.


Not expecting it to be instantaneous as it needs to do checks etc.


Jon


________________________________
From: Parth Patel <parthpatel2597@xxxxxxxxx>
Sent: 24 May 2018 06:52
To: users@xxxxxxxxxxxxxxxxxxxxx
Subject: Re: 4.11 without Host-HA framework

Hi Jon and Angus,

I did not shutdown the VMs as Yiping Zhang said, but I have confirmed this
and discussed earlier in the users list that my HA-enabled VMs got started
on another suitable available host in the cluster even when I didn't have
IPMI-enabled hardware and did no configuration for OOBM and Host-HA. I
simply pulled the ethernet cable connecting the host to entire network (I
did use just one NIC) and according to the value set in ping timeout event,
the HA-enabled VMs were restarted on another available host. I tested the
scenario using both the scenarios: the echo command as well as good old
plugging out the NIC from the host. My VMs were successfully started on
another available host after CS manager confirmed they were not reachable.

I too want to understand how the failover mechanism in CloudStack actually
works. I used ACS 4.11 packages available here:
http://cloudstack.apt-get.eu/centos/7/4.11/

Regards,
Parth Patel


On Thu, 24 May 2018 at 10:53 Paul Angus <paul.angus@xxxxxxxxxxxxx> wrote:

> I'm afraid that is not a host crash.  When shutting down the guest OS, the
> CloudStack agent on the host is still able to report to the management
> server that the VM has stopped.
>
> This is my point. VM-HA relies on the management sever communication with
> the host agent.
>
> Kind regards,
>
> Paul Angus
>
> paul.angus@xxxxxxxxxxxxx
> www.shapeblue.com<http://www.shapeblue.com>
> 53 Chandos Place, Covent Garden, London  WC2N 4HSUK
> @shapeblue
>
>
>
>
> -----Original Message-----
> From: Yiping Zhang <yzhang@xxxxxxxxxxx>
> Sent: 24 May 2018 00:44
> To: users@xxxxxxxxxxxxxxxxxxxxx
> Subject: Re: 4.11 without Host-HA framework
>
> I can say for fact that VM's using a HA enabled service offering will be
> restarted by CS on another host, assuming there are enough
> capacity/resources in the cluster, when their original host crashes,
> regardless that host comes back or not.
>
> The simplest way to test VM HA feature with a VM instance using HA enabled
> service offering is to issue shutdown command in guest OS, and watching it
> gets restarted by CS manager.
>
> On 5/23/18, 1:23 PM, "Paul Angus" <paul.angus@xxxxxxxxxxxxx> wrote:
>
>     Hi Jon,
>
>     Don't worry, TBH I'm dubious about those claiming to have VM-HA
> working when a host crashes (but doesn't restart).
>     I'll check in with the guys that set values for host-ha when testing,
> to see which ones they change and what they set them to.
>
>     paul.angus@xxxxxxxxxxxxx
>     www.shapeblue.com<http://www.shapeblue.com>
>     53 Chandos Place, Covent Garden, London  WC2N 4HSUK
>     @shapeblue
>
>
>
>
>     -----Original Message-----
>     From: Jon Marshall <jms.123@xxxxxxxxxxxxx>
>     Sent: 23 May 2018 21:10
>     To: users@xxxxxxxxxxxxxxxxxxxxx
>     Subject: Re: 4.11 without Host-HA framework
>
>     Rohit / Paul
>
>
>     Thanks again for answering.
>
>
>     I am a Cisco guy with an ex Unix background but no virtualisation
> experience and I can honestly say I have never felt this stupid before 😊
>
>
>     I have Cloudstack working but failover is killing me.
>
>
>     When you say VM HA relies on the host telling CS the VM is down how
> does that work because if you crash the host how does it tell CS anything ?
> And when you say tell CS do you mean the CS manager  ?
>
>
>     I guess I am just not understanding all the moving parts. I have had
> HOST HA working (to an extent) although it takes a long time to failover
> even after tweaking the timers but the fact that I keep finding references
> to people saying even without HOST HA it should failover (and mine doesn't)
> makes me think I have configured it incorrectly somewhere along the line.
>
>
>     I have configured a compute offering with HA and I am crashing the
> host with the echo command as suggested but still nothing.
>
>
>     I understand what you are saying Paul about it not being a good idea
> to rely on VM HA so I will go back to Host HA and try to speed up failover
> times.
>
>
>     Can I ask, from your experiences, what is a realistic fail over time
> for CS ie. if a host fails for example ?
>
>
>     Jon
>
>
>
>
>     ________________________________
>     From: Paul Angus <paul.angus@xxxxxxxxxxxxx>
>     Sent: 23 May 2018 19:55
>     To: users@xxxxxxxxxxxxxxxxxxxxx
>     Subject: RE: 4.11 without Host-HA framework
>
>     Jon,
>
>     As Rohit says, it is very important to understand the difference
> between VM HA and host HA.
>     VM HA relies on the HOST telling CloudStack that the VM is down on
> order for CloudStack start it again (wherever that ends up being).
>     Any sequence of events that ends up with VM HA restarting the VM when
> CloudStack can't contact the host is luck/fluke/unreliable/bad(tm)
>
>     The purpose of Host HA was to create a reliable mechanism to determine
> that a host has 'crashed' and that the VMs within it are inoperative. Then
> take appropriate action, including ultimately telling VM HA to restart the
> VM elsewhere.
>
>
>
>
>
>     paul.angus@xxxxxxxxxxxxx
>     www.shapeblue.com<http://www.shapeblue.com>
>     Shapeblue - The CloudStack Company<http://www.shapeblue.com/>
>     www.shapeblue.com<http://www.shapeblue.com>
>     ShapeBlue are the largest independent integrator of CloudStack
> technologies globally and are specialists in the design and implementation
> of IaaS cloud infrastructures for both private and public cloud
> implementations.
>
>
>
>     53 Chandos Place, Covent Garden, London  WC2N 4HSUK @shapeblue
>
>
>
>
>     -----Original Message-----
>     From: Rohit Yadav <rohit.yadav@xxxxxxxxxxxxx>
>     Sent: 23 May 2018 10:45
>     To: users@xxxxxxxxxxxxxxxxxxxxx
>     Subject: Re: 4.11 without Host-HA framework
>
>     Jon,
>
>
>     In the VM's compute offering, make sure that HA is ticked/enabled.
> Then use that HA-enabled VM offering while deploying a VM. Around testing -
> it depends how you're crashing. In case of KVM, you can try to cause host
> crash (example: echo c > /proc/sysrq-trigger) and see if HA-enabled VMs
> gets started on a different host.
>
>
>     - Rohit
>
>     <https://cloudstack.apache.org>
>
>
>
>     ________________________________
>     From: Jon Marshall <jms.123@xxxxxxxxxxxxx>
>     Sent: Tuesday, May 22, 2018 8:28:06 PM
>     To: users@xxxxxxxxxxxxxxxxxxxxx
>     Subject: Re: 4.11 without Host-HA framework
>
>     Hi Rohit
>
>
>     Thanks for responding.
>
>
>     I have not had much luck with HA at all.  I crash a server and nothing
> happens  in terms of VMs migrating to another host. Monitoring the
> management log file it seems the management server recognises the host has
> stopped responding to pings but doesn't think it has to do anything.
>
>
>     I am currently running v4.11 with basic network but 3 separate NICs,
> one for management, one for storage and one for VMs themselves.
>
>
>     Should it make it any difference ie. would it be worth trying to run
> management and storage over the same NIC ?
>
>
>     I am just lost as to why I see no failover at all whereas others are
> reporting it works fine.
>
>
>     Jon
>
>
>     ________________________________
>     From: Rohit Yadav <rohit.yadav@xxxxxxxxxxxxx>
>     Sent: 22 May 2018 12:12
>     To: users@xxxxxxxxxxxxxxxxxxxxx
>     Subject: Re: 4.11 without Host-HA framework
>
>     Hi Jon,
>
>
>     Yes, Host-HA is different from VM-HA and without Host HA enabled a HA
> enabled VM should be recovered/run on a different host when it crashes.
> Historically the term 'HA' in CloudStack is used around high availability
> of a VM.
>
>
>     Host HA as the name tries to imply is around HA of a physical
> hypervisor host by means of out-of-band management technologies such as
> ipmi and currently supporting ipmi as OOBM and KVM hosts with NFS storage.
>
>
>     - Rohit
>
>     <https://cloudstack.apache.org>
>     [https://cloudstack.apache.org/images/monkey-144.png]<
> https://cloudstack.apache.org/>
>
>     Apache CloudStack: Open Source Cloud Computing<
> https://cloudstack.apache.org/>
>     cloudstack.apache.org
>     CloudStack is open source cloud computing software for creating,
> managing, and deploying infrastructure cloud services
>
>
>
>
>
>
>     ________________________________
>     From: Jon Marshall <jms.123@xxxxxxxxxxxxx>
>     Sent: Monday, May 21, 2018 8:36:04 PM
>     To: users@xxxxxxxxxxxxxxxxxxxxx
>     Subject: 4.11 without Host-HA framework
>
>     I keep seeing conflicting information about this in the mailing lists
> and in blogs etc.
>
>     If I run 4.11 without enabling Host HA framework should HA still work
> if I crash a compute node because my understanding was the new framework
> was added for certain cases only.
>
>     It doesn't work for me but I can find a number of people saying you
> don't need to enable the new framework for it to work.
>
>     Thanks
>
>     Jon
>
>     rohit.yadav@xxxxxxxxxxxxx
>     www.shapeblue.com<http://www.shapeblue.com>
>     53 Chandos Place, Covent Garden, London  WC2N 4HSUK @shapeblue
>
>
>
>
>     rohit.yadav@xxxxxxxxxxxxx
>     www.shapeblue.com<http://www.shapeblue.com>
>     53 Chandos Place, Covent Garden, London  WC2N 4HSUK @shapeblue
>
>
>
>
>
>