[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: Questions on HA cluster and split brain


Thanks, again, for your quick response.

Anindya Haldar
Oracle Marketing Cloud


> On Jun 14, 2018, at 3:34 PM, Justin Bertram <jbertram@xxxxxxxxxx> wrote:
> 
>> 1) It is possible to define multiple groups within a cluster, and a
> subset of the brokers in the cluster can be members of a specific group. Is
> that correct?
> 
> Yes.
> 
>> 2) The live-backup relationship is guided by group membership, when there
> is explicit group membership defined. Is that correct?
> 
> Yes.
> 
>> 3) When a backup or a live server in a group starts the quorum voting
> process, other live servers in the cluster, even if though they may not be
> part of the same group, can participate in the quorum. Meaning the ability
> to participate in quorum voting is defined by cluster membership, and not
> by group membership within the cluster. Is that understanding correct?
> 
> Yes.
> 
> 
> In short, a "group" allows the pairing of specific live and backup brokers
> together in the replicated HA use-case.
> 
> 
> Justin
> 
> 
> On Thu, Jun 14, 2018 at 5:19 PM, Anindya Haldar <anindya.haldar@xxxxxxxxxx>
> wrote:
> 
>> I have a few quick follow up questions. From the discussion here, and from
>> what I understand reading the Artemis manual, here is my understanding
>> about the idea of a cluster vs. the idea of a group within a cluster:
>> 
>> 1) It is possible to define multiple groups within a cluster, and a subset
>> of the brokers in the cluster can be members of a specific group. Is that
>> correct?
>> 
>> 2) The live-backup relationship is guided by group membership, when there
>> is explicit group membership defined. Is that correct?
>> 
>> 3) When a backup or a live server in a group starts the quorum voting
>> process, other live servers in the cluster, even if though they may not be
>> part of the same group, can participate in the quorum. Meaning the ability
>> to participate in quorum voting is defined by cluster membership, and not
>> by group membership within the cluster. Is that understanding correct?
>> 
>> Thanks,
>> 
>> Anindya Haldar
>> Oracle Marketing Cloud
>> 
>> 
>>> On Jun 14, 2018, at 9:57 AM, Anindya Haldar <anindya.haldar@xxxxxxxxxx>
>> wrote:
>>> 
>>> Many thanks, Justin. This makes things much clearer for us when it comes
>> to designing the HA cluster.
>>> 
>>> As for the Artemis evaluation scope, we want to use it as one of the
>> supported messaging backbones in our application suite. The application
>> suite requires strong transactional guarantees, high availability, and high
>> performance and scale, amongst other things. We are looking towards a full
>> blown technology evaluation with those needs in mind.
>>> 
>>> Thanks,
>>> 
>>> Anindya Haldar
>>> Oracle Marketing Cloud
>>> 
>>> 
>>>> On Jun 13, 2018, at 7:23 PM, Justin Bertram <jbertram@xxxxxxxxxx>
>> wrote:
>>>> 
>>>>> Q1: At this point, will the transaction logs replicate from A to C?
>>>> 
>>>> No.  A will be replicating to B since B is the designated backup.
>> Also, by
>>>> "transaction logs" I assume you mean what the Artemis documentation
>> refers
>>>> to as the journal (i.e. all persistent message data).
>>>> 
>>>>> Q2: At this point will C become to new new back up for B, assuming A
>>>> remains in failed state?
>>>> 
>>>> Yes.
>>>> 
>>>>> Q3: If the answer to Q2 is yes, B will start replicating its journals
>> to
>>>> C; is that correct?
>>>> 
>>>> Yes.
>>>> 
>>>>> Q4: At this point, which nodes are expected to participate in quorum
>>>> voting? All of A, B and C? Or A and C only (B excludes itself from the
>>>> set)? When it says "half the servers”, I read it in a way that B
>> includes
>>>> itself in the quorum voting. Is that the case?
>>>> 
>>>> A would be the only server available to participate in the quorum voting
>>>> since it is the only live server.  However, since B can't reach A then B
>>>> would not receive any quorum vote responses.  B doesn't vote; it simply
>>>> asks for a vote.
>>>> 
>>>>> Q5: This implies only the live servers participate in quorum voting. Is
>>>> that correct?
>>>> 
>>>> Yes.
>>>> 
>>>>> Q6: If the answer to Q5 is yes, then how does the split brain detection
>>>> (as described in the quoted text right before Q4) work?
>>>> 
>>>> It works by having multiple voting members (i.e. live servers) in the
>>>> cluster.  The topology you've described with a single live and 2
>> backups is
>>>> not sufficient to mitigate against split brain.
>>>> 
>>>>> Q7: The text implies that in order to avoid split brain, a cluster
>> needs
>>>> at least 3 live/backup PAIRS.
>>>> 
>>>> That is correct - 3 live/backup pairs.
>>>> 
>>>>> To me that implies at least 6 broker instances are needed in such a
>>>> cluster; but that is kind of hard to believe, and I feel (I may be
>> wrong)
>>>> it actually means 3 broker instances, assuming scenarios 1 and 2 as
>>>> described earlier are valid ones. Can you please clarify?
>>>> 
>>>> What you feel is incorrect.  That said, the live & backup instances can
>> be
>>>> colocated which means although there are 6 total broker instances only 3
>>>> machines are required.
>>>> 
>>>> I think implementing a feature whereby backups can participate in the
>>>> quorum vote would be a great addition to the broker.  Unfortunately I
>>>> haven't had time to contribute such a feature.
>>>> 
>>>> 
>>>> If I may ask a question of my own...Your emails to this list have
>> piqued my
>>>> interest and I'm curious to know to what end you are evaluating Artemis
>>>> since you apparently work for Oracle on a cloud related team and Oracle
>>>> already has a cloud messaging solution.  Can you elaborate at all?
>>>> 
>>>> 
>>>> Justin
>>>> 
>>>> 
>>>> On Wed, Jun 13, 2018 at 7:56 PM, Anindya Haldar <
>> anindya.haldar@xxxxxxxxxx>
>>>> wrote:
>>>> 
>>>>> BTW, these are questions related to Artemis 2.4.0, which is what we are
>>>>> evaluating right now for our solution.
>>>>> 
>>>>> 
>>>>>> On Jun 13, 2018, at 5:52 PM, Anindya Haldar <
>> anindya.haldar@xxxxxxxxxx>
>>>>> wrote:
>>>>>> 
>>>>>> I have some questions related to the HA cluster, failover and
>>>>> split-brain cases.
>>>>>> 
>>>>>> Suppose I have set up a 3 node cluster with:
>>>>>> 
>>>>>> A = master
>>>>>> B = slave 1
>>>>>> C = slave 2
>>>>>> 
>>>>>> Also suppose they are all part of same group, and are set up to offer
>>>>> replication based HA.
>>>>>> 
>>>>>> Scenario 1
>>>>>> ========
>>>>>> Say,
>>>>>> 
>>>>>> B starts up and finds A
>>>>>> B becomes the designated backup for A
>>>>>> C starts up, and tries to find a live server in this group
>>>>>> C figures that A already has a designated backup, which is B
>>>>>> C keeps waiting until the network topology is changed
>>>>>> 
>>>>>> 
>>>>>> Q1: At this point, will the transaction logs replicate from A to C?
>>>>>> 
>>>>>> Now let’s say
>>>>>> 
>>>>>> Node A (the current master) fails
>>>>>> B becomes the new master
>>>>>> 
>>>>>> Q2: At this point will C become to new new back up for B, assuming A
>>>>> remains in failed state?
>>>>>> 
>>>>>> Q3: If the answer to Q2 is yes, B will start replicating its journals
>> to
>>>>> C; is that correct?
>>>>>> 
>>>>>> 
>>>>>> Scenario 2 (split brain detection case)
>>>>>> =============================
>>>>>> Say,
>>>>>> 
>>>>>> B detects a transient network failure with A
>>>>>> B wants to figure out if it needs to take over and be the new master
>>>>>> B starts a quorum voting process
>>>>>> 
>>>>>> The manual says this in the ‘High Availability and Failover’ section:
>>>>>> 
>>>>>> "Specifically, the backup will become active when it loses connection
>> to
>>>>> its live server. This can be problematic because this can also happen
>>>>> because of a temporary network problem. In order to address this
>> issue, the
>>>>> backup will try to determine whether it still can connect to the other
>>>>> servers in the cluster. If it can connect to more than half the
>> servers, it
>>>>> will become active, if more than half the servers also disappeared
>> with the
>>>>> live, the backup will wait and try reconnecting with the live. This
>> avoids
>>>>> a split brain situation."
>>>>>> 
>>>>>> Q4: At this point, which nodes are expected to participate in quorum
>>>>> voting? All of A, B and C? Or A and C only (B excludes itself from the
>>>>> set)? When it says "half the servers”, I read it in a way that B
>> includes
>>>>> itself in the quorum voting. Is that the case?
>>>>>> 
>>>>>> Whereas in the ‘Avoiding Network Isolation’ section, the manual says
>>>>> this:
>>>>>> 
>>>>>> “Quorum voting is used by both the live and the backup to decide what
>> to
>>>>> do if a replication connection is disconnected. Basically the server
>> will
>>>>> request each live server in the cluster to vote as to whether it
>> thinks the
>>>>> server it is replicating to or from is still alive. This being the
>> case the
>>>>> minimum number of live/backup pairs needed is 3."
>>>>>> 
>>>>>> Q5: This implies only the live servers participate in quorum voting.
>> Is
>>>>> that correct?
>>>>>> 
>>>>>> Q6: If the answer to Q5 is yes, then how does the split brain
>> detection
>>>>> (as described in the quoted text right before Q4) work?
>>>>>> 
>>>>>> Q7: The text implies that in order to avoid split brain, a cluster
>> needs
>>>>> at least 3 live/backup PAIRS. To me that implies at least 6 broker
>>>>> instances are needed in such a cluster; but that is kind of hard to
>>>>> believe, and I feel (I may be wrong) it actually means 3 broker
>> instances,
>>>>> assuming scenarios 1 and 2 as described earlier are valid ones. Can you
>>>>> please clarify?
>>>>>> 
>>>>>> Would appreciate if someone can offer clarity on these questions.
>>>>>> 
>>>>>> Thanks,
>>>>>> Anindya Haldar
>>>>>> Oracle Marketing Cloud
>>>>>> 
>>>>> 
>>>>> 
>>> 
>> 
>>