Re: RE: RE: [EXTERNAL] Cluster is unbalanced

Partition key has value as: 

MWY4MmI0MTQtYTk2YS00YmRjLTkxNDMtOWU0MjM1OWU2NzUy other column is blob.

On Tuesday, June 19, 2018, 6:07:59 PM EDT, Joshua Galbraith <jgalbraith@xxxxxxxxxxxx.INVALID> wrote:

> id text PRIMARY KEY

What values are written to this id field? Can you give us some examples or explain the general use case?

On Tue, Jun 19, 2018 at 1:18 PM, learner dba <cassandradba@xxxxxxxxx.invalid> wrote:

Hi Sean,

Here is create table:

CREATE TABLE ks.cf (

    id text PRIMARY KEY,

    accessdata blob

) WITH bloom_filter_fp_chance = 0.01

    AND caching = {'keys': 'ALL', 'rows_per_partition': 'NONE'}

    AND comment = ''

    AND compaction = {'class': 'org.apache.cassandra.db. compaction. SizeTieredCompactionStrategy', 'max_threshold': '32', 'min_threshold': '4'}

    AND compression = {'chunk_length_in_kb': '64', 'class': 'org.apache.cassandra.io. compress.LZ4Compressor'}

    AND crc_check_chance = 1.0

    AND dclocal_read_repair_chance = 0.1

    AND default_time_to_live = 0

    AND gc_grace_seconds = 864000

    AND max_index_interval = 2048

    AND memtable_flush_period_in_ms = 0

    AND min_index_interval = 128

    AND read_repair_chance = 0.0

    AND speculative_retry = '99PERCENTILE';

Nodetool status: 

Datacenter: dc1

=======================

Status=Up/Down

|/ State=Normal/Leaving/Joining/ Moving

--  Address     Load       Tokens       Owns (effective)  Host ID                               Rack

UN  xxxxx   20.66 GiB  256          61.4%             f4f54949-83c9-419b-9a43- cb630b36d8c2  RAC1

UN  xxxxx  65.77 GiB  256          59.3%             3db430ae-45ef-4746-a273- bc1f66ac8981  RAC1

UN  xxxxxx  60.58 GiB  256          58.4%             1f23e869-1823-4b75-8d3e- f9b32acba9a6  RAC1

UN  xxxxx  47.08 GiB  256          57.5%             7aca9a36-823f-4185-be44- c1464a799084  RAC1

UN  xxxxx  51.47 GiB  256          63.4%             18cff010-9b83-4cf8-9dc2- f05ac63df402  RAC1

Datacenter: dc2

========================

Status=Up/Down

|/ State=Normal/Leaving/Joining/ Moving

--  Address     Load       Tokens       Owns (effective)  Host ID                               Rack

UN  xxxx   24.37 GiB  256          59.5%             1b694180-210a-4b75-8f2a- 748f4a5b6a3d  RAC1

UN  xxxxx 30.76 GiB  256          56.7%             597bac04-c57a-4487-8924- 72e171e45514  RAC1

UN  xxxx  10.73 GiB  256          63.9%             6e7e474e-e292-4433-afd4- 372d30e0f3e1  RAC1

UN  xxxxxx 19.77 GiB  256          61.5%             58751418-7b76-40f7-8b8f- a5bf8fe7d9a2  RAC1

UN  xxxxx  10.33 GiB  256          58.4%             6d58d006-2095-449c-8c67- 50e8cbdfe7a7  RAC1

cassandra-rackdc.properties:

dc=dc1

rack=RAC1 --> same in all nodes

cassandra.yaml:
num_tokens: 256

endpoint_snitch: GossipingPropertyFileSnitch

I can see cassandra-topology.properties, I believe it shouldn't be there with GossipPropertyFileSnitch. Can this file be causing any trouble in data distribution.

cat /opt/cassandra/conf/cassandra- topology.properties 

# Licensed to the Apache Software Foundation (ASF) under one

# or more contributor license agreements.  See the NOTICE file

# distributed with this work for additional information

# regarding copyright ownership.  The ASF licenses this file

# to you under the Apache License, Version 2.0 (the

# "License"); you may not use this file except in compliance

# with the License.  You may obtain a copy of the License at

#

#     http://www.apache.org/ licenses/LICENSE-2.0

#

# Unless required by applicable law or agreed to in writing, software

# distributed under the License is distributed on an "AS IS" BASIS,

# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.

# See the License for the specific language governing permissions and

# limitations under the License.

# Cassandra Node IP=Data Center:Rack

192.168.1.100=DC1:RAC1

192.168.2.200=DC2:RAC2

10.0.0.10=DC1:RAC1

10.0.0.11=DC1:RAC1

10.0.0.12=DC1:RAC2

10.20.114.10=DC2:RAC1

10.20.114.11=DC2:RAC1

10.21.119.13=DC3:RAC1

10.21.119.10=DC3:RAC1

10.0.0.13=DC1:RAC2

10.21.119.14=DC3:RAC2

10.20.114.15=DC2:RAC2

# default for unknown nodes

default=DC1:r1

# Native IPv6 is supported, however you must escape the colon in the IPv6 Address

# Also be sure to comment out JVM_OPTS="$JVM_OPTS -Djava.net.preferIPv4Stack= true"

# in cassandra-env.sh

fe80\:0\:0\:0\:202\:b3ff\: fe1e\:8329=DC1:RAC3

On Tuesday, June 19, 2018, 12:51:34 PM EDT, Durity, Sean R <SEAN_R_DURITY@xxxxxxxxxxxxx> wrote:

You are correct that the cluster decides where data goes (based on the hash of the partition key). However, if you choose a “bad” partition key, you may not get good distribution of the data, because the hash is deterministic (it always goes to the same nodes/replicas). For example, if you have a partition key of a datetime, it is possible that there is more data written for a certain time period – thus a larger partition and an imbalance across the cluster. Choosing a “good” partition key is one of the most important decisions for a Cassandra table.

 

Also, I have seen the use of racks in the topology cause an imbalance in the “first” node of the rack.

 

To help you more, we would need the create table statement(s) for your keyspace and the topology of the cluster (like with nodetool status).

 

 

Sean Durity

From: learner dba <cassandradba@xxxxxxxxx. INVALID>
Sent: Tuesday, June 19, 2018 9:50 AM
To: user@xxxxxxxxxxxxxxxxxxxx
Subject: Re: RE: [EXTERNAL] Cluster is unbalanced

 

We do not chose the node where partition will go. I thought it is snitch's role to chose replica nodes. Even the partition size does not vary on our largest column family:

Percentile  SSTables     Write Latency      Read Latency    Partition Size        Cell Count

                              (micros)          (micros)           (bytes)                  

50%             0.00             17.08             61.21              3311                 1

75%             0.00             20.50             88.15              3973                 1

95%             0.00             35.43            105.78              3973                 1

98%             0.00             42.51            126.93              3973                 1

99%             0.00             51.01            126.93              3973                 1

Min             0.00              3.97             17.09                61                

Max             0.00             73.46            126.93             11864                 1

 

We are kinda stuck here to identify, what could be causing this un-balance.

 

On Tuesday, June 19, 2018, 7:15:28 AM EDT, Joshua Galbraith <jgalbraith@xxxxxxxxxxxx. INVALID> wrote:

 

 

>If it was partition key issue, we would see similar number of partition keys across nodes. If we look closely number of keys across nodes vary a lot.

I'm not sure about that, is it possible you're writing more new partitions to some nodes even though each node owns the same number of tokens?

 

On Mon, Jun 18, 2018 at 6:07 PM, learner dba <cassandradba@xxxxxxxxx. invalid> wrote:

Hi Sean,

 

Are you using any rack aware topology? --> we are using gossip file

Are you using any rack aware topology? --> we are using gossip file

 What are your partition keys? --> Partition key is uniq

Is it possible that your partition keys do not divide up as cleanly as you would like across the cluster because the data is not evenly distributed (by partition key)?  --> No, we verified it.

 

If it was partition key issue, we would see similar number of partition keys across nodes. If we look closely number of keys across nodes vary a lot.

 

 

Number of partitions (estimate): 3142552

Number of partitions (estimate): 15625442

Number of partitions (estimate): 15244021

Number of partitions (estimate): 9592992

Number of partitions (estimate): 15839280

 

 

 

 

 

On Monday, June 18, 2018, 5:39:08 PM EDT, Durity, Sean R <SEAN_R_DURITY@xxxxxxxxxxxxx> wrote:

 

 

Are you using any rack aware topology? What are your partition keys? Is it possible that your partition keys do not divide up as cleanly as you would like across the cluster because the data is not evenly distributed (by partition key)?

 

 

Sean Durity

lord of the (C*) rings (Staff Systems Engineer – Cassandra)

MTC 2250

#cassandra - for the latest news and updates

 

From: learner dba <cassandradba@xxxxxxxxx. INVALID>
Sent: Monday, June 18, 2018 2:06 PM
To: User cassandra.apache.org <user@xxxxxxxxxxxxxxxxxxxx>
Subject: [EXTERNAL] Cluster is unbalanced

 

Hi,

 

Data volume varies a lot in our two DC cluster:

 Load       Tokens       Owns  

 20.01 GiB  256          ?     

 65.32 GiB  256          ?     

 60.09 GiB  256          ?     

 46.95 GiB  256          ?     

 50.73 GiB  256          ?     

kaiprodv2

=========

/Leaving/Joining/Moving

 Load       Tokens       Owns  

 25.19 GiB  256          ?     

 30.26 GiB  256          ?     

 9.82 GiB   256          ?     

 20.54 GiB  256          ?     

 9.7 GiB    256          ?     

 

I ran clearsnapshot, garbagecollect and cleanup, but it increased the size on heavier nodes instead of decreasing. Based on nodetool cfstats, I can see partition keys on each node varies a lot:

 

Number of partitions (estimate): 3142552

Number of partitions (estimate): 15625442

Number of partitions (estimate): 15244021

Number of partitions (estimate): 9592992

Number of partitions (estimate): 15839280

 

How can I diagnose this imbalance further?

 

 

--

Joshua Galbraith | Senior Software Engineer | New Relic
C: 907-209-1208 | jgalbraith@ newrelic.com

--

Joshua Galbraith | Senior Software Engineer | New Relic