OSDir


[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: Cluster is unbalanced


CREATE KEYSPACE data WITH replication = {'class': 'NetworkTopologyStrategy', 'dc1': '3', 'dc2': '3'}  AND durable_writes = true;


cqlsh> select * from system_schema.keyspaces ;


 keyspace_name      | durable_writes | replication

--------------------+----------------+------------------------------------------------------------------------------------------------------------

               apim |           True | {'class': 'org.apache.cassandra.locator.NetworkTopologyStrategy', 'iwakaluaint': '3', 'maikaiprodv2': '3'}

        system_auth |           True |                        {'class': 'org.apache.cassandra.locator.SimpleStrategy', 'replication_factor': '1'}

         eventspace |           True | {'class': 'org.apache.cassandra.locator.NetworkTopologyStrategy', 'iwakaluaint': '3', 'maikaiprodv2': '3'}

      system_schema |           True |                                                    {'class': 'org.apache.cassandra.locator.LocalStrategy'}

            metrics |           True | {'class': 'org.apache.cassandra.locator.NetworkTopologyStrategy', 'iwakaluaint': '3', 'maikaiprodv2': '3'}

                sim |           True | {'class': 'org.apache.cassandra.locator.NetworkTopologyStrategy', 'iwakaluaint': '3', 'maikaiprodv2': '3'}

            billing |           True | {'class': 'org.apache.cassandra.locator.NetworkTopologyStrategy', 'iwakaluaint': '1', 'maikaiprodv2': '1'}

 system_distributed |           True | {'class': 'org.apache.cassandra.locator.NetworkTopologyStrategy', 'iwakaluaint': '3', 'maikaiprodv2': '3'}

             system |           True |                                                    {'class': 'org.apache.cassandra.locator.LocalStrategy'}

              audit |           True | {'class': 'org.apache.cassandra.locator.NetworkTopologyStrategy', 'iwakaluaint': '3', 'maikaiprodv2': '3'}

          dakota_ks |           True | {'class': 'org.apache.cassandra.locator.NetworkTopologyStrategy', 'iwakaluaint': '3', 'maikaiprodv2': '3'}

        credentials |           True | {'class': 'org.apache.cassandra.locator.NetworkTopologyStrategy', 'iwakaluaint': '3', 'maikaiprodv2': '3'}

      system_traces |           True | {'class': 'org.apache.cassandra.locator.NetworkTopologyStrategy', 'iwakaluaint': '3', 'maikaiprodv2': '3'}

               data |           True | {'class': 'org.apache.cassandra.locator.NetworkTopologyStrategy', 'iwakaluaint': '3', 'maikaiprodv2': '3'}


Keyspaces with replica 1, are unused keyspaces. Most space is occupied by data and eventspace keyspaces.

On Wednesday, June 20, 2018, 12:24:18 AM EDT, anil patimidi <arpcassandra@xxxxxxxxx> wrote:


what is your keyspace configuration. Do you have all the keyspaces configured for both DCs?

can you run below query from cqlsh and see if the keyspace is configured to use both DCs 

select * from system.schema_keyspaces;   # if your cluster is on 2.1 or less
select * from system_schema.keyspaces    # for 3.0 clusters

- Anil


On Mon, Jun 18, 2018 at 11:06 AM, learner dba <cassandradba@xxxxxxxxx.invalid> wrote:
Hi,

Data volume varies a lot in our two DC cluster:

 Load       Tokens       Owns  

 20.01 GiB  256          ?     

 65.32 GiB  256          ?     

 60.09 GiB  256          ?     

 46.95 GiB  256          ?     

 50.73 GiB  256          ?     

kaiprodv2

=========

/Leaving/Joining/Moving

 Load       Tokens       Owns  

 25.19 GiB  256          ?     

 30.26 GiB  256          ?     

 9.82 GiB   256          ?     

 20.54 GiB  256          ?     

 9.7 GiB    256          ?     


I ran clearsnapshot, garbagecollect and cleanup, but it increased the size on heavier nodes instead of decreasing. Based on nodetool cfstats, I can see partition keys on each node varies a lot:

Number of partitions (estimate): 3142552

Number of partitions (estimate): 15625442

Number of partitions (estimate): 15244021

Number of partitions (estimate): 9592992

Number of partitions (estimate): 15839280

How can I diagnose this imbalance further?