osdir.com


[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: Problem with hints


Hello,

It's hard to say what the best configuration is. 

max_hints_delivery_threads:  I think it's said you should use 2 delivery thread per DC (as a starting point).
max_hint_window_in_ms: depends on your need (and what the cluster can bear of course). Generally, the default of 3 hours is used.
hinted_handoff_throttle_in_kb: this one is a trickier one. You want to reduce it for hints delivery not to be harmful. See the description here: http://cassandra.apache.org/doc/4.0/configuration/cassandra_config_file.html?highlight=max_hints_delivery_threads#hinted-handoff-throttle-in-kb

Notes:
- if the cluster crashed, it might not be directly due to hints, but to the pressure on the remaining nodes, that is too big for nodes to cope with.
- C*3+ (off the top of my head) is storing hints in files rather than using a system table. This should be way more reliable. If you're still using a version of Cassandra storing hints in the system table, you might want to upgrade?

C*heers,
-----------------------
Alain Rodriguez - alain@xxxxxxxxxxxxxxxxx
France / Spain

The Last Pickle - Apache Cassandra Consulting

Le jeu. 1 nov. 2018 à 16:22, Ghazi Naceur <ghazi.naceur.contact@xxxxxxxxx> a écrit :
Hello everyone,

I have a problem with the hints in my cluster.
In fact, my cluster is composed of 5 nodes : 3 nodes in DC1 and 2 nodes in DC2.
The size of the hints folder increases rapidly and it reaches nearly the 90% of the hard drive (50 Gb).
The test scenario is the following : I applied 1000 UPDATE requests per second on the first DC. I wanted to make this test lasts 3 hours. The first node of the second DC stopped, and after few minutes, the second node stopped as well. The hints files started to increase rapidly and consumed a big part of my hard drive.
This test fails after few minutes.
The replication factor is 2 for both DCs and the consistency level is LOCAL_ONE.
Could you please tell me what is the best configuration of the hints properties : hinted_handoff_throttle_in_kb, max_hint_window_in_ms and max_hints_delivery_threads ?

Best regards.