Following situation in cluster with 10 nodes:
Node A's disk read IO is ~20 times higher than the read load of node B.
The nodes are exactly the same except:
- Node A has 512 tokens and Node B 256. So it has double the load (data).
- Node A also has 2 SSDs, Node B only 1 SSD (according to load)
Node A has roughly 460GB, Node B 260GB total disk usage.
Both nodes have 128GB RAM and 40 cores.
Of course I assumed that Node A does more reads because cache / load ratio is worse but a factor of 20 makes me very sceptic.
Of course Node A has a much higher and less predictable latency due to the wait states.
Has anybody experienced similar situations?
Any hints how to analyze or optimize this - I mean 128GB cache for 460GB payload is not that few. I am pretty sure that not the whole dataset of 460GB is "hot".
Jaumo GmbH · www.jaumo.com
Wehrstraße 46 · 73035 Göppingen · Germany
Phone +49 7161 304880-6 · Fax +49 7161 304880-1
AG Ulm · HRB 731058 · Managing Director: Jens Kammerer