|1s young gcs are horrible and likely cause of some of your bad metrics. How large are your mutations/query results and what gc/heap settings are you using?|
You can use https://github.com/aragozin/jvm-tools to see the threads generating allocation pressure and using the cpu (ttop) and what garbage is being created (hh --dead-young).
Just a shot in the dark, I would guess you have rather large mutations putting pressure on commitlog and heap. G1 with a larger heap might help in that scenario to reduce fragmentation and adjust its eden and survivor regions to the allocation rate better (but give it a bigger reserve space) but theres limits to what can help if you cant change your workload. Without more info on schema etc its hard to tell but maybe that can help give you some ideas on places to look. It could just as likely be repair coordination, wide partition reads, or compactions so need to look more at what within the app is causing the pressure to know if its possible to improve with settings or if the load your application is producing exceeds what your cluster can handle (needs more nodes).