osdir.com


[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: Tracing in cassandra


Query :
SELECT * FROM keysoace.tablenameWHERE user_id = 390797583 LIMIT 5000; 
-Error: ReadTimeout: Error from server: code=1200 [Coordinator node timed out waiting for replica nodes' responses] message="Operation timed out - received only 0 responses." info={'received_responses': 0, 'required_responses': 1, 'consistency': 'ONE'}

e70ac650-cd9e-11e8-8e99-15807bff4dfd | e70bd7c0-cd9e-11e8-8e99-15807bff4dfd |                                                                    Parsing SELECT * FROM keysoace.tablenameWHERE user_id = 390797583 LIMIT 5000; | 10.54.145.32 |           4020 |                   Native-Transport-Requests-3

e70ac650-cd9e-11e8-8e99-15807bff4dfd | e70bfed0-cd9e-11e8-8e99-15807bff4dfd |                                                                                                                               Preparing statement | 10.54.145.32 |           5065 |                   Native-Transport-Requests-3

e70ac650-cd9e-11e8-8e99-15807bff4dfd | e70c25e0-cd9e-11e8-8e99-15807bff4dfd |                                                                                                         Executing single-partition query on roles | 10.54.145.32 |           6171 |                                   ReadStage-2

e70ac650-cd9e-11e8-8e99-15807bff4dfd | e70c4cf0-cd9e-11e8-8e99-15807bff4dfd |                                                                                                                      Acquiring sstable references | 10.54.145.32 |           6362 |                                   ReadStage-2

e70ac650-cd9e-11e8-8e99-15807bff4dfd | e70c4cf1-cd9e-11e8-8e99-15807bff4dfd |                                                                         Skipped 0/2 non-slice-intersecting sstables, included 0 due to tombstones | 10.54.145.32 |           6641 |                                   ReadStage-2

e70ac650-cd9e-11e8-8e99-15807bff4dfd | e70c4cf2-cd9e-11e8-8e99-15807bff4dfd |                                                                                                                     Key cache hit for sstable 346 | 10.54.145.32 |           6955 |                                   ReadStage-2

e70ac650-cd9e-11e8-8e99-15807bff4dfd | e70c4cf3-cd9e-11e8-8e99-15807bff4dfd |                                                                                                          Bloom filter allows skipping sstable 347 | 10.54.145.32 |           7202 |                                   ReadStage-2

e70ac650-cd9e-11e8-8e99-15807bff4dfd | e70c7400-cd9e-11e8-8e99-15807bff4dfd |                                                                                                         Merged data from memtables and 2 sstables | 10.54.145.32 |           7386 |                                   ReadStage-2

e70ac650-cd9e-11e8-8e99-15807bff4dfd | e70c7401-cd9e-11e8-8e99-15807bff4dfd |                                                                                                                 Read 1 live and 0 tombstone cells | 10.54.145.32 |           7519 |                                   ReadStage-2

e70ac650-cd9e-11e8-8e99-15807bff4dfd | e70c7402-cd9e-11e8-8e99-15807bff4dfd |                                                                                                         Executing single-partition query on roles | 10.54.145.32 |           7826 |                                   ReadStage-4

e70ac650-cd9e-11e8-8e99-15807bff4dfd | e70c7403-cd9e-11e8-8e99-15807bff4dfd |                                                                                                                      Acquiring sstable references | 10.54.145.32 |           7924 |                                   ReadStage-4

e70ac650-cd9e-11e8-8e99-15807bff4dfd | e70c7404-cd9e-11e8-8e99-15807bff4dfd |                                                                         Skipped 0/2 non-slice-intersecting sstables, included 0 due to tombstones | 10.54.145.32 |           8060 |                                   ReadStage-4

e70ac650-cd9e-11e8-8e99-15807bff4dfd | e70c7405-cd9e-11e8-8e99-15807bff4dfd |                                                                                                                     Key cache hit for sstable 346 | 10.54.145.32 |           8137 |                                   ReadStage-4

e70ac650-cd9e-11e8-8e99-15807bff4dfd | e70c7406-cd9e-11e8-8e99-15807bff4dfd |                                                                                                          Bloom filter allows skipping sstable 347 | 10.54.145.32 |           8187 |                                   ReadStage-4

e70ac650-cd9e-11e8-8e99-15807bff4dfd | e70c7407-cd9e-11e8-8e99-15807bff4dfd |                                                                                                         Merged data from memtables and 2 sstables | 10.54.145.32 |           8318 |                                   ReadStage-4

e70ac650-cd9e-11e8-8e99-15807bff4dfd | e70c9b10-cd9e-11e8-8e99-15807bff4dfd |                                                                                                                 Read 1 live and 0 tombstone cells | 10.54.145.32 |           8941 |                                   ReadStage-4

e70ac650-cd9e-11e8-8e99-15807bff4dfd | e70cc220-cd9e-11e8-8e99-15807bff4dfd |                                                                                                                              Read-repair DC_LOCAL | 10.54.145.32 |           9468 |                   Native-Transport-Requests-3

e70ac650-cd9e-11e8-8e99-15807bff4dfd | e70cc221-cd9e-11e8-8e99-15807bff4dfd |                                                                                                                   reading data from /10.54.145.31 | 10.54.145.32 |           9754 |                   Native-Transport-Requests-3

e70ac650-cd9e-11e8-8e99-15807bff4dfd | e70cc222-cd9e-11e8-8e99-15807bff4dfd |                                                                                                       Executing single-partition query on session | 10.54.145.32 |          10310 |                                   ReadStage-4

e70ac650-cd9e-11e8-8e99-15807bff4dfd | e70ce930-cd9e-11e8-8e99-15807bff4dfd |                                                                                                                 reading digest from /10.54.101.56 | 10.54.145.32 |          10343 |                   Native-Transport-Requests-3

e70ac650-cd9e-11e8-8e99-15807bff4dfd | e70ce931-cd9e-11e8-8e99-15807bff4dfd |                                                                                                                      Acquiring sstable references | 10.54.145.32 |          10356 |                                   ReadStage-4

e70ac650-cd9e-11e8-8e99-15807bff4dfd | e70ce932-cd9e-11e8-8e99-15807bff4dfd |                                                                                                                   Key cache hit for sstable 68642 | 10.54.145.32 |          10445 |                                   ReadStage-4

e70ac650-cd9e-11e8-8e99-15807bff4dfd | e70ce933-cd9e-11e8-8e99-15807bff4dfd |                                                                                                                 reading digest from /10.54.101.57 | 10.54.145.32 |          10480 |                   Native-Transport-Requests-3

e70ac650-cd9e-11e8-8e99-15807bff4dfd | e70ce934-cd9e-11e8-8e99-15807bff4dfd |                                                                                                        Bloom filter allows skipping sstable 68397 | 10.54.145.32 |          10496 |                                   ReadStage-4

e70ac650-cd9e-11e8-8e99-15807bff4dfd | e70ce935-cd9e-11e8-8e99-15807bff4dfd |                                                                         Skipped 0/3 non-slice-intersecting sstables, included 0 due to tombstones | 10.54.145.32 |          10547 |                                   ReadStage-4

e70ac650-cd9e-11e8-8e99-15807bff4dfd | e70ce936-cd9e-11e8-8e99-15807bff4dfd |                                                                                                             Sending READ message to /10.54.145.31 | 10.54.145.32 |          10575 | MessagingService-Outgoing-/10.54.145.31-Small

e70ac650-cd9e-11e8-8e99-15807bff4dfd | e70ce937-cd9e-11e8-8e99-15807bff4dfd |                                                                                                                   Key cache hit for sstable 68655 | 10.54.145.32 |          10631 |                                   ReadStage-4

e70ac650-cd9e-11e8-8e99-15807bff4dfd | e70ce938-cd9e-11e8-8e99-15807bff4dfd |                                                                                                             Sending READ message to /10.54.101.56 | 10.54.145.32 |          10695 | MessagingService-Outgoing-/10.54.101.56-Small

e70ac650-cd9e-11e8-8e99-15807bff4dfd | e70ce939-cd9e-11e8-8e99-15807bff4dfd |                                                                                                             Sending READ message to /10.54.101.57 | 10.54.145.32 |          10823 | MessagingService-Outgoing-/10.54.101.57-Small

e70ac650-cd9e-11e8-8e99-15807bff4dfd | e70ce93a-cd9e-11e8-8e99-15807bff4dfd |                                                                                                         Merged data from memtables and 3 sstables | 10.54.145.32 |          10863 |                                   ReadStage-4

e70ac650-cd9e-11e8-8e99-15807bff4dfd | e70ce93b-cd9e-11e8-8e99-15807bff4dfd |                                                                                                                Read 13 live and 0 tombstone cells | 10.54.145.32 |          10914 |                                   ReadStage-4

e70ac650-cd9e-11e8-8e99-15807bff4dfd | e70dac80-cd9e-11e8-9ad7-9f2cb164b2ba |                                                                                                          READ message received from /10.54.145.32 | 10.54.145.31 |            883 |       MessagingService-Incoming-/10.54.145.32

e70ac650-cd9e-11e8-8e99-15807bff4dfd | e70dac80-cd9e-11e8-ba8b-bbf19da89f90 |                                                                                                          READ message received from /10.54.145.32 | 10.54.101.57 |            803 |       MessagingService-Incoming-/10.54.145.32

e70ac650-cd9e-11e8-8e99-15807bff4dfd | e70dd390-cd9e-11e8-bed7-f13ab36bfa4c |                                                                                                          READ message received from /10.54.145.32 | 10.54.101.56 |            961 |       MessagingService-Incoming-/10.54.145.32

e70ac650-cd9e-11e8-8e99-15807bff4dfd | e70dfaa0-cd9e-11e8-9ad7-9f2cb164b2ba |                                                                                                       Executing single-partition query on session | 10.54.145.31 |           3208 |                                   ReadStage-3

e70ac650-cd9e-11e8-8e99-15807bff4dfd | e70dfaa0-cd9e-11e8-ba8b-bbf19da89f90 |                                                                                                       Executing single-partition query on session | 10.54.101.57 |           3258 |                                   ReadStage-3

e70ac650-cd9e-11e8-8e99-15807bff4dfd | e70dfaa1-cd9e-11e8-9ad7-9f2cb164b2ba |                                                                                                                      Acquiring sstable references | 10.54.145.31 |           3483 |                                   ReadStage-3

e70ac650-cd9e-11e8-8e99-15807bff4dfd | e70dfaa1-cd9e-11e8-ba8b-bbf19da89f90 |                                                                                                                      Acquiring sstable references | 10.54.101.57 |           3549 |                                   ReadStage-3

e70ac650-cd9e-11e8-8e99-15807bff4dfd | e70e21b0-cd9e-11e8-9ad7-9f2cb164b2ba |                                                                                                                   Key cache hit for sstable 74713 | 10.54.145.31 |           3732 |                                   ReadStage-3

e70ac650-cd9e-11e8-8e99-15807bff4dfd | e70e21b0-cd9e-11e8-ba8b-bbf19da89f90 |                                                                                                                   Key cache hit for sstable 85409 | 10.54.101.57 |           3978 |                                   ReadStage-3

e70ac650-cd9e-11e8-8e99-15807bff4dfd | e70e21b0-cd9e-11e8-bed7-f13ab36bfa4c |                                                                                                       Executing single-partition query on session | 10.54.101.56 |           3497 |                                   ReadStage-3

e70ac650-cd9e-11e8-8e99-15807bff4dfd | e70e21b1-cd9e-11e8-9ad7-9f2cb164b2ba |                                                                                                                   Key cache hit for sstable 74445 | 10.54.145.31 |           3893 |                                   ReadStage-3

e70ac650-cd9e-11e8-8e99-15807bff4dfd | e70e21b1-cd9e-11e8-ba8b-bbf19da89f90 |                                                                                                        Bloom filter allows skipping sstable 85201 | 10.54.101.57 |           4193 |                                   ReadStage-3

e70ac650-cd9e-11e8-8e99-15807bff4dfd | e70e21b2-cd9e-11e8-9ad7-9f2cb164b2ba |                                                                                                        Bloom filter allows skipping sstable 74383 | 10.54.145.31 |           3975 |                                   ReadStage-3

e70ac650-cd9e-11e8-8e99-15807bff4dfd | e70e21b2-cd9e-11e8-ba8b-bbf19da89f90 |                                                                         Skipped 0/3 non-slice-intersecting sstables, included 0 due to tombstones | 10.54.101.57 |           4354 |                                   ReadStage-3

e70ac650-cd9e-11e8-8e99-15807bff4dfd | e70e21b3-cd9e-11e8-9ad7-9f2cb164b2ba |                                                                         Skipped 0/3 non-slice-intersecting sstables, included 0 due to tombstones | 10.54.145.31 |           4142 |                                   ReadStage-3

e70ac650-cd9e-11e8-8e99-15807bff4dfd | e70e21b3-cd9e-11e8-ba8b-bbf19da89f90 |                                                                                                                   Key cache hit for sstable 85418 | 10.54.101.57 |           4512 |                                   ReadStage-3

e70ac650-cd9e-11e8-8e99-15807bff4dfd | e70e48c0-cd9e-11e8-ba8b-bbf19da89f90 |                                                                                                         Merged data from memtables and 3 sstables | 10.54.101.57 |           4989 |                                   ReadStage-3

e70ac650-cd9e-11e8-8e99-15807bff4dfd | e70e48c0-cd9e-11e8-bed7-f13ab36bfa4c |                                                                                                                      Acquiring sstable references | 10.54.101.56 |           3751 |                                   ReadStage-3

e70ac650-cd9e-11e8-8e99-15807bff4dfd | e70e48c1-cd9e-11e8-ba8b-bbf19da89f90 |                                                                                                                Read 13 live and 0 tombstone cells | 10.54.101.57 |           5336 |                                   ReadStage-3

e70ac650-cd9e-11e8-8e99-15807bff4dfd | e70e48c1-cd9e-11e8-bed7-f13ab36bfa4c |                                                                                                                  Key cache hit for sstable 101147 | 10.54.101.56 |           4091 |                                   ReadStage-3

e70ac650-cd9e-11e8-8e99-15807bff4dfd | e70e48c2-cd9e-11e8-ba8b-bbf19da89f90 |                                                                                                                Read 13 live and 0 tombstone cells | 10.54.101.57 |           5586 |                                   ReadStage-3

e70ac650-cd9e-11e8-8e99-15807bff4dfd | e70e6fd0-cd9e-11e8-ba8b-bbf19da89f90 |                                                                                                               Enqueuing response to /10.54.145.32 | 10.54.101.57 |           5745 |                                   ReadStage-3

e70ac650-cd9e-11e8-8e99-15807bff4dfd | e70e6fd1-cd9e-11e8-ba8b-bbf19da89f90 |                                                                                                 Sending REQUEST_RESPONSE message to /10.54.145.32 | 10.54.101.57 |           6473 | MessagingService-Outgoing-/10.54.145.32-Small

e70ac650-cd9e-11e8-8e99-15807bff4dfd | e70e96e0-cd9e-11e8-8e99-15807bff4dfd |                                                                                              REQUEST_RESPONSE message received from /10.54.101.57 | 10.54.145.32 |             57 |       MessagingService-Incoming-/10.54.101.57

e70ac650-cd9e-11e8-8e99-15807bff4dfd | e70e96e0-cd9e-11e8-9ad7-9f2cb164b2ba |                                                                                                         Merged data from memtables and 3 sstables | 10.54.145.31 |           6600 |                                   ReadStage-3

e70ac650-cd9e-11e8-8e99-15807bff4dfd | e70e96e0-cd9e-11e8-bed7-f13ab36bfa4c |                                                                                                                  Key cache hit for sstable 100896 | 10.54.101.56 |           6496 |                                   ReadStage-3

e70ac650-cd9e-11e8-8e99-15807bff4dfd | e70e96e1-cd9e-11e8-8e99-15807bff4dfd |                                                                                                            Processing response from /10.54.101.57 | 10.54.145.32 |            587 |                        RequestResponseStage-1

e70ac650-cd9e-11e8-8e99-15807bff4dfd | e70e96e1-cd9e-11e8-9ad7-9f2cb164b2ba |                                                                                                                Read 13 live and 0 tombstone cells | 10.54.145.31 |           6892 |                                   ReadStage-3

e70ac650-cd9e-11e8-8e99-15807bff4dfd | e70e96e2-cd9e-11e8-9ad7-9f2cb164b2ba |                                                                                                               Enqueuing response to /10.54.145.32 | 10.54.145.31 |           7038 |                                   ReadStage-3

e70ac650-cd9e-11e8-8e99-15807bff4dfd | e70ebdf0-cd9e-11e8-8e99-15807bff4dfd |                                                                                              REQUEST_RESPONSE message received from /10.54.145.31 | 10.54.145.32 |             15 |       MessagingService-Incoming-/10.54.145.31

e70ac650-cd9e-11e8-8e99-15807bff4dfd | e70ebdf0-cd9e-11e8-9ad7-9f2cb164b2ba |                                                                                                 Sending REQUEST_RESPONSE message to /10.54.145.32 | 10.54.145.31 |           7641 | MessagingService-Outgoing-/10.54.145.32-Small

e70ac650-cd9e-11e8-8e99-15807bff4dfd | e70ebdf0-cd9e-11e8-bed7-f13ab36bfa4c |                                                                                                       Bloom filter allows skipping sstable 100887 | 10.54.101.56 |           6652 |                                   ReadStage-3

e70ac650-cd9e-11e8-8e99-15807bff4dfd | e70ebdf1-cd9e-11e8-8e99-15807bff4dfd |                                                                                                            Processing response from /10.54.145.31 | 10.54.145.32 |             95 |                        RequestResponseStage-1

e70ac650-cd9e-11e8-8e99-15807bff4dfd | e70ebdf1-cd9e-11e8-bed7-f13ab36bfa4c |                                                                         Skipped 0/4 non-slice-intersecting sstables, included 0 due to tombstones | 10.54.101.56 |           6828 |                                   ReadStage-3

e70ac650-cd9e-11e8-8e99-15807bff4dfd | e70ebdf2-cd9e-11e8-bed7-f13ab36bfa4c |                                                                                                                  Key cache hit for sstable 101149 | 10.54.101.56 |           7005 |                                   ReadStage-3

e70ac650-cd9e-11e8-8e99-15807bff4dfd | e70f3320-cd9e-11e8-bed7-f13ab36bfa4c |                                                                                                         Merged data from memtables and 4 sstables | 10.54.101.56 |           9897 |                                   ReadStage-3

e70ac650-cd9e-11e8-8e99-15807bff4dfd | e70f8140-cd9e-11e8-bed7-f13ab36bfa4c |                                                                                                                Read 13 live and 0 tombstone cells | 10.54.101.56 |          11969 |                                   ReadStage-3

e70ac650-cd9e-11e8-8e99-15807bff4dfd | e70f8141-cd9e-11e8-bed7-f13ab36bfa4c |                                                                                                                Read 13 live and 0 tombstone cells | 10.54.101.56 |          12067 |                                   ReadStage-3

e70ac650-cd9e-11e8-8e99-15807bff4dfd | e70f8142-cd9e-11e8-bed7-f13ab36bfa4c |                                                                                                               Enqueuing response to /10.54.145.32 | 10.54.101.56 |          12361 |                                   ReadStage-3

e70ac650-cd9e-11e8-8e99-15807bff4dfd | e70fcf60-cd9e-11e8-bed7-f13ab36bfa4c |                                                                                                 Sending REQUEST_RESPONSE message to /10.54.145.32 | 10.54.101.56 |          14161 | MessagingService-Outgoing-/10.54.145.32-Small

e70ac650-cd9e-11e8-8e99-15807bff4dfd | e70ff670-cd9e-11e8-8e99-15807bff4dfd |                                                                                              REQUEST_RESPONSE message received from /10.54.101.56 | 10.54.145.32 |             51 |       MessagingService-Incoming-/10.54.101.56

e70ac650-cd9e-11e8-8e99-15807bff4dfd | e70ff671-cd9e-11e8-8e99-15807bff4dfd |                                                                                                            Processing response from /10.54.101.56 | 10.54.145.32 |            206 |                        RequestResponseStage-7

f65d32f0-cd9e-11e8-8e99-15807bff4dfd | f65d32f1-cd9e-11e8-8e99-15807bff4dfd | Parsing SELECT * FROM keysoace.tablenameWHERE status = 0 AND token(user_id) >= token(769281382) AND token(user_id) <= token(769281382) LIMIT 12; | 10.54.145.32 |            519 |                   Native-Transport-Requests-2

f65d32f0-cd9e-11e8-8e99-15807bff4dfd | f65d5a00-cd9e-11e8-8e99-15807bff4dfd |                                                                                                                               Preparing statement | 10.54.145.32 |            906 |                   Native-Transport-Requests-2

f65d32f0-cd9e-11e8-8e99-15807bff4dfd | f65d8110-cd9e-11e8-8e99-15807bff4dfd |                                                                                                         Executing single-partition query on roles | 10.54.145.32 |           1820 |                                   ReadStage-1

f65d32f0-cd9e-11e8-8e99-15807bff4dfd | f65d8111-cd9e-11e8-8e99-15807bff4dfd |                                                                                                                      Acquiring sstable references | 10.54.145.32 |           1884 |                                   ReadStage-1

f65d32f0-cd9e-11e8-8e99-15807bff4dfd | f65d8112-cd9e-11e8-8e99-15807bff4dfd |                                                                         Skipped 0/2 non-slice-intersecting sstables, included 0 due to tombstones | 10.54.145.32 |           1968 |                                   ReadStage-1

f65d32f0-cd9e-11e8-8e99-15807bff4dfd | f65d8113-cd9e-11e8-8e99-15807bff4dfd |                                                                                                                     Key cache hit for sstable 346 | 10.54.145.32 |           2023 |                                   ReadStage-1

f65d32f0-cd9e-11e8-8e99-15807bff4dfd | f65d8114-cd9e-11e8-8e99-15807bff4dfd |                                                                                                          Bloom filter allows skipping sstable 347 | 10.54.145.32 |           2076 |                                   ReadStage-1

f65d32f0-cd9e-11e8-8e99-15807bff4dfd | f65d8115-cd9e-11e8-8e99-15807bff4dfd |                                                                                                         Merged data from memtables and 2 sstables | 10.54.145.32 |           2133 |                                   ReadStage-1

f65d32f0-cd9e-11e8-8e99-15807bff4dfd | f65d8116-cd9e-11e8-8e99-15807bff4dfd |                                                                                                                 Read 1 live and 0 tombstone cells | 10.54.145.32 |           2179 |                                   ReadStage-1

f65d32f0-cd9e-11e8-8e99-15807bff4dfd | f65d8117-cd9e-11e8-8e99-15807bff4dfd |                                                                                                         Executing single-partition query on roles | 10.54.145.32 |           2290 |                                   ReadStage-3

f65d32f0-cd9e-11e8-8e99-15807bff4dfd | f65d8118-cd9e-11e8-8e99-15807bff4dfd |                                                                                                                      Acquiring sstable references | 10.54.145.32 |           2371 |                                   ReadStage-3

f65d32f0-cd9e-11e8-8e99-15807bff4dfd | f65d8119-cd9e-11e8-8e99-15807bff4dfd |                                                                         Skipped 0/2 non-slice-intersecting sstables, included 0 due to tombstones | 10.54.145.32 |           2494 |                                   ReadStage-3

f65d32f0-cd9e-11e8-8e99-15807bff4dfd | f65da820-cd9e-11e8-8e99-15807bff4dfd |                                                                                                                     Key cache hit for sstable 346 | 10.54.145.32 |           2560 |                                   ReadStage-3

f65d32f0-cd9e-11e8-8e99-15807bff4dfd | f65da821-cd9e-11e8-8e99-15807bff4dfd |                                                                                                          Bloom filter allows skipping sstable 347 | 10.54.145.32 |           2618 |                                   ReadStage-3

f65d32f0-cd9e-11e8-8e99-15807bff4dfd | f65da822-cd9e-11e8-8e99-15807bff4dfd |                                                                                                         Merged data from memtables and 2 sstables | 10.54.145.32 |           2689 |                                   ReadStage-3

f65d32f0-cd9e-11e8-8e99-15807bff4dfd | f65da823-cd9e-11e8-8e99-15807bff4dfd |                                                                                                                 Read 1 live and 0 tombstone cells | 10.54.145.32 |           2750 |                                   ReadStage-3

f65d32f0-cd9e-11e8-8e99-15807bff4dfd | f65df640-cd9e-11e8-8e99-15807bff4dfd |                                                                              Index mean cardinalities are status_idx:1. Scanning with status_idx. | 10.54.145.32 |           4656 |                   Native-Transport-Requests-2

f65d32f0-cd9e-11e8-8e99-15807bff4dfd | f65df641-cd9e-11e8-8e99-15807bff4dfd |                                                                                                                         Computing ranges to query | 10.54.145.32 |           4844 |                   Native-Transport-Requests-2

f65d32f0-cd9e-11e8-8e99-15807bff4dfd | f65df642-cd9e-11e8-8e99-15807bff4dfd |                                              Submitting range requests on 1 ranges with a concurrency of 1 (2.9296876E-4 rows per range expected) | 10.54.145.32 |           5109 |                   Native-Transport-Requests-2

f65d32f0-cd9e-11e8-8e99-15807bff4dfd | f65df643-cd9e-11e8-8e99-15807bff4dfd |                                                                                                             Submitted 1 concurrent range requests | 10.54.145.32 |           5286 |                   Native-Transport-Requests-2

f65d32f0-cd9e-11e8-8e99-15807bff4dfd | f65df644-cd9e-11e8-8e99-15807bff4dfd |                                                                                       Executing read on keysoace.tablenameusing index status_idx | 10.54.145.32 |           5386 |                                   ReadStage-6

f65d32f0-cd9e-11e8-8e99-15807bff4dfd | f65df645-cd9e-11e8-8e99-15807bff4dfd |                                                                                            Executing single-partition query on session.status_idx | 10.54.145.32 |           5476 |                                   ReadStage-6

f65d32f0-cd9e-11e8-8e99-15807bff4dfd | f65df646-cd9e-11e8-8e99-15807bff4dfd |                                                                                                                      Acquiring sstable references | 10.54.145.32 |           5499 |                                   ReadStage-6

f65d32f0-cd9e-11e8-8e99-15807bff4dfd | f65df647-cd9e-11e8-8e99-15807bff4dfd |                                                                                                                   Key cache hit for sstable 42279 | 10.54.145.32 |           5539 |                                   ReadStage-6

f65d32f0-cd9e-11e8-8e99-15807bff4dfd | f65e1d50-cd9e-11e8-8e99-15807bff4dfd |                                                                                                                   Key cache hit for sstable 42267 | 10.54.145.32 |           5574 |                                   ReadStage-6

f65d32f0-cd9e-11e8-8e99-15807bff4dfd | f65e1d51-cd9e-11e8-8e99-15807bff4dfd |                                                                                                                   Key cache hit for sstable 41890 | 10.54.145.32 |           5601 |                                   ReadStage-6

f65d32f0-cd9e-11e8-8e99-15807bff4dfd | f65e1d52-cd9e-11e8-8e99-15807bff4dfd |                                                                                                                   Key cache hit for sstable 37159 | 10.54.145.32 |           5704 |                                   ReadStage-6

 

---MORE---

session_id                           | event_id                             | activity                                                                                                                                            | source       | source_elapsed | thread

--------------------------------------+--------------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------+--------------+----------------+-----------------------------------------------

f65d32f0-cd9e-11e8-8e99-15807bff4dfd | f65e1d53-cd9e-11e8-8e99-15807bff4dfd |                                                                           Skipped 0/4 non-slice-intersecting sstables, included 0 due to tombstones | 10.54.145.32 |           5800 |                                   ReadStage-6

f65d32f0-cd9e-11e8-8e99-15807bff4dfd | fc533b00-cd9e-11e8-8e99-15807bff4dfd |                                                                                                                Timed out; received 0 of 1 responses | 10.54.145.32 |       10000610 |                   Native-Transport-Requests-2

76e98e60-cd6c-11e8-af38-373d9c5cfc0c | 76eac6e0-cd6c-11e8-af38-373d9c5cfc0c | Parsing SELECT * FROM keysoace.tablenameWHERE status = 0 AND token(user_id) >= token(371384583) AND token(user_id) <= token(371384583) LIMIT 11\n; | 10.153.42.83 |           3819 |                   Native-Transport-Requests-3

76e98e60-cd6c-11e8-af38-373d9c5cfc0c | 76eaedf0-cd6c-11e8-af38-373d9c5cfc0c |                                                                                                                                 Preparing statement | 10.153.42.83 |           5486 |                   Native-Transport-Requests-3

76e98e60-cd6c-11e8-af38-373d9c5cfc0c | 76ebff60-cd6c-11e8-af38-373d9c5cfc0c |                                                                                                                     reading data from /10.153.42.84 | 10.153.42.83 |          12093 |                   Native-Transport-Requests-3

76e98e60-cd6c-11e8-af38-373d9c5cfc0c | 76ec4d80-cd6c-11e8-af38-373d9c5cfc0c |                                                                                                               Sending READ message to /10.153.42.84 | 10.153.42.83 |          13801 | MessagingService-Outgoing-/10.153.42.84-Small

76e98e60-cd6c-11e8-af38-373d9c5cfc0c | 76edad10-cd6c-11e8-a6c8-79961efb2cb8 |                                                                                                            READ message received from /10.153.42.83 | 10.153.42.84 |            928 |       MessagingService-Incoming-/10.153.42.83

76e98e60-cd6c-11e8-af38-373d9c5cfc0c | 76ee7060-cd6c-11e8-a6c8-79961efb2cb8 |                                                                                                           Executing single-partition query on roles | 10.153.42.84 |           5919 |                                   ReadStage-7

76e98e60-cd6c-11e8-af38-373d9c5cfc0c | 76ee7061-cd6c-11e8-a6c8-79961efb2cb8 |                                                                                                                        Acquiring sstable references | 10.153.42.84 |           6327 |                                   ReadStage-7

76e98e60-cd6c-11e8-af38-373d9c5cfc0c | 76ee9770-cd6c-11e8-a6c8-79961efb2cb8 |                                                                           Skipped 0/2 non-slice-intersecting sstables, included 0 due to tombstones | 10.153.42.84 |           6802 |                                   ReadStage-7

76e98e60-cd6c-11e8-af38-373d9c5cfc0c | 76eee590-cd6c-11e8-a6c8-79961efb2cb8 |                                                                                                                       Key cache hit for sstable 477 | 10.153.42.84 |           8557 |                                   ReadStage-7

76e98e60-cd6c-11e8-af38-373d9c5cfc0c | 76eee591-cd6c-11e8-a6c8-79961efb2cb8 |                                                                                                            Bloom filter allows skipping sstable 476 | 10.153.42.84 |           8836 |                                   ReadStage-7

76e98e60-cd6c-11e8-af38-373d9c5cfc0c | 76eee592-cd6c-11e8-a6c8-79961efb2cb8 |                                                                                                           Merged data from memtables and 2 sstables | 10.153.42.84 |           9252 |                                   ReadStage-7

76e98e60-cd6c-11e8-af38-373d9c5cfc0c | 76ef0ca0-cd6c-11e8-a6c8-79961efb2cb8 |                                                                                                                   Read 1 live and 0 tombstone cells | 10.153.42.84 |           9428 |                                   ReadStage-7

76e98e60-cd6c-11e8-af38-373d9c5cfc0c | 76ef0ca1-cd6c-11e8-a6c8-79961efb2cb8 |                                                                                                                 Enqueuing response to /10.153.42.83 | 10.153.42.84 |           9554 |                                   ReadStage-7

76e98e60-cd6c-11e8-af38-373d9c5cfc0c | 76ef0ca2-cd6c-11e8-a6c8-79961efb2cb8 |                                                                                                   Sending REQUEST_RESPONSE message to /10.153.42.83 | 10.153.42.84 |          10130 | MessagingService-Outgoing-/10.153.42.83-Small

76e98e60-cd6c-11e8-af38-373d9c5cfc0c | 76ef33b0-cd6c-11e8-af38-373d9c5cfc0c |                                                                                                REQUEST_RESPONSE message received from /10.153.42.84 | 10.153.42.83 |          32954 |       MessagingService-Incoming-/10.153.42.84

76e98e60-cd6c-11e8-af38-373d9c5cfc0c | 76ef5ac0-cd6c-11e8-af38-373d9c5cfc0c |                                                                                                              Processing response from /10.153.42.84 | 10.153.42.83 |          34053 |                        RequestResponseStage-2

76e98e60-cd6c-11e8-af38-373d9c5cfc0c | 76ef5ac1-cd6c-11e8-af38-373d9c5cfc0c |                                                                                                                     reading data from /10.153.42.84 | 10.153.42.83 |          34499 |                   Native-Transport-Requests-3

76e98e60-cd6c-11e8-af38-373d9c5cfc0c | 76ef5ac2-cd6c-11e8-af38-373d9c5cfc0c |                                                                                                               Sending READ message to /10.153.42.84 | 10.153.42.83 |          34775 | MessagingService-Outgoing-/10.153.42.84-Small

76e98e60-cd6c-11e8-af38-373d9c5cfc0c | 76ef81d0-cd6c-11e8-a6c8-79961efb2cb8 |                                                                                                            READ message received from /10.153.42.83 | 10.153.42.84 |             14 |       MessagingService-Incoming-/10.153.42.83

76e98e60-cd6c-11e8-af38-373d9c5cfc0c | 76ef81d1-cd6c-11e8-a6c8-79961efb2cb8 |                                                                                                           Executing single-partition query on roles | 10.153.42.84 |            399 |                                   ReadStage-4

76e98e60-cd6c-11e8-af38-373d9c5cfc0c | 76ef81d2-cd6c-11e8-a6c8-79961efb2cb8 |                                                                                                                        Acquiring sstable references | 10.153.42.84 |            521 |                                   ReadStage-4

76e98e60-cd6c-11e8-af38-373d9c5cfc0c | 76ef81d3-cd6c-11e8-a6c8-79961efb2cb8 |                                                                           Skipped 0/2 non-slice-intersecting sstables, included 0 due to tombstones | 10.153.42.84 |            607 |                                   ReadStage-4

76e98e60-cd6c-11e8-af38-373d9c5cfc0c | 76ef81d4-cd6c-11e8-a6c8-79961efb2cb8 |                                                                                                                       Key cache hit for sstable 477 | 10.153.42.84 |            736 |                                   ReadStage-4

76e98e60-cd6c-11e8-af38-373d9c5cfc0c | 76efa8e0-cd6c-11e8-a6c8-79961efb2cb8 |                                                                                                            Bloom filter allows skipping sstable 476 | 10.153.42.84 |            810 |                                   ReadStage-4

76e98e60-cd6c-11e8-af38-373d9c5cfc0c | 76efa8e0-cd6c-11e8-af38-373d9c5cfc0c |                                                                                                REQUEST_RESPONSE message received from /10.153.42.84 | 10.153.42.83 |          36504 |       MessagingService-Incoming-/10.153.42.84

76e98e60-cd6c-11e8-af38-373d9c5cfc0c | 76efa8e1-cd6c-11e8-a6c8-79961efb2cb8 |                                                                                                           Merged data from memtables and 2 sstables | 10.153.42.84 |           1021 |                                   ReadStage-4

76e98e60-cd6c-11e8-af38-373d9c5cfc0c | 76efa8e1-cd6c-11e8-af38-373d9c5cfc0c |                                                                                                              Processing response from /10.153.42.84 | 10.153.42.83 |          36729 |                        RequestResponseStage-5

76e98e60-cd6c-11e8-af38-373d9c5cfc0c | 76efa8e2-cd6c-11e8-a6c8-79961efb2cb8 |                                                                                                                   Read 1 live and 0 tombstone cells | 10.153.42.84 |           1132 |                                   ReadStage-4

76e98e60-cd6c-11e8-af38-373d9c5cfc0c | 76efa8e3-cd6c-11e8-a6c8-79961efb2cb8 |                                                                                                                 Enqueuing response to /10.153.42.83 | 10.153.42.84 |           1180 |                                   ReadStage-4

76e98e60-cd6c-11e8-af38-373d9c5cfc0c | 76efa8e4-cd6c-11e8-a6c8-79961efb2cb8 |                                                                                                   Sending REQUEST_RESPONSE message to /10.153.42.83 | 10.153.42.84 |           1279 | MessagingService-Outgoing-/10.153.42.83-Small

76e98e60-cd6c-11e8-af38-373d9c5cfc0c | 76f0ba50-cd6c-11e8-af38-373d9c5cfc0c |                                                                                Index mean cardinalities are status_idx:1. Scanning with status_idx. | 10.153.42.83 |          43356 |                   Native-Transport-Requests-3

76e98e60-cd6c-11e8-af38-373d9c5cfc0c | 76f0ba51-cd6c-11e8-af38-373d9c5cfc0c |                                                                                                                           Computing ranges to query | 10.153.42.83 |          43791 |                   Native-Transport-Requests-3

76e98e60-cd6c-11e8-af38-373d9c5cfc0c | 76f0e160-cd6c-11e8-af38-373d9c5cfc0c |                                                Submitting range requests on 1 ranges with a concurrency of 1 (2.9296876E-4 rows per range expected) | 10.153.42.83 |          44274 |                   Native-Transport-Requests-3

76e98e60-cd6c-11e8-af38-373d9c5cfc0c | 76f0e161-cd6c-11e8-af38-373d9c5cfc0c |                                                                                                                  Enqueuing request to /10.153.42.85 | 10.153.42.83 |          44631 |                   Native-Transport-Requests-3

76e98e60-cd6c-11e8-af38-373d9c5cfc0c | 76f10870-cd6c-11e8-af38-373d9c5cfc0c |                                                                                                               Submitted 1 concurrent range requests | 10.153.42.83 |          45063 |                   Native-Transport-Requests-3

76e98e60-cd6c-11e8-af38-373d9c5cfc0c | 76f10871-cd6c-11e8-af38-373d9c5cfc0c |                                                                                                        Sending RANGE_SLICE message to /10.153.42.85 | 10.153.42.83 |          45074 | MessagingService-Outgoing-/10.153.42.85-Small

76e98e60-cd6c-11e8-af38-373d9c5cfc0c | 76f1cbc0-cd6c-11e8-9866-ade6e01c2040 |                                                                                                     RANGE_SLICE message received from /10.153.42.83 | 10.153.42.85 |            605 |       MessagingService-Incoming-/10.153.42.83

76e98e60-cd6c-11e8-af38-373d9c5cfc0c | 76f1f2d0-cd6c-11e8-9866-ade6e01c2040 |                                                                                         Executing read on keysoace.tablenameusing index status_idx | 10.153.42.85 |           2375 |                                   ReadStage-2

76e98e60-cd6c-11e8-af38-373d9c5cfc0c | 76f1f2d1-cd6c-11e8-9866-ade6e01c2040 |                                                                                              Executing single-partition query on session.status_idx | 10.153.42.85 |           2599 |                                   ReadStage-2

76e98e60-cd6c-11e8-af38-373d9c5cfc0c | 76f1f2d2-cd6c-11e8-9866-ade6e01c2040 |                                                                                                                        Acquiring sstable references | 10.153.42.85 |           2709 |                                   ReadStage-2

76e98e60-cd6c-11e8-af38-373d9c5cfc0c | 76f219e0-cd6c-11e8-9866-ade6e01c2040 |                                                                                             Partition index with 456 entries found for sstable 1386 | 10.153.42.85 |           2954 |                                   ReadStage-2

76e98e60-cd6c-11e8-af38-373d9c5cfc0c | 76f219e1-cd6c-11e8-9866-ade6e01c2040 |                                                                                           Partition index with 11352 entries found for sstable 1367 | 10.153.42.85 |           3240 |                                   ReadStage-2

76e98e60-cd6c-11e8-af38-373d9c5cfc0c | 76f219e2-cd6c-11e8-9866-ade6e01c2040 |                                                                                            Partition index with 16589 entries found for sstable 838 | 10.153.42.85 |           3413 |                                   ReadStage-2

76e98e60-cd6c-11e8-af38-373d9c5cfc0c | 76f219e3-cd6c-11e8-9866-ade6e01c2040 |                                                                           Skipped 0/3 non-slice-intersecting sstables, included 0 due to tombstones | 10.153.42.85 |           3599 |                                   ReadStage-2

76e98e60-cd6c-11e8-af38-373d9c5cfc0c | 7cdf4850-cd6c-11e8-af38-373d9c5cfc0c |                                                                                                                Timed out; received 0 of 1 responses | 10.153.42.83 |        9995766 |                   Native-Transport-Requests-3

76e98e60-cd6c-11e8-af38-373d9c5cfc0c | 923a5aa0-cd6c-11e8-9866-ade6e01c2040 |                                                                                                         Executing single-partition query on session | 10.153.42.85 |       45774985 |                                   ReadStage-2

76e98e60-cd6c-11e8-af38-373d9c5cfc0c | 923a5aa1-cd6c-11e8-9866-ade6e01c2040 |                                                                                                                        Acquiring sstable references | 10.153.42.85 |       45775171 |                                   ReadStage-2

76e98e60-cd6c-11e8-af38-373d9c5cfc0c | 923a5aa2-cd6c-11e8-9866-ade6e01c2040 |                                                                                                                           Merging memtable contents | 10.153.42.85 |       45775241 |                                   ReadStage-2

76e98e60-cd6c-11e8-af38-373d9c5cfc0c | 923a5aa3-cd6c-11e8-9866-ade6e01c2040 |                                                                                                                      Merging data from sstable 2489 | 10.153.42.85 |       45775454 |                                   ReadStage-2

76e98e60-cd6c-11e8-af38-373d9c5cfc0c | 923a5aa4-cd6c-11e8-9866-ade6e01c2040 |                                                                                               Partition index with 0 entries found for sstable 2489 | 10.153.42.85 |       45775602 |                                   ReadStage-2

76e98e60-cd6c-11e8-af38-373d9c5cfc0c | 923a81b0-cd6c-11e8-9866-ade6e01c2040 |                                                                                                                      Merging data from sstable 2455 | 10.153.42.85 |       45775894 |                                   ReadStage-2

76e98e60-cd6c-11e8-af38-373d9c5cfc0c | 923a81b1-cd6c-11e8-9866-ade6e01c2040 |                                                                                               Partition index with 2 entries found for sstable 2455 | 10.153.42.85 |       45776026 |                                   ReadStage-2

76e98e60-cd6c-11e8-af38-373d9c5cfc0c | 923aa8c0-cd6c-11e8-9866-ade6e01c2040 |                                                                                                                      Merging data from sstable 2335 | 10.153.42.85 |       45776836 |                                   ReadStage-2

76e98e60-cd6c-11e8-af38-373d9c5cfc0c | 923aa8c1-cd6c-11e8-9866-ade6e01c2040 |                                                                                                                      Key cache hit for sstable 2335 | 10.153.42.85 |       45776958 |                                   ReadStage-2

76e98e60-cd6c-11e8-af38-373d9c5cfc0c | 923be140-cd6c-11e8-9866-ade6e01c2040 |                                                                                                                      Merging data from sstable 2293 | 10.153.42.85 |       45785346 |                                   ReadStage-2

76e98e60-cd6c-11e8-af38-373d9c5cfc0c | 923be141-cd6c-11e8-9866-ade6e01c2040 |                                                                                                                      Key cache hit for sstable 2293 | 10.153.42.85 |       45785532 |                                   ReadStage-2

76e98e60-cd6c-11e8-af38-373d9c5cfc0c | 923ccba0-cd6c-11e8-9866-ade6e01c2040 |                                                                                                                   Read 0 live and 0 tombstone cells | 10.153.42.85 |       45791600 |                                   ReadStage-2

76e98e60-cd6c-11e8-af38-373d9c5cfc0c | 923cf2b0-cd6c-11e8-9866-ade6e01c2040 |                                                                                                           Merged data from memtables and 3 sstables | 10.153.42.85 |       45791761 |                                   ReadStage-2

76e98e60-cd6c-11e8-af38-373d9c5cfc0c | 923cf2b1-cd6c-11e8-9866-ade6e01c2040 |                                                                                            Discarding partial response to /10.153.42.83 (timed out) | 10.153.42.85 |       45791858 |                                   ReadStage-2

 


On Thu, Oct 11, 2018 at 4:36 PM Adam Zegelin <adam@xxxxxxxxxxxxxxx> wrote:
Hi Abdul,

Can you post the timeout error and also the trace output?

The trace output is detailed, but often refers to Cassandra internals. If we can see the error and the trace it'd be easier to pin-point the problem.

- Adam


On Thu, 11 Oct 2018 at 08:53, Abdul Patel <abd786.ap@xxxxxxxxx> wrote:
Hi,

We have multiple timeouts with select queries.
I ran trace to see why its timing out ..do we jave a good document to interpretate whats causing the queries to timeout?