osdir.com


[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: sstableloader making no progress


Adding to the above, each host shows the following log messages that, despite being at INFO level, appear like stack traces to me:

2017-02-13 15:09:22,166 INFO  [STREAM-INIT-/10.128.X.Y:60306]  StreamResultFuture.java:116 - [Stream #afe548d0-f230-11e6-bc5d-8f99f25bfcf7, ID#0] Received streaming plan for Bulk Load
at clojure.lang.Var.invoke(Var.java:401)
at opsagent.config_service$update_system$fn__20140.invoke(config_service.clj:205)
at clojure.core$reduce.invoke(core.clj:6518)
at clojure.lang.RestFn.invoke(RestFn.java:425)
at opsagent.config_service$fn__20217$fn__20218$state_machine__4128__auto____20219$fn__20221.invoke(config_service.clj:250)
at clojure.core.async.impl.ioc_macros$run_state_machine.invoke(ioc_macros.clj:940)
at clojure.core.async$ioc_alts_BANG_$fn__4293.invoke(async.clj:362)
at clojure.core.async.impl.channels.ManyToManyChannel$fn__624.invoke(channels.clj:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.lang.Thread.run(Thread.java:745)

2017-02-13 15:09:22,208 INFO  [STREAM-IN-/10.128.X.Y]  StreamResultFuture.java:166 - [Stream #afe548d0-f230-11e6-bc5d-8f99f25bfcf7 ID#0] Prepare completed. Receiving 3 files(3963 bytes), sending 0 files(0 bytes)
at clojure.lang.ArraySeq.reduce(ArraySeq.java:114)
at opsagent.config_service$update_system.doInvoke(config_service.clj:199)
at opsagent.config_service$start_system_BANG_.invoke(config_service.clj:224)
at opsagent.config_service$fn__20217$fn__20218$state_machine__4128__auto____20219.invoke(config_service.clj:247)
at clojure.core.async.impl.ioc_macros$run_state_machine_wrapped.invoke(ioc_macros.clj:944)
at clojure.core.async$do_alts$fn__4247$fn__4250.invoke(async.clj:231)
at clojure.lang.AFn.run(AFn.java:22)


On Fri, Feb 10, 2017 at 4:28 PM, Simone Franzini <captainfranz@xxxxxxxxx> wrote:
I am trying to ingest some data from a cluster to a different cluster via sstableloader. I am running DSE 4.8.7 / Cassandra 2.1.14. 
I have re-created the schemas and followed other instructions here:

I am initially testing the ingest process with a single table, containing 3 really small sstables (just a few KB each):
sstableloader -v -d <mynode> <mykeyspace>/<mytable>
From the console, it appears that the progress quickly reaches 100%, but the command never returns:
progress: [/10.128.X.Y]0:3/3 100% [/10.192.Z.W]0:3/3 100% ... total: 100% 0  MB/s(avg: 0 MB/s)

nodetool netstats shows that there is no progress:
Mode: NORMAL
Bulk Load e495cea0-efde-11e6-9ec0-8f99f25bfcf7
    /10.128.X.Y
        Receiving 3 files, 3963 bytes total. Already received 0 files, 0 bytes total
Bulk Load b2566980-efb7-11e6-a467-8f99f25bfcf7
    /10.128.X.Y
        Receiving 3 files, 3963 bytes total. Already received 0 files, 0 bytes total
Bulk Load f31e7810-efdd-11e6-8484-8f99f25bfcf7
    /10.128.X.Y
        Receiving 3 files, 3963 bytes total. Already received 0 files, 0 bytes total
...
Read Repair Statistics:
Attempted: 8
Mismatch (Blocking): 0
Mismatch (Background): 0
Pool Name                    Active   Pending      Completed
Commands                        n/a         0        2148112
Responses                       n/a         0         977176


The logs show the following, but no error or warning message:
2017-02-10 16:18:49,096 INFO  [STREAM-INIT-/10.128.X.Y:33302]  StreamResultFuture.java:109 - [Stream #e495cea0-efde-11e6-9ec0-8f99f25bfcf7 ID#0] Creating new streaming plan for Bulk Load
2017-02-10 16:18:49,105 INFO  [STREAM-INIT-/10.128.X.Y:33302]  StreamResultFuture.java:116 - [Stream #e495cea0-efde-11e6-9ec0-8f99f25bfcf7, ID#0] Received streaming plan for Bulk Load
2017-02-10 16:18:49,110 INFO  [STREAM-INIT-/10.128.X.Y:33306]  StreamResultFuture.java:116 - [Stream #e495cea0-efde-11e6-9ec0-8f99f25bfcf7, ID#0] Received streaming plan for Bulk Load
2017-02-10 16:18:49,110 INFO  [STREAM-IN-/10.128.X.Y]  StreamResultFuture.java:166 - [Stream #e495cea0-efde-11e6-9ec0-8f99f25bfcf7 ID#0] Prepare completed. Receiving 3 files(3963 bytes), sending 0 files(0 bytes)


Any help would be greatly appreciated.