[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: Does Calcite hold all records output from a node before passing them to a higher node ?

In theory it certainly should be possible to stream the results. This isn't
guaranteed however. You would have to look at the entire query pipeline to
see where things are being materialized. A full stack trace without
elements removed would be a good start.

Michael Mior

Le lun. 28 mai 2018 à 19:05, Muhammad Gelbana <m.gelbana@xxxxxxxxx> a
écrit :

> I'm not sure if I phrased my question correctly so let me explain more.
> I'm running a (SELECT * FROM TABLE) query against a 50 million records
> table (Following the BINDABLE convention, so it sends it's rows through a
> "sink"). Since the extracted rows aren't processed in any way, I was
> expecting that the output JDBC resultset would be able to enumerate through
> all the results in a matter of seconds, but instead, my machine didn't
> print anything. What exactly happens is that
> (PreparedStatement.executeQuery) doesn't return a resultset promptly even
> after a few minutes have passed.
> I tried a table with hundreds of rows and my testing code printed those
> results right away so it's not something I missed there, but probably a
> configuration I didn't set ? Or may be that's just how it is ? Does anyone
> else believe that the behaviour I expected is reasonable ? It would also
> lower the amount of memory consumed to hold the complete results before
> bursting them to their final destination, if that's the case in the first
> place.
> Thanks,
> Gelbana