OSDir


[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: Does Calcite hold all records output from a node before passing them to a higher node ?


I'm also hitting the following exception when selecting all records from a
huge table.

java.lang.OutOfMemoryError: Java heap space
    at java.util.stream.IntPipeline$4.opWrapSink(IntPipeline.java:247)
    at java.util.stream.AbstractPipeline.wrapSink(AbstractPipeline.java:517)
    at
java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471)
    at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:545)
    at
java.util.stream.AbstractPipeline.evaluateToArrayNode(AbstractPipeline.java:260)
    at
java.util.stream.ReferencePipeline.toArray(ReferencePipeline.java:438)
    at
java.util.stream.ReferencePipeline.toArray(ReferencePipeline.java:444)
    at
com.sqlinterface.data.schema.ResultsEnumerator.current(ResultsEnumerator.java:96)
    // Removed the rest of stacktrace

Thanks,
Gelbana

On Tue, May 29, 2018 at 1:05 AM, Muhammad Gelbana <m.gelbana@xxxxxxxxx>
wrote:

> I'm not sure if I phrased my question correctly so let me explain more.
>
> I'm running a (SELECT * FROM TABLE) query against a 50 million records
> table (Following the BINDABLE convention, so it sends it's rows through a
> "sink"). Since the extracted rows aren't processed in any way, I was
> expecting that the output JDBC resultset would be able to enumerate through
> all the results in a matter of seconds, but instead, my machine didn't
> print anything. What exactly happens is that (PreparedStatement.executeQuery)
> doesn't return a resultset promptly even after a few minutes have passed.
>
> I tried a table with hundreds of rows and my testing code printed those
> results right away so it's not something I missed there, but probably a
> configuration I didn't set ? Or may be that's just how it is ? Does anyone
> else believe that the behaviour I expected is reasonable ? It would also
> lower the amount of memory consumed to hold the complete results before
> bursting them to their final destination, if that's the case in the first
> place.
>
>
> Thanks,
> Gelbana
>