osdir.com

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [v1.15] A NPE that looks inevitable !


I found this issue and it explains what I was facing:
https://issues.apache.org/jira/browse/CALCITE-1436
I excluded the bindable aggregate rule and the query executed fine

@Override
    public void register(RelOptPlanner planner) {
        RULES.forEach(planner::addRule);
        Bindables.RULES.stream().filter(rule -> rule !=
Bindables.BINDABLE_AGGREGATE_RULE).forEach(planner::addRule);
    }

The weird part is that if I don't add any bindable rule, I don't get any
rows from Calcite's resultset ! I thought that Calcite will automatically
convert my nodes results to enumerable format for further enumerable nodes.
Anyway, no error was thrown and no result was returned. I appreciate if
someone would explain to me why is that.

I build my planner this way:

private static Planner buildPlanner(Session session) {
        //@formatter:off
        Config parserConfig = configBuilder()
                .setUnquotedCasing(Casing.UNCHANGED)
                .setQuotedCasing(Casing.UNCHANGED)
                .setCaseSensitive(true)
                .setQuoting(Quoting.DOUBLE_QUOTE)
                .setParserFactory(SqlParserImpl.FACTORY)
                .build();
        //@formatter:on

        SchemaPlus rootSchema = Frameworks.createRootSchema(true);
        session.getDataSchemas().stream().map(ds -> dataSchemaToSQLi(ds,
session)).forEach(sqliSchema -> {
            rootSchema.add(sqliSchema.getName(), sqliSchema);
        });

        List<RelTraitDef> traitDefs = new ArrayList<>();
        traitDefs.add(ConventionTraitDef.INSTANCE);

        //@formatter:off
        FrameworkConfig frameworkConfig = Frameworks.newConfigBuilder()
                .parserConfig(parserConfig)
                .defaultSchema(rootSchema)
                //.ruleSets(RuleSets.ofList(SQLiQuery.RULES)) -- This seems
to have no effect, I don't know why !
                .traitDefs(traitDefs)
                .build();
        //@formatter:on
        return Frameworks.getPlanner(frameworkConfig);
    }

Thanks,
Gelbana

On Mon, May 21, 2018 at 12:13 AM, Muhammad Gelbana <m.gelbana@xxxxxxxxx>
wrote:

> Back to the original problem (NPE at AggregateNode.getAccumulator, line
> 150), I've hit this bug again and I'm still struggling to reproduce it
> through a test case or evade it altogether.
>
> The bug is reproducible through my custom adaptor using this query:
>
> SELECT cn."COUNTRY_NAME", count(prd."PROD_NAME") Product_Count, count(
> *DISTINCT* (prd."PROD_NAME")) Product_Distinct FROM SALES.SALES1 s LEFT
> OUTER JOIN SALES.PRODUCTS prd ON s."PROD_ID" = prd."PROD_ID" LEFT OUTER
> JOIN SALES.CUSTOMERS c ON s."CUSTOMER_ID" = c."CUST_ID" LEFT OUTER JOIN
> SALES.COUNTRIES cn ON c."COUNTRY_ID" = cn."COUNTRY_ID" GROUP BY
> cn."COUNTRY_NAME"
>
> I noticed that if I remove the DISTINCT keyword (highlighted in bold), the
> bug isn't activated. I developed my adaptor by following the Druid adapter
> way, I think I missed something but I can't decide what it is.
>
> I've been thinking that since this piece of code should not be activated,
> as it will always throw an exception. Could someone please tell me why
> would an executing query hit in the first place ? Or how should I configure
> my adaptor, planner, query..etc to NOT hit that code ?
>
> Thanks,
> Gelbana
>
> On Wed, Apr 25, 2018 at 9:56 PM, Muhammad Gelbana <m.gelbana@xxxxxxxxx>
> wrote:
>
>> Thank you! That helped a lot.
>>
>> Thanks,
>> Gelbana
>>
>> On Wed, Apr 25, 2018 at 12:00 PM, Stamatis Zampetakis <zabetak@xxxxxxxxx>
>> wrote:
>>
>>> Hi Gelbana,
>>>
>>> I haven't looked at your snippet but if you want to debug into generated
>>> code have a look at the following link:
>>> https://calcite.apache.org/docs/howto.html#debugging-generat
>>> ed-classes-in-intellij
>>>
>>> Best,
>>> Stamatis
>>>
>>> 2018-04-21 14:07 GMT+02:00 Muhammad Gelbana <m.gelbana@xxxxxxxxx>:
>>>
>>> > I'm trying to create the test case for this issue but I can't run a
>>> simple
>>> > query as a start.
>>> >
>>> > I'm facing the following exception
>>> >
>>> > java.sql.SQLException: exception while executing query: null
>>> >
>>> >     at org.apache.calcite.avatica.Helper.createException(Helper.jav
>>> a:56)
>>> >     at org.apache.calcite.avatica.Helper.createException(Helper.jav
>>> a:41)
>>> >     at
>>> > org.apache.calcite.avatica.AvaticaConnection.executeQueryInternal(
>>> > AvaticaConnection.java:540)
>>> >     at
>>> > org.apache.calcite.avatica.AvaticaPreparedStatement.executeQuery(
>>> > AvaticaPreparedStatement.java:133)
>>> >     at
>>> > org.apache.calcite.tools.FrameworksTest.aggNPE(FrameworksTes
>>> t.java:184)
>>> >     at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>> >     at
>>> > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAcce
>>> ssorImpl.java:
>>> > 62)
>>> >     at
>>> > sun.reflect.DelegatingMethodAccessorImpl.invoke(
>>> > DelegatingMethodAccessorImpl.java:43)
>>> >     at java.lang.reflect.Method.invoke(Method.java:498)
>>> >     at
>>> > org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(
>>> > FrameworkMethod.java:50)
>>> >     at
>>> > org.junit.internal.runners.model.ReflectiveCallable.run(
>>> > ReflectiveCallable.java:12)
>>> >     at
>>> > org.junit.runners.model.FrameworkMethod.invokeExplosively(
>>> > FrameworkMethod.java:47)
>>> >     at
>>> > org.junit.internal.runners.statements.InvokeMethod.
>>> > evaluate(InvokeMethod.java:17)
>>> >     at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
>>> >     at
>>> > org.junit.runners.BlockJUnit4ClassRunner.runChild(
>>> > BlockJUnit4ClassRunner.java:78)
>>> >     at
>>> > org.junit.runners.BlockJUnit4ClassRunner.runChild(
>>> > BlockJUnit4ClassRunner.java:57)
>>> >     at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
>>> >     at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
>>> >     at org.junit.runners.ParentRunner.runChildren(ParentRunner.java
>>> :288)
>>> >     at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
>>> >     at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:
>>> 268)
>>> >     at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
>>> >     at org.junit.runner.JUnitCore.run(JUnitCore.java:137)
>>> >     at
>>> > com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(
>>> > JUnit4IdeaTestRunner.java:68)
>>> >     at
>>> > com.intellij.rt.execution.junit.IdeaTestRunner$Repeater.
>>> > startRunnerWithArgs(IdeaTestRunner.java:47)
>>> >     at
>>> > com.intellij.rt.execution.junit.JUnitStarter.prepareStreamsAndStart(
>>> > JUnitStarter.java:242)
>>> >     at
>>> > com.intellij.rt.execution.junit.JUnitStarter.main(JUnitStart
>>> er.java:70)
>>> >
>>> > *Caused by: java.lang.NullPointerException    at Baz.bind(Unknown
>>> Source)*
>>> >     at
>>> > org.apache.calcite.adapter.enumerable.EnumerableBindable.
>>> > bind(EnumerableBindable.java:70)
>>> >     at
>>> > org.apache.calcite.jdbc.CalcitePrepare$CalciteSignature.enumerable(
>>> > CalcitePrepare.java:365)
>>> >     at
>>> > org.apache.calcite.jdbc.CalciteConnectionImpl.enumerable(
>>> > CalciteConnectionImpl.java:301)
>>> >     at
>>> > org.apache.calcite.jdbc.CalciteMetaImpl._createIterable(
>>> > CalciteMetaImpl.java:559)
>>> >     at
>>> > org.apache.calcite.jdbc.CalciteMetaImpl.createIterable(
>>> > CalciteMetaImpl.java:550)
>>> >     at
>>> > org.apache.calcite.avatica.AvaticaResultSet.execute(
>>> > AvaticaResultSet.java:172)
>>> >     at
>>> > org.apache.calcite.jdbc.CalciteResultSet.execute(CalciteResu
>>> ltSet.java:67)
>>> >     at
>>> > org.apache.calcite.jdbc.CalciteResultSet.execute(CalciteResu
>>> ltSet.java:44)
>>> >     at
>>> > org.apache.calcite.avatica.AvaticaConnection.executeQueryInternal(
>>> > AvaticaConnection.java:536)
>>> >     ... 24 more
>>> >
>>> > This is the test case I'm working on
>>> >
>>> > @Test
>>> > public void aggNPE() throws SqlParseException, ValidationException,
>>> > RelConversionException, SQLException {
>>> >   Function<Holder, Object> hook = (o1) -> {
>>> >     o1.set(true);
>>> >     return null;
>>> >   };
>>> >   Hook.ENABLE_BINDABLE.add(hook);
>>> >
>>> >   SchemaPlus rootSchema = Frameworks.createRootSchema(true);
>>> >   final FrameworkConfig config = Frameworks.newConfigBuilder()
>>> >           .defaultSchema(CalciteAssert.addSchema(rootSchema,
>>> > CalciteAssert.SchemaSpec.JDBC_FOODMART))
>>> >           .ruleSets(RuleSets.ofList(Bindables.RULES))
>>> >           .build();
>>> >
>>> >   Planner planner = Frameworks.getPlanner(config);
>>> >   SqlNode parsed = planner.parse("SELECT * FROM \"sales_fact_1997\"");
>>> >   SqlNode validated = planner.validate(parsed);
>>> >   RelRoot rel = planner.rel(validated);
>>> >
>>> >   PreparedStatement statement = RelRunners.run(rel.rel);
>>> >   ResultSet rs = statement.executeQuery();
>>> >
>>> >   rs.close();
>>> >   statement.close();
>>> > }
>>> >
>>> > I'm trying to be as close as possible to what I have in my schema so I
>>> > enabled the bindable convention and enabled the bindables rules. What
>>> am I
>>> > doing wrong here to cause this NPE ?
>>> > I can't debug into the *Baz* class to figure out what's going on as the
>>> > class seems to be generated at runtime or something.
>>> >
>>> >
>>> > Thanks,
>>> > Gelbana
>>> >
>>> > On Thu, Mar 29, 2018 at 1:34 AM, Muhammad Gelbana <m.gelbana@xxxxxxxxx
>>> >
>>> > wrote:
>>> >
>>> > > With pleasure. I'll do that ASAP.
>>> > >
>>> > > Thanks,
>>> > > Gelbana
>>> > >
>>> > > On Thu, Mar 29, 2018 at 12:38 AM, Shuyi Chen <suez1224@xxxxxxxxx>
>>> wrote:
>>> > >
>>> > >> It will be great if you can provide a test to reproduce the
>>> problem, and
>>> > >> attach it in the JIRA.
>>> > >>
>>> > >> On Wed, Mar 28, 2018 at 2:50 PM, Muhammad Gelbana <
>>> m.gelbana@xxxxxxxxx>
>>> > >> wrote:
>>> > >>
>>> > >> > Absolutely: https://issues.apache.org/jira/browse/CALCITE-2228
>>> > >> >
>>> > >> > Please tell me if I should modify any of the information I
>>> supplied.
>>> > >> Like
>>> > >> > the affected version\component.
>>> > >> >
>>> > >> > Thanks,
>>> > >> > Gelbana
>>> > >> >
>>> > >> > On Wed, Mar 28, 2018 at 11:26 PM, Julian Hyde <jhyde@xxxxxxxxxx>
>>> > wrote:
>>> > >> >
>>> > >> > > It does indeed look as if that piece of code (AggregateNode.java
>>> > line
>>> > >> > 149)
>>> > >> > > has never been run. Can you please log a JIRA case?
>>> > >> > >
>>> > >> > > > On Mar 28, 2018, at 1:42 PM, Muhammad Gelbana <
>>> > m.gelbana@xxxxxxxxx>
>>> > >> > > wrote:
>>> > >> > > >
>>> > >> > > > Looking at this line
>>> > >> > > > <https://github.com/apache/calcite/blob/master/core/src/
>>> > >> > > main/java/org/apache/calcite/interpreter/AggregateNode.java#
>>> L150>,
>>> > >> > > > and the referenced instance variable
>>> > >> > > > <https://github.com/apache/calcite/blob/master/core/src/
>>> > >> > > main/java/org/apache/calcite/adapter/enumerable/AggImpState.
>>> > >> java#L33>,
>>> > >> > > > it looks to me that *int stateSize = agg.state.size* will
>>> always
>>> > >> throw
>>> > >> > a
>>> > >> > > > NPE because the *agg.state* object reference is never
>>> initialized
>>> > >> > before
>>> > >> > > > it's being used !
>>> > >> > > >
>>> > >> > > > I'm facing the following exception
>>> > >> > > >
>>> > >> > > > java.sql.SQLException: Error while executing SQL "SELECT
>>> > >> > MAX(COUNTRY_ID),
>>> > >> > > > MAX(COUNTRY_TOTAL_ID) FROM SALES.COUNTRIES": null
>>> > >> > > >    at org.apache.calcite.avatica.Hel
>>> per.createException(Helper.
>>> > >> > java:56)
>>> > >> > > >    at org.apache.calcite.avatica.Hel
>>> per.createException(Helper.
>>> > >> > java:41)
>>> > >> > > >    at
>>> > >> > > > org.apache.calcite.avatica.AvaticaStatement.executeInternal(
>>> > >> > > AvaticaStatement.java:156)
>>> > >> > > >    at
>>> > >> > > > org.apache.calcite.avatica.AvaticaStatement.executeQuery(
>>> > >> > > AvaticaStatement.java:218)
>>> > >> > > >    at
>>> > >> > > > com.mg.sqlinterface.data.provider.calcite.internal.
>>> > >> > > Test_SQLi.testMG(Test_SQLi.java:35)
>>> > >> > > >    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
>>> Method)
>>> > >> > > >    at
>>> > >> > > > sun.reflect.NativeMethodAccessorImpl.invoke(
>>> > >> > > NativeMethodAccessorImpl.java:62)
>>> > >> > > >    at
>>> > >> > > > sun.reflect.DelegatingMethodAccessorImpl.invoke(
>>> > >> > > DelegatingMethodAccessorImpl.java:43)
>>> > >> > > >    at java.lang.reflect.Method.invoke(Method.java:498)
>>> > >> > > >    at
>>> > >> > > > org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(
>>> > >> > > FrameworkMethod.java:50)
>>> > >> > > >    at
>>> > >> > > > org.junit.internal.runners.model.ReflectiveCallable.run(
>>> > >> > > ReflectiveCallable.java:12)
>>> > >> > > >    at
>>> > >> > > > org.junit.runners.model.FrameworkMethod.invokeExplosively(
>>> > >> > > FrameworkMethod.java:47)
>>> > >> > > >    at
>>> > >> > > > org.junit.internal.runners.statements.InvokeMethod.
>>> > >> > > evaluate(InvokeMethod.java:17)
>>> > >> > > >    at org.junit.runners.ParentRunner.runLeaf(
>>> > ParentRunner.java:325)
>>> > >> > > >    at
>>> > >> > > > org.junit.runners.BlockJUnit4ClassRunner.runChild(
>>> > >> > > BlockJUnit4ClassRunner.java:78)
>>> > >> > > >    at
>>> > >> > > > org.junit.runners.BlockJUnit4ClassRunner.runChild(
>>> > >> > > BlockJUnit4ClassRunner.java:57)
>>> > >> > > >    at org.junit.runners.ParentRunner
>>> $3.run(ParentRunner.java:290)
>>> > >> > > >    at org.junit.runners.ParentRunner$1.schedule(
>>> > ParentRunner.java:
>>> > >> 71)
>>> > >> > > >    at org.junit.runners.ParentRunner.runChildren(
>>> > >> > ParentRunner.java:288)
>>> > >> > > >    at org.junit.runners.ParentRunner.access$000(
>>> > ParentRunner.java:
>>> > >> 58)
>>> > >> > > >    at org.junit.runners.ParentRunner$2.evaluate(
>>> > ParentRunner.java:
>>> > >> 268)
>>> > >> > > >    at
>>> > >> > > > org.junit.internal.runners.statements.RunBefores.
>>> > >> > > evaluate(RunBefores.java:26)
>>> > >> > > >    at
>>> > >> > > > org.junit.internal.runners.statements.RunAfters.evaluate(
>>> > >> > > RunAfters.java:27)
>>> > >> > > >    at org.junit.runners.ParentRunner
>>> .run(ParentRunner.java:363)
>>> > >> > > >    at
>>> > >> > > > org.eclipse.jdt.internal.junit4.runner.JUnit4TestReference.r
>>> un(
>>> > >> > > JUnit4TestReference.java:86)
>>> > >> > > >    at
>>> > >> > > > org.eclipse.jdt.internal.junit.runner.TestExecution.
>>> > >> > > run(TestExecution.java:38)
>>> > >> > > >    at
>>> > >> > > > org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.
>>> > >> > > runTests(RemoteTestRunner.java:459)
>>> > >> > > >    at
>>> > >> > > > org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.
>>> > >> > > runTests(RemoteTestRunner.java:678)
>>> > >> > > >    at
>>> > >> > > > org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.
>>> > >> > > run(RemoteTestRunner.java:382)
>>> > >> > > >    at
>>> > >> > > > org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.
>>> > >> > > main(RemoteTestRunner.java:192)
>>> > >> > > > Caused by: *java.lang.NullPointerException*
>>> > >> > > >    at org.apache.calcite.interpreter.AggregateNode.
>>> > getAccumulator(
>>> > >> > > > *AggregateNode.java:150*)
>>> > >> > > >    at
>>> > >> > > > org.apache.calcite.interpreter.AggregateNode.<
>>> > >> > > init>(AggregateNode.java:86)
>>> > >> > > >    at
>>> > >> > > > org.apache.calcite.interpreter.Nodes$CoreCompiler.visit(
>>> > >> Nodes.java:47)
>>> > >> > > >    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
>>> Method)
>>> > >> > > >    at
>>> > >> > > > sun.reflect.NativeMethodAccessorImpl.invoke(
>>> > >> > > NativeMethodAccessorImpl.java:62)
>>> > >> > > >    at
>>> > >> > > > sun.reflect.DelegatingMethodAccessorImpl.invoke(
>>> > >> > > DelegatingMethodAccessorImpl.java:43)
>>> > >> > > >    at java.lang.reflect.Method.invoke(Method.java:498)
>>> > >> > > >    at
>>> > >> > > > org.apache.calcite.util.ReflectUtil.invokeVisitorInternal(
>>> > >> > > ReflectUtil.java:257)
>>> > >> > > >    at
>>> > >> > > > org.apache.calcite.util.ReflectUtil.invokeVisitor(
>>> > >> > ReflectUtil.java:214)
>>> > >> > > >    at
>>> > >> > > > org.apache.calcite.util.ReflectUtil$1.invokeVisitor(
>>> > >> > > ReflectUtil.java:464)
>>> > >> > > >    at
>>> > >> > > > org.apache.calcite.interpreter.Interpreter$Compiler.visit(
>>> > >> Interpreter.
>>> > >> > > java:480)
>>> > >> > > >    at
>>> > >> > > > org.apache.calcite.interpreter.Interpreter$Compiler.visitRoo
>>> t(
>>> > >> > > Interpreter.java:437)
>>> > >> > > >    at
>>> > >> > > > org.apache.calcite.interpreter.Interpreter.<init>(
>>> > >> Interpreter.java:78)
>>> > >> > > >    at Baz.bind(Unknown Source)
>>> > >> > > >    at
>>> > >> > > > org.apache.calcite.jdbc.CalcitePrepare$
>>> > CalciteSignature.enumerable(
>>> > >> > > CalcitePrepare.java:365)
>>> > >> > > >    at
>>> > >> > > > org.apache.calcite.jdbc.CalciteConnectionImpl.enumerable(
>>> > >> > > CalciteConnectionImpl.java:301)
>>> > >> > > >    at
>>> > >> > > > org.apache.calcite.jdbc.CalciteMetaImpl._createIterable(
>>> > >> > > CalciteMetaImpl.java:559)
>>> > >> > > >    at
>>> > >> > > > org.apache.calcite.jdbc.CalciteMetaImpl.createIterable(
>>> > >> > > CalciteMetaImpl.java:550)
>>> > >> > > >    at
>>> > >> > > > org.apache.calcite.avatica.AvaticaResultSet.execute(
>>> > >> > > AvaticaResultSet.java:204)
>>> > >> > > >    at
>>> > >> > > > org.apache.calcite.jdbc.CalciteResultSet.execute(
>>> > >> > > CalciteResultSet.java:67)
>>> > >> > > >    at
>>> > >> > > > org.apache.calcite.jdbc.CalciteResultSet.execute(
>>> > >> > > CalciteResultSet.java:44)
>>> > >> > > >    at
>>> > >> > > > org.apache.calcite.avatica.AvaticaConnection$1.execute(
>>> > >> > > AvaticaConnection.java:630)
>>> > >> > > >    at
>>> > >> > > > org.apache.calcite.jdbc.CalciteMetaImpl.prepareAndExecute(
>>> > >> > > CalciteMetaImpl.java:619)
>>> > >> > > >    at
>>> > >> > > > org.apache.calcite.avatica.AvaticaConnection.
>>> > >> > prepareAndExecuteInternal(
>>> > >> > > AvaticaConnection.java:638)
>>> > >> > > >    at
>>> > >> > > > org.apache.calcite.avatica.AvaticaStatement.executeInternal(
>>> > >> > > AvaticaStatement.java:149)
>>> > >> > > >    ... 27 more
>>> > >> > > >
>>> > >> > > >
>>> > >> > > > Thanks,
>>> > >> > > > Gelbana
>>> > >> > > > ​​
>>> > >> > >
>>> > >> > >
>>> > >> >
>>> > >>
>>> > >>
>>> > >>
>>> > >> --
>>> > >> "So you have to trust that the dots will somehow connect in your
>>> > future."
>>> > >>
>>> > >
>>> > >
>>> >
>>>
>>
>>
>