osdir.com


[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: Intermediate failure of materialized views in Cassandra adapter


Build is passing now. JDK11 explicitly skipped due to cassandra
compatibility. Have to wait for Cassandra 4.x (to run on JDK11) which is
not yet released.

Feel free to take my commit 981ab2888 from PR#769 or merge it directly.

On Fri, Jul 27, 2018 at 5:23 PM Andrei Sereda <andrei@xxxxxxxxx> wrote:

> I have created a PR perhaps you can rebase on the top :
> https://github.com/apache/calcite/pull/769
>
> There is one issue with JDK11. I believe cassandra 3.x doesn't work with
> JDK11 yet: see CASSANDRA-9608
> <https://issues.apache.org/jira/browse/CASSANDRA-9608>
>
> Can this test be ignored on JDK11 ?
>
> On Fri, Jul 27, 2018 at 4:39 PM Andrei Sereda <andrei@xxxxxxxxx> wrote:
>
>> Take a look at my branch c389451 :
>>
>>
>> https://github.com/asereda-gs/calcite/commit/c38945122e1edf4b417db6b82823b4a1b8c2c8b5
>>
>> It works both with Maven and IntelliJ
>>
>> On Fri, Jul 27, 2018 at 10:36 AM Michael Mior <mmior@xxxxxxxxxxxx> wrote:
>>
>>> That would be great if you could check that out as I'm not an IntelliJ
>>> user
>>> myself. Thanks!
>>>
>>> --
>>> Michael Mior
>>> mmior@xxxxxxxxxxxx
>>>
>>>
>>> Le jeu. 26 juil. 2018 à 22:11, Andrei Sereda <andrei@xxxxxxxxx> a écrit
>>> :
>>>
>>> > I think there are several problems:
>>> >
>>> > 1) "target/embeddedCassandra" is not in the classpath (of cassandra
>>> daemon)
>>> > so .properties file is not found.
>>> > 2) There are multiple loggers in the classpath. First one (which
>>> happens to
>>> > be logback) is selected by slf4j.
>>> >
>>> > SLF4J: Class path contains multiple SLF4J bindings.
>>> > SLF4J: Found binding in
>>> >
>>> >
>>> [jar:file:/Users/pupkin/.m2/repository/ch/qos/logback/logback-classic/1.1.3/logback-classic-1.1.3.jar!/org/slf4j/impl/StaticLoggerBinder.class]
>>> > SLF4J: Found binding in
>>> >
>>> >
>>> [jar:file:/Users/pupkin/.m2/repository/org/slf4j/slf4j-log4j12/1.7.13/slf4j-log4j12-1.7.13.jar!/org/slf4j/impl/StaticLoggerBinder.class]
>>> > SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an
>>> > explanation.
>>> >
>>> > To reduce DEBUG noise remove slf4j-log4j12 as dependency (from
>>> pom.xml) and
>>> > add logback-test.xml with sensitive defaults to src/test/resources/
>>> >
>>> > Running from IDE (IntelliJ)
>>> > One problem I have noticed is that test fails when running manually
>>> from
>>> > IntelliJ. Do you want me to take a look ?
>>> >
>>> > java.lang.ExceptionInInitializerError
>>> > at
>>> >
>>> >
>>> org.cassandraunit.utils.EmbeddedCassandraServerHelper.mkdir(EmbeddedCassandraServerHelper.java:321)
>>> > at
>>> >
>>> >
>>> org.cassandraunit.utils.EmbeddedCassandraServerHelper.copy(EmbeddedCassandraServerHelper.java:306)
>>> > at
>>> >
>>> >
>>> org.cassandraunit.utils.EmbeddedCassandraServerHelper.startEmbeddedCassandra(EmbeddedCassandraServerHelper.java:106)
>>> > at
>>> >
>>> >
>>> org.cassandraunit.utils.EmbeddedCassandraServerHelper.startEmbeddedCassandra(EmbeddedCassandraServerHelper.java:88)
>>> > at
>>> >
>>> >
>>> org.cassandraunit.utils.EmbeddedCassandraServerHelper.startEmbeddedCassandra(EmbeddedCassandraServerHelper.java:80)
>>> > at
>>> org.cassandraunit.BaseCassandraUnit.before(BaseCassandraUnit.java:31)
>>> >
>>> > *Caused by: java.lang.NullPointerException*
>>> > * at
>>> >
>>> >
>>> org.apache.cassandra.config.DatabaseDescriptor.getDiskFailurePolicy(DatabaseDescriptor.java:1881)*
>>> > at
>>> >
>>> >
>>> org.apache.cassandra.utils.JVMStabilityInspector.inspectThrowable(JVMStabilityInspector.java:82)
>>> > at org.apache.cassandra.io.util.FileUtils.<clinit>(FileUtils.java:79)
>>> > ... 14 more
>>> >
>>> > Regards,
>>> > Andrei.
>>> >
>>> > On Thu, Jul 26, 2018 at 2:55 PM Andrei Sereda <andrei@xxxxxxxxx>
>>> wrote:
>>> >
>>> > > I'll check that tonight
>>> > >
>>> > > On Thu, Jul 26, 2018, 13:29 Michael Mior <mmior@xxxxxxxxxx> wrote:
>>> > >
>>> > >> I already tried basically the same myself and the test consistently
>>> > passes
>>> > >> now. I updated the PR. If you're able to give it a try before I
>>> merge,
>>> > >> that
>>> > >> would be great. Also, I can't seem to figure out the right
>>> configuration
>>> > >> to
>>> > >> turn off the incredibly verbose logging CassandraUnit uses. It's
>>> not a
>>> > >> blocker, but would be nice to fix. (You can see the config which
>>> gets
>>> > >> pulled in under
>>> > >>
>>> cassandra/target/embeddedCassandra/log4j-embedded-cassandra.properties)
>>> > >>
>>> > >> --
>>> > >> Michael Mior
>>> > >> mmior@xxxxxxxxxx
>>> > >>
>>> > >>
>>> > >>
>>> > >> Le mer. 25 juil. 2018 à 19:41, Andrei Sereda <andrei@xxxxxxxxx> a
>>> > écrit :
>>> > >>
>>> > >> > My issue was with JDK10 and maven javadoc plugin (see MJAVADOC-517
>>> > >> > <https://issues.apache.org/jira/browse/MJAVADOC-517>)
>>> > >> >
>>> > >> > Can you double-check if @NotThreadSafe fixes the issue with the
>>> > >> following
>>> > >> > commands:
>>> > >> >
>>> > >> > $ cd cassandra
>>> > >> >
>>> > >> > $ seq 20 | xargs -i sh -c "mvn -Pit integration-test verify ||
>>> exit
>>> > 255"
>>> > >> >
>>> > >> >
>>> > >> >  You can also use -DforkCount=0 to run tests sequentially.
>>> > >> >
>>> > >> >
>>> > >> > On Wed, Jul 25, 2018 at 7:16 PM Michael Mior <mmior@xxxxxxxxxx>
>>> > wrote:
>>> > >> >
>>> > >> > > Good call. Adding a @NotThreadSafe annotation to the test class
>>> > seems
>>> > >> to
>>> > >> > > solve the problem. The setup and teardown of CassandraUnit is
>>> still
>>> > >> > shared
>>> > >> > > between all the tests and each test (after setup) runs rather
>>> > quickly
>>> > >> so
>>> > >> > I
>>> > >> > > don't think running the tests serially is a problem. I've never
>>> seen
>>> > >> the
>>> > >> > > error you're seeing before. A full stack trace might help.
>>> > >> > >
>>> > >> > > --
>>> > >> > > Michael Mior
>>> > >> > > mmior@xxxxxxxxxx
>>> > >> > >
>>> > >> > >
>>> > >> > >
>>> > >> > > Le mer. 25 juil. 2018 à 18:16, Andrei Sereda <andrei@xxxxxxxxx>
>>> a
>>> > >> écrit
>>> > >> > :
>>> > >> > >
>>> > >> > > > Did you run the tests from IDE or command line (maven) ? Do
>>> you
>>> > get
>>> > >> > some
>>> > >> > > > behaviour in both ?
>>> > >> > > >
>>> > >> > > > Integration tests (-Pit) run by maven (surefire) are executed
>>> in
>>> > >> > parallel
>>> > >> > > > (see pom.xml <
>>> > >> > https://github.com/apache/calcite/blob/master/pom.xml#L876
>>> > >> > > > >).
>>> > >> > > > This may contribute to non-determinism (eg. sharing some
>>> static
>>> > >> > variable
>>> > >> > > > like Hook).
>>> > >> > > >
>>> > >> > > > I will try to reproduce it myself to better understand what's
>>> > >> happening
>>> > >> > > > with cassandra adapter. Still getting NPE in maven for some
>>> > reason.
>>> > >> > > >
>>> > >> > > > On Wed, Jul 25, 2018 at 4:53 PM Michael Mior <
>>> mmior@xxxxxxxxxx>
>>> > >> wrote:
>>> > >> > > >
>>> > >> > > > > This is unrelated to that branch (except that I don't want
>>> to
>>> > >> merge
>>> > >> > the
>>> > >> > > > > branch with CassandraUnit until tests pass).
>>> > >> > > > >
>>> > >> > > > > --
>>> > >> > > > > Michael Mior
>>> > >> > > > > mmior@xxxxxxxxxx
>>> > >> > > > >
>>> > >> > > > >
>>> > >> > > > >
>>> > >> > > > > Le mer. 25 juil. 2018 à 16:36, Andrei Sereda
>>> <andrei@xxxxxxxxx>
>>> > a
>>> > >> > > écrit
>>> > >> > > > :
>>> > >> > > > >
>>> > >> > > > > > Mike,
>>> > >> > > > > >
>>> > >> > > > > > Is your branch michaelmior:2419-cassandra-unit up to date
>>> ?
>>> > I'm
>>> > >> > > getting
>>> > >> > > > > > NPE:
>>> > >> > > > > >
>>> > >> > > > > >
>>> > >> > > > > > > Caused by: java.lang.NullPointerException
>>> > >> > > > > > > at
>>> > >> > > > > > >
>>> > >> > > > > >
>>> > >> > > > >
>>> > >> > > >
>>> > >> > >
>>> > >> >
>>> > >>
>>> >
>>> org.apache.cassandra.config.DatabaseDescriptor.getDiskFailurePolicy(DatabaseDescriptor.java:1881)
>>> > >> > > > > > > at
>>> > >> > > > > > >
>>> > >> > > > > >
>>> > >> > > > >
>>> > >> > > >
>>> > >> > >
>>> > >> >
>>> > >>
>>> >
>>> org.apache.cassandra.utils.JVMStabilityInspector.inspectThrowable(JVMStabilityInspector.java:82)
>>> > >> > > > > > > at
>>> > >> > >
>>> org.apache.cassandra.io.util.FileUtils.<clinit>(FileUtils.java:79)
>>> > >> > > > > > > ... 13 more
>>> > >> > > > > >
>>> > >> > > > > >
>>> > >> > > > > > What error are you getting ?
>>> > >> > > > > >
>>> > >> > > > > > Regards,
>>> > >> > > > > > Andrei.
>>> > >> > > > > >
>>> > >> > > > > > On Wed, Jul 25, 2018 at 1:29 PM Michael Mior <
>>> > mmior@xxxxxxxxxx>
>>> > >> > > wrote:
>>> > >> > > > > >
>>> > >> > > > > > > I'd appreciate some help with the JIRA below. I'm using
>>> > >> > > Hook.TRIMMED
>>> > >> > > > > > > to expose materialized views to the adapter and a test
>>> which
>>> > >> > checks
>>> > >> > > > > > > that the view can be used for query rewriting is failing
>>> > about
>>> > >> > 50%
>>> > >> > > of
>>> > >> > > > > > > the time. I don't see why there should be any
>>> > non-determinism
>>> > >> > here
>>> > >> > > so
>>> > >> > > > > > > any ideas appreciated :)
>>> > >> > > > > > >
>>> > >> > > > > > > https://issues.apache.org/jira/browse/CALCITE-2423
>>> > >> > > > > > >
>>> > >> > > > > > > --
>>> > >> > > > > > > Michael Mior
>>> > >> > > > > > > mmior@xxxxxxxxxx
>>> > >> > > > > > >
>>> > >> > > > > >
>>> > >> > > > >
>>> > >> > > >
>>> > >> > >
>>> > >> >
>>> > >>
>>> > >
>>> >
>>>
>>