osdir.com

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: Build failure on 0.13.SNAPSHOT


After some experiments, I figured out the following:

1. Druid uses above 8gb of memory for testing. (building-druid.png)
2. With 8gb(physical)+4gb(swap) of memory, the test succeeds regardless of maven version (3.3.9, 3.5.2, 3.5.4) or MAVEN_OPTS. However, with 8gb(physical)+2gb(swap) of memory[^1], some tests failed. The list of failing tests differs between maven 3.3.9 and 3.5.2.

In short, retaining sufficient memory solved the problem - It seems like 12gb of memory is a recommended setting for building druid. (I guess lots of you are working with the MacBook Pro with 16gb RAM, right? In that case, you must not have encountered this problem.)

If you are okay, may I update the building documentation for the newbies like me?

Thanks,
Dongjin

+1. While building Druid, I found another problem. But this issue should be discussed in another thread.

[^1]: You know, the other processes also occupy the memory.


On Tue, Jul 24, 2018 at 3:07 AM Jihoon Son <jihoonson@xxxxxxxxxx> wrote:
I'm also using Maven 3.5.2 and not using any special configurations for
Maven, but I have never seen that error too.
Most of our Travis jobs have been working with only 512 MB of direct
memory. Only the 'strict compilation' Travis job requires 3 GB of memory.

I think it's worthwhile to look into this more. Maybe we somehow use more
memory when we run all tests by 'mvn install'. Maybe this relates to the
frequent transient failures of 'processing module test', one of our Travis
jobs.

Jihoon

On Mon, Jul 23, 2018 at 9:32 AM Gian Merlino <gian@xxxxxxxxxx> wrote:

> Interesting. Fwiw, I am using Maven 3.5.2 for building Druid and it has
> been working for for me. I don't think I"m using any special Maven
> overrides (at least, I don't see anything interesting in my ~/.m2 directory
> or in my environment variables). It might have to do with how much memory
> our machines have? I do most of my builds on a Mac with 16GB RAM. Maybe try
> checking .travis.yml in the druid repo. It sets -Xmx3000m for mvn install
> commands, which might be needed for more low memory environments.
>
> $ mvn --version
> Apache Maven 3.5.2 (138edd61fd100ec658bfa2d307c43b76940a5d7d;
> 2017-10-18T00:58:13-07:00)
> Maven home: /usr/local/Cellar/maven/3.5.2/libexec
> Java version: 1.8.0_161, vendor: Oracle Corporation
> Java home:
> /Library/Java/JavaVirtualMachines/jdk1.8.0_161.jdk/Contents/Home/jre
> Default locale: en_US, platform encoding: UTF-8
> OS name: "mac os x", version: "10.13.5", arch: "x86_64", family: "mac"
>
> On Mon, Jul 23, 2018 at 6:40 AM Dongjin Lee <dongjin@xxxxxxxxxx> wrote:
>
> > Finally, it seems like I found the reason. It was a composition of
> several
> > problems:
> >
> > - Druid should not be built with maven 3.5.x. With 3.5.2, Test suites
> like
> > `GroupByQueryRunnerFailureTest` fails. After I switched into 3.3.9 which
> is
> > built in the latest version of IntelliJ, those errors disappeared. It
> seems
> > like maven 3.5.x is not stable yet - it applied a drastic change, and it
> is
> > also why they skipped 3.4.x.
> > - It seems like Druid requires some MaxDirectMemorySize configuration for
> > some test suites. With some JVM parameter like
> `-XX:MaxDirectMemorySize=4g`
> > some test suites were passed, but not all. I am now trying the other
> > options with enlarged swap space.
> >
> > Question: How much MaxDirectMemorySize configuration are you using?
> >
> > Best,
> > Dongjin
> >
> > On Sat, Jul 21, 2018 at 3:01 AM Jihoon Son <ghoonson@xxxxxxxxx> wrote:
> >
> > > Hi Dongjin,
> > >
> > > that is weird. It looks like the vm crashed because of out of memory
> > while
> > > testing.
> > > It might be a real issue or not.
> > > Have you set any memory configuration for your maven?
> > >
> > > Jihoon
> > >
> > > On Thu, Jul 19, 2018 at 7:09 PM Dongjin Lee <dongjin@xxxxxxxxxx>
> wrote:
> > >
> > > > Hi Jihoon,
> > > >
> > > > I ran `mvn clean package` following development/build
> > > > <
> > > >
> > >
> >
> https://github.com/apache/incubator-druid/blob/master/docs/content/development/build.md
> > > > >
> > > > .
> > > >
> > > > Dongjin
> > > >
> > > > On Fri, Jul 20, 2018 at 12:30 AM Jihoon Son <jihoonson@xxxxxxxxxx>
> > > wrote:
> > > >
> > > > > Hi Dongjin,
> > > > >
> > > > > what maven command did you run?
> > > > >
> > > > > Jihoon
> > > > >
> > > > > On Wed, Jul 18, 2018 at 10:38 PM Dongjin Lee <dongjin@xxxxxxxxxx>
> > > wrote:
> > > > >
> > > > > > Hello. I am trying to build druid, but it fails. My environment
> is
> > > like
> > > > > the
> > > > > > following:
> > > > > >
> > > > > > - CPU: Intel(R) Core(TM) i7-7560U CPU @ 2.40GHz
> > > > > > - RAM: 7704 MB
> > > > > > - OS: ubuntu 18.04
> > > > > > - JDK: openjdk version "1.8.0_171" (default configuration, with
> > > > > MaxHeapSize
> > > > > > = 1928 MB)
> > > > > > - Branch: master (commit: cd8ea3d)
> > > > > >
> > > > > > The error message I got is:
> > > > > >
> > > > > > [INFO]
> > > > > > >
> > > > >
> > >
> ------------------------------------------------------------------------
> > > > > > > [INFO] Reactor Summary:
> > > > > > > [INFO]
> > > > > > > [INFO] io.druid:druid .....................................
> > > SUCCESS [
> > > > > > > 50.258 s]
> > > > > > > [INFO] java-util ..........................................
> > SUCCESS
> > > > > > [03:57
> > > > > > > min]
> > > > > > > [INFO] druid-api ..........................................
> > > SUCCESS [
> > > > > > > 22.694 s]
> > > > > > > [INFO] druid-common .......................................
> > > SUCCESS [
> > > > > > > 14.083 s]
> > > > > > > [INFO] druid-hll ..........................................
> > > SUCCESS [
> > > > > > > 17.126 s]
> > > > > > > [INFO] extendedset ........................................
> > > SUCCESS [
> > > > > > > 10.856 s]
> > > > > > >
> > > > > > > *[INFO] druid-processing ...................................
> > > FAILURE
> > > > > > > [04:36 min]*[INFO] druid-aws-common
> > > > ...................................
> > > > > > > SKIPPED
> > > > > > > [INFO] druid-server .......................................
> > SKIPPED
> > > > > > > [INFO] druid-examples .....................................
> > SKIPPED
> > > > > > > ...
> > > > > > > [INFO]
> > > > > > >
> > > > >
> > >
> ------------------------------------------------------------------------
> > > > > > > [INFO] BUILD FAILURE
> > > > > > > [INFO]
> > > > > > >
> > > > >
> > >
> ------------------------------------------------------------------------
> > > > > > > [INFO] Total time: 10:29 min
> > > > > > > [INFO] Finished at: 2018-07-19T13:23:31+09:00
> > > > > > > [INFO] Final Memory: 88M/777M
> > > > > > > [INFO]
> > > > > > >
> > > > >
> > >
> ------------------------------------------------------------------------
> > > > > > >
> > > > > > > *[ERROR] Failed to execute goal
> > > > > > > org.apache.maven.plugins:maven-surefire-plugin:2.19.1:test
> > > > > (default-test)
> > > > > > > on project druid-processing: Execution default-test of goal
> > > > > > > org.apache.maven.plugins:maven-surefire-plugin:2.19.1:test
> > failed:
> > > > The
> > > > > > > forked VM terminated without properly saying goodbye. VM crash
> or
> > > > > > > System.exit called?*[ERROR] Command was /bin/sh -c cd
> > > > > > > /home/djlee/workspace/java/druid/processing &&
> > > > > > > /usr/lib/jvm/java-8-openjdk-amd64/jre/bin/java -Xmx3000m
> > > > > > -Duser.language=en
> > > > > > > -Duser.country=US -Dfile.encoding=UTF-8 -Duser.timezone=UTC
> > > > > > >
> > -Djava.util.logging.manager=org.apache.logging.log4j.jul.LogManager
> > > > > > > -Ddruid.indexing.doubleStorage=double -jar
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> /home/djlee/workspace/java/druid/processing/target/surefire/surefirebooter1075382243904099051.jar
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> /home/djlee/workspace/java/druid/processing/target/surefire/surefire5444459351134757209tmp
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> /home/djlee/workspace/java/druid/processing/target/surefire/surefire_5173894389718744688tmp
> > > > > >
> > > > > >
> > > > > > It seems like it fails when it runs tests on `druid-processing`
> > > module
> > > > > but
> > > > > > I can't certain. Is there anyone who can give me some hints?
> Thanks
> > > in
> > > > > > advance.
> > > > > >
> > > > > > Best,
> > > > > > Dongjin
> > > > > >
> > > > > > --
> > > > > > *Dongjin Lee*
> > > > > >
> > > > > > *A hitchhiker in the mathematical world.*
> > > > > >
> > > > > > *github:  <http://goog_969573159/>github.com/dongjinleekr
> > > > > > <http://github.com/dongjinleekr>linkedin:
> > > > > kr.linkedin.com/in/dongjinleekr
> > > > > > <http://kr.linkedin.com/in/dongjinleekr>slideshare:
> > > > > > www.slideshare.net/dongjinleekr
> > > > > > <http://www.slideshare.net/dongjinleekr>*
> > > > > >
> > > > >
> > > > > --
> > > > > *Dongjin Lee*
> > > > >
> > > > > *A hitchhiker in the mathematical world.*
> > > > >
> > > > > *github:  <http://goog_969573159/>github.com/dongjinleekr
> > > > > <http://github.com/dongjinleekr>linkedin:
> > > > kr.linkedin.com/in/dongjinleekr
> > > > > <http://kr.linkedin.com/in/dongjinleekr>slideshare:
> > > > www.slideshare.net/dongjinleekr
> > > > > <http://www.slideshare.net/dongjinleekr>*
> > > > >
> > > >
> > >
> > > --
> > > *Dongjin Lee*
> > >
> > > *A hitchhiker in the mathematical world.*
> > >
> > > *github:  <http://goog_969573159/>github.com/dongjinleekr
> > > <http://github.com/dongjinleekr>linkedin:
> > kr.linkedin.com/in/dongjinleekr
> > > <http://kr.linkedin.com/in/dongjinleekr>slideshare:
> > www.slideshare.net/dongjinleekr
> > > <http://www.slideshare.net/dongjinleekr>*
> > >
> >
>

--
Dongjin Lee

A hitchhiker in the mathematical world.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@xxxxxxxxxxxxxxxx
For additional commands, e-mail: dev-help@xxxxxxxxxxxxxxxx