osdir.com

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: Build failure on 0.13.SNAPSHOT


@Dongjin, thank you for looking into this. I got some follow-up questions.

> Druid uses above 8gb of memory for testing.

How did you get this result? Did you use some tools to get the memory usage
of only the maven process for Druid tests? When I checked last time using
JMC, I could see that running all unit tests of each Travis Job usually use
up to 512 MB of memory (you can find our Travis Job configuration here:
https://github.com/apache/incubator-druid/blob/master/.travis.yml). That's
why I set the memory limit to 512 MB for our Travis CI. Maybe we need to
check again and raise the memory limit to avoid frequent OOM failures in
groupBy query tests.

Also, if it does use 8 GB of memory, I suspect it's a sort of bug or
something. I don't think even our integration tests don't use such a large
memory which run all Druid processes, ZooKeeper, and Kafka. So, we should
figure out which tests use such a lot of memory and fix it instead of
fixing documents.

> Fwiw I think 8GB should be enough memory with the right settings, since
our Travis CI build environment only has about 7.5GB.

@Gian, the memory of the Travis instance is shared by all processes
including OS. AFAIK, the OS requires about 1 GB of memory. It means that
our unit tests can't be run on Travis if it needs 8 GB of memory.

Best,
Jihoon

On Wed, Jul 25, 2018 at 10:01 AM Charles Allen
<charles.allen@xxxxxxxx.invalid> wrote:

> OOME seems to be showing up in some of the Travis testing as well for group
> by related stuff. Unsure what's going on there.
>
> On Tue, Jul 24, 2018 at 9:46 PM Dongjin Lee <dongjin@xxxxxxxxxx> wrote:
>
> > After some experiments, I figured out the following:
> >
> > 1. Druid uses above 8gb of memory for testing. (building-druid.png)
> > 2. With 8gb(physical)+4gb(swap) of memory, the test succeeds regardless
> of
> > maven version (3.3.9, 3.5.2, 3.5.4) or MAVEN_OPTS. However, with
> > 8gb(physical)+2gb(swap) of memory[^1], some tests failed. The list of
> > failing tests differs between maven 3.3.9 and 3.5.2.
> >
> > In short, retaining sufficient memory solved the problem - *It seems like
> > 12gb of memory is a recommended setting for building druid.* (I guess
> > lots of you are working with the MacBook Pro with 16gb RAM, right? In
> that
> > case, you must not have encountered this problem.)
> >
> > If you are okay, may I update the building documentation for the newbies
> > like me?
> >
> > Thanks,
> > Dongjin
> >
> > +1. While building Druid, I found another problem. But this issue should
> > be discussed in another thread.
> >
> > [^1]: You know, the other processes also occupy the memory.
> >
> >
> > On Tue, Jul 24, 2018 at 3:07 AM Jihoon Son <jihoonson@xxxxxxxxxx> wrote:
> >
> >> I'm also using Maven 3.5.2 and not using any special configurations for
> >> Maven, but I have never seen that error too.
> >> Most of our Travis jobs have been working with only 512 MB of direct
> >> memory. Only the 'strict compilation' Travis job requires 3 GB of
> memory.
> >>
> >> I think it's worthwhile to look into this more. Maybe we somehow use
> more
> >> memory when we run all tests by 'mvn install'. Maybe this relates to the
> >> frequent transient failures of 'processing module test', one of our
> Travis
> >> jobs.
> >>
> >> Jihoon
> >>
> >> On Mon, Jul 23, 2018 at 9:32 AM Gian Merlino <gian@xxxxxxxxxx> wrote:
> >>
> >> > Interesting. Fwiw, I am using Maven 3.5.2 for building Druid and it
> has
> >> > been working for for me. I don't think I"m using any special Maven
> >> > overrides (at least, I don't see anything interesting in my ~/.m2
> >> directory
> >> > or in my environment variables). It might have to do with how much
> >> memory
> >> > our machines have? I do most of my builds on a Mac with 16GB RAM.
> Maybe
> >> try
> >> > checking .travis.yml in the druid repo. It sets -Xmx3000m for mvn
> >> install
> >> > commands, which might be needed for more low memory environments.
> >> >
> >> > $ mvn --version
> >> > Apache Maven 3.5.2 (138edd61fd100ec658bfa2d307c43b76940a5d7d;
> >> > 2017-10-18T00:58:13-07:00)
> >> > Maven home: /usr/local/Cellar/maven/3.5.2/libexec
> >> > Java version: 1.8.0_161, vendor: Oracle Corporation
> >> > Java home:
> >> > /Library/Java/JavaVirtualMachines/jdk1.8.0_161.jdk/Contents/Home/jre
> >> > Default locale: en_US, platform encoding: UTF-8
> >> > OS name: "mac os x", version: "10.13.5", arch: "x86_64", family: "mac"
> >> >
> >> > On Mon, Jul 23, 2018 at 6:40 AM Dongjin Lee <dongjin@xxxxxxxxxx>
> wrote:
> >> >
> >> > > Finally, it seems like I found the reason. It was a composition of
> >> > several
> >> > > problems:
> >> > >
> >> > > - Druid should not be built with maven 3.5.x. With 3.5.2, Test
> suites
> >> > like
> >> > > `GroupByQueryRunnerFailureTest` fails. After I switched into 3.3.9
> >> which
> >> > is
> >> > > built in the latest version of IntelliJ, those errors disappeared.
> It
> >> > seems
> >> > > like maven 3.5.x is not stable yet - it applied a drastic change,
> and
> >> it
> >> > is
> >> > > also why they skipped 3.4.x.
> >> > > - It seems like Druid requires some MaxDirectMemorySize
> configuration
> >> for
> >> > > some test suites. With some JVM parameter like
> >> > `-XX:MaxDirectMemorySize=4g`
> >> > > some test suites were passed, but not all. I am now trying the other
> >> > > options with enlarged swap space.
> >> > >
> >> > > Question: How much MaxDirectMemorySize configuration are you using?
> >> > >
> >> > > Best,
> >> > > Dongjin
> >> > >
> >> > > On Sat, Jul 21, 2018 at 3:01 AM Jihoon Son <ghoonson@xxxxxxxxx>
> >> wrote:
> >> > >
> >> > > > Hi Dongjin,
> >> > > >
> >> > > > that is weird. It looks like the vm crashed because of out of
> memory
> >> > > while
> >> > > > testing.
> >> > > > It might be a real issue or not.
> >> > > > Have you set any memory configuration for your maven?
> >> > > >
> >> > > > Jihoon
> >> > > >
> >> > > > On Thu, Jul 19, 2018 at 7:09 PM Dongjin Lee <dongjin@xxxxxxxxxx>
> >> > wrote:
> >> > > >
> >> > > > > Hi Jihoon,
> >> > > > >
> >> > > > > I ran `mvn clean package` following development/build
> >> > > > > <
> >> > > > >
> >> > > >
> >> > >
> >> >
> >>
> https://github.com/apache/incubator-druid/blob/master/docs/content/development/build.md
> >> > > > > >
> >> > > > > .
> >> > > > >
> >> > > > > Dongjin
> >> > > > >
> >> > > > > On Fri, Jul 20, 2018 at 12:30 AM Jihoon Son <
> jihoonson@xxxxxxxxxx
> >> >
> >> > > > wrote:
> >> > > > >
> >> > > > > > Hi Dongjin,
> >> > > > > >
> >> > > > > > what maven command did you run?
> >> > > > > >
> >> > > > > > Jihoon
> >> > > > > >
> >> > > > > > On Wed, Jul 18, 2018 at 10:38 PM Dongjin Lee <
> >> dongjin@xxxxxxxxxx>
> >> > > > wrote:
> >> > > > > >
> >> > > > > > > Hello. I am trying to build druid, but it fails. My
> >> environment
> >> > is
> >> > > > like
> >> > > > > > the
> >> > > > > > > following:
> >> > > > > > >
> >> > > > > > > - CPU: Intel(R) Core(TM) i7-7560U CPU @ 2.40GHz
> >> > > > > > > - RAM: 7704 MB
> >> > > > > > > - OS: ubuntu 18.04
> >> > > > > > > - JDK: openjdk version "1.8.0_171" (default configuration,
> >> with
> >> > > > > > MaxHeapSize
> >> > > > > > > = 1928 MB)
> >> > > > > > > - Branch: master (commit: cd8ea3d)
> >> > > > > > >
> >> > > > > > > The error message I got is:
> >> > > > > > >
> >> > > > > > > [INFO]
> >> > > > > > > >
> >> > > > > >
> >> > > >
> >> >
> ------------------------------------------------------------------------
> >> > > > > > > > [INFO] Reactor Summary:
> >> > > > > > > > [INFO]
> >> > > > > > > > [INFO] io.druid:druid
> .....................................
> >> > > > SUCCESS [
> >> > > > > > > > 50.258 s]
> >> > > > > > > > [INFO] java-util
> ..........................................
> >> > > SUCCESS
> >> > > > > > > [03:57
> >> > > > > > > > min]
> >> > > > > > > > [INFO] druid-api
> ..........................................
> >> > > > SUCCESS [
> >> > > > > > > > 22.694 s]
> >> > > > > > > > [INFO] druid-common
> .......................................
> >> > > > SUCCESS [
> >> > > > > > > > 14.083 s]
> >> > > > > > > > [INFO] druid-hll
> ..........................................
> >> > > > SUCCESS [
> >> > > > > > > > 17.126 s]
> >> > > > > > > > [INFO] extendedset
> ........................................
> >> > > > SUCCESS [
> >> > > > > > > > 10.856 s]
> >> > > > > > > >
> >> > > > > > > > *[INFO] druid-processing
> ...................................
> >> > > > FAILURE
> >> > > > > > > > [04:36 min]*[INFO] druid-aws-common
> >> > > > > ...................................
> >> > > > > > > > SKIPPED
> >> > > > > > > > [INFO] druid-server
> .......................................
> >> > > SKIPPED
> >> > > > > > > > [INFO] druid-examples
> .....................................
> >> > > SKIPPED
> >> > > > > > > > ...
> >> > > > > > > > [INFO]
> >> > > > > > > >
> >> > > > > >
> >> > > >
> >> >
> ------------------------------------------------------------------------
> >> > > > > > > > [INFO] BUILD FAILURE
> >> > > > > > > > [INFO]
> >> > > > > > > >
> >> > > > > >
> >> > > >
> >> >
> ------------------------------------------------------------------------
> >> > > > > > > > [INFO] Total time: 10:29 min
> >> > > > > > > > [INFO] Finished at: 2018-07-19T13:23:31+09:00
> >> > > > > > > > [INFO] Final Memory: 88M/777M
> >> > > > > > > > [INFO]
> >> > > > > > > >
> >> > > > > >
> >> > > >
> >> >
> ------------------------------------------------------------------------
> >> > > > > > > >
> >> > > > > > > > *[ERROR] Failed to execute goal
> >> > > > > > > > org.apache.maven.plugins:maven-surefire-plugin:2.19.1:test
> >> > > > > > (default-test)
> >> > > > > > > > on project druid-processing: Execution default-test of
> goal
> >> > > > > > > > org.apache.maven.plugins:maven-surefire-plugin:2.19.1:test
> >> > > failed:
> >> > > > > The
> >> > > > > > > > forked VM terminated without properly saying goodbye. VM
> >> crash
> >> > or
> >> > > > > > > > System.exit called?*[ERROR] Command was /bin/sh -c cd
> >> > > > > > > > /home/djlee/workspace/java/druid/processing &&
> >> > > > > > > > /usr/lib/jvm/java-8-openjdk-amd64/jre/bin/java -Xmx3000m
> >> > > > > > > -Duser.language=en
> >> > > > > > > > -Duser.country=US -Dfile.encoding=UTF-8
> -Duser.timezone=UTC
> >> > > > > > > >
> >> > > -Djava.util.logging.manager=org.apache.logging.log4j.jul.LogManager
> >> > > > > > > > -Ddruid.indexing.doubleStorage=double -jar
> >> > > > > > > >
> >> > > > > > >
> >> > > > > >
> >> > > > >
> >> > > >
> >> > >
> >> >
> >>
> /home/djlee/workspace/java/druid/processing/target/surefire/surefirebooter1075382243904099051.jar
> >> > > > > > > >
> >> > > > > > >
> >> > > > > >
> >> > > > >
> >> > > >
> >> > >
> >> >
> >>
> /home/djlee/workspace/java/druid/processing/target/surefire/surefire5444459351134757209tmp
> >> > > > > > > >
> >> > > > > > >
> >> > > > > >
> >> > > > >
> >> > > >
> >> > >
> >> >
> >>
> /home/djlee/workspace/java/druid/processing/target/surefire/surefire_5173894389718744688tmp
> >> > > > > > >
> >> > > > > > >
> >> > > > > > > It seems like it fails when it runs tests on
> >> `druid-processing`
> >> > > > module
> >> > > > > > but
> >> > > > > > > I can't certain. Is there anyone who can give me some hints?
> >> > Thanks
> >> > > > in
> >> > > > > > > advance.
> >> > > > > > >
> >> > > > > > > Best,
> >> > > > > > > Dongjin
> >> > > > > > >
> >> > > > > > > --
> >> > > > > > > *Dongjin Lee*
> >> > > > > > >
> >> > > > > > > *A hitchhiker in the mathematical world.*
> >> > > > > > >
> >>
> > > > > > > > *github:  <http://goog_969573159/
> >> <
> https://urldefense.proofpoint.com/v2/url?u=http-3A__goog-5F969573159_&d=DwMFaQ&c=ncDTmphkJTvjIDPh0hpF_w&r=HrLGT1qWNhseJBMYABL0GFSZESht5gBoLejor3SqMSo&m=thcH_tBRIMD3fQlav3ogOGyksut5JF_p3v21U7VlKyM&s=S55krYhaUwTC-7EGIgllQk3Jl9k8T3fZxfZAS_5nPA4&e=
> >
> >> >github.com/dongjinleekr
> >> > > > > > > <http://github.com/dongjinleekr>linkedin:
> >> > > > > > kr.linkedin.com/in/dongjinleekr
> >> <
> https://urldefense.proofpoint.com/v2/url?u=http-3A__kr.linkedin.com_in_dongjinleekr&d=DwMFaQ&c=ncDTmphkJTvjIDPh0hpF_w&r=HrLGT1qWNhseJBMYABL0GFSZESht5gBoLejor3SqMSo&m=thcH_tBRIMD3fQlav3ogOGyksut5JF_p3v21U7VlKyM&s=G-v0Qp9IHM8nieX8oUBvnQfbqWN_OC_MHzk52goEzqM&e=
> >
> >> > > > > > > <http://kr.linkedin.com/in/dongjinleekr
> >> <
> https://urldefense.proofpoint.com/v2/url?u=http-3A__kr.linkedin.com_in_dongjinleekr&d=DwMFaQ&c=ncDTmphkJTvjIDPh0hpF_w&r=HrLGT1qWNhseJBMYABL0GFSZESht5gBoLejor3SqMSo&m=thcH_tBRIMD3fQlav3ogOGyksut5JF_p3v21U7VlKyM&s=G-v0Qp9IHM8nieX8oUBvnQfbqWN_OC_MHzk52goEzqM&e=
> >
> >> >slideshare:
> >> > > > > > > www.slideshare.net/dongjinleekr
> >> <
> https://urldefense.proofpoint.com/v2/url?u=http-3A__www.slideshare.net_dongjinleekr&d=DwMFaQ&c=ncDTmphkJTvjIDPh0hpF_w&r=HrLGT1qWNhseJBMYABL0GFSZESht5gBoLejor3SqMSo&m=thcH_tBRIMD3fQlav3ogOGyksut5JF_p3v21U7VlKyM&s=SKr88hVIw32a2NGp3MDw-_EUvn0nFf_yISLrwD_ySYk&e=
> >
> >> > > > > > > <http://www.slideshare.net/dongjinleekr
> >> <
> https://urldefense.proofpoint.com/v2/url?u=http-3A__www.slideshare.net_dongjinleekr&d=DwMFaQ&c=ncDTmphkJTvjIDPh0hpF_w&r=HrLGT1qWNhseJBMYABL0GFSZESht5gBoLejor3SqMSo&m=thcH_tBRIMD3fQlav3ogOGyksut5JF_p3v21U7VlKyM&s=SKr88hVIw32a2NGp3MDw-_EUvn0nFf_yISLrwD_ySYk&e=
> >
> >> >*
> >
> >
> >> > > > > > >
> >> > > > > >
> >> > > > > > --
> >> > > > > > *Dongjin Lee*
> >> > > > > >
> >> > > > > > *A hitchhiker in the mathematical world.*
> >> > > > > >
> >>
> > > > > > > *github:  <http://goog_969573159/
> >> <
> https://urldefense.proofpoint.com/v2/url?u=http-3A__goog-5F969573159_&d=DwMFaQ&c=ncDTmphkJTvjIDPh0hpF_w&r=HrLGT1qWNhseJBMYABL0GFSZESht5gBoLejor3SqMSo&m=thcH_tBRIMD3fQlav3ogOGyksut5JF_p3v21U7VlKyM&s=S55krYhaUwTC-7EGIgllQk3Jl9k8T3fZxfZAS_5nPA4&e=
> >
> >> >github.com/dongjinleekr
> >> > > > > > <http://github.com/dongjinleekr>linkedin:
> >> > > > > kr.linkedin.com/in/dongjinleekr
> >> <
> https://urldefense.proofpoint.com/v2/url?u=http-3A__kr.linkedin.com_in_dongjinleekr&d=DwMFaQ&c=ncDTmphkJTvjIDPh0hpF_w&r=HrLGT1qWNhseJBMYABL0GFSZESht5gBoLejor3SqMSo&m=thcH_tBRIMD3fQlav3ogOGyksut5JF_p3v21U7VlKyM&s=G-v0Qp9IHM8nieX8oUBvnQfbqWN_OC_MHzk52goEzqM&e=
> >
> >> > > > > > <http://kr.linkedin.com/in/dongjinleekr
> >> <
> https://urldefense.proofpoint.com/v2/url?u=http-3A__kr.linkedin.com_in_dongjinleekr&d=DwMFaQ&c=ncDTmphkJTvjIDPh0hpF_w&r=HrLGT1qWNhseJBMYABL0GFSZESht5gBoLejor3SqMSo&m=thcH_tBRIMD3fQlav3ogOGyksut5JF_p3v21U7VlKyM&s=G-v0Qp9IHM8nieX8oUBvnQfbqWN_OC_MHzk52goEzqM&e=
> >
> >> >slideshare:
> >> > > > > www.slideshare.net/dongjinleekr
> >> <
> https://urldefense.proofpoint.com/v2/url?u=http-3A__www.slideshare.net_dongjinleekr&d=DwMFaQ&c=ncDTmphkJTvjIDPh0hpF_w&r=HrLGT1qWNhseJBMYABL0GFSZESht5gBoLejor3SqMSo&m=thcH_tBRIMD3fQlav3ogOGyksut5JF_p3v21U7VlKyM&s=SKr88hVIw32a2NGp3MDw-_EUvn0nFf_yISLrwD_ySYk&e=
> >
> >> > > > > > <http://www.slideshare.net/dongjinleekr
> >> <
> https://urldefense.proofpoint.com/v2/url?u=http-3A__www.slideshare.net_dongjinleekr&d=DwMFaQ&c=ncDTmphkJTvjIDPh0hpF_w&r=HrLGT1qWNhseJBMYABL0GFSZESht5gBoLejor3SqMSo&m=thcH_tBRIMD3fQlav3ogOGyksut5JF_p3v21U7VlKyM&s=SKr88hVIw32a2NGp3MDw-_EUvn0nFf_yISLrwD_ySYk&e=
> >
> >> >*
> >
> >
> >> > > > > >
> >> > > > >
> >> > > >
> >> > > > --
> >> > > > *Dongjin Lee*
> >> > > >
> >> > > > *A hitchhiker in the mathematical world.*
> >> > > >
> >>
> > > > > *github:  <http://goog_969573159/
> >> <
> https://urldefense.proofpoint.com/v2/url?u=http-3A__goog-5F969573159_&d=DwMFaQ&c=ncDTmphkJTvjIDPh0hpF_w&r=HrLGT1qWNhseJBMYABL0GFSZESht5gBoLejor3SqMSo&m=thcH_tBRIMD3fQlav3ogOGyksut5JF_p3v21U7VlKyM&s=S55krYhaUwTC-7EGIgllQk3Jl9k8T3fZxfZAS_5nPA4&e=
> >
> >> >github.com/dongjinleekr
> >> > > > <http://github.com/dongjinleekr>linkedin:
> >> > > kr.linkedin.com/in/dongjinleekr
> >> <
> https://urldefense.proofpoint.com/v2/url?u=http-3A__kr.linkedin.com_in_dongjinleekr&d=DwMFaQ&c=ncDTmphkJTvjIDPh0hpF_w&r=HrLGT1qWNhseJBMYABL0GFSZESht5gBoLejor3SqMSo&m=thcH_tBRIMD3fQlav3ogOGyksut5JF_p3v21U7VlKyM&s=G-v0Qp9IHM8nieX8oUBvnQfbqWN_OC_MHzk52goEzqM&e=
> >
> >> > > > <http://kr.linkedin.com/in/dongjinleekr
> >> <
> https://urldefense.proofpoint.com/v2/url?u=http-3A__kr.linkedin.com_in_dongjinleekr&d=DwMFaQ&c=ncDTmphkJTvjIDPh0hpF_w&r=HrLGT1qWNhseJBMYABL0GFSZESht5gBoLejor3SqMSo&m=thcH_tBRIMD3fQlav3ogOGyksut5JF_p3v21U7VlKyM&s=G-v0Qp9IHM8nieX8oUBvnQfbqWN_OC_MHzk52goEzqM&e=
> >
> >> >slideshare:
> >> > > www.slideshare.net/dongjinleekr
> >> <
> https://urldefense.proofpoint.com/v2/url?u=http-3A__www.slideshare.net_dongjinleekr&d=DwMFaQ&c=ncDTmphkJTvjIDPh0hpF_w&r=HrLGT1qWNhseJBMYABL0GFSZESht5gBoLejor3SqMSo&m=thcH_tBRIMD3fQlav3ogOGyksut5JF_p3v21U7VlKyM&s=SKr88hVIw32a2NGp3MDw-_EUvn0nFf_yISLrwD_ySYk&e=
> >
> >> > > > <http://www.slideshare.net/dongjinleekr
> >> <
> https://urldefense.proofpoint.com/v2/url?u=http-3A__www.slideshare.net_dongjinleekr&d=DwMFaQ&c=ncDTmphkJTvjIDPh0hpF_w&r=HrLGT1qWNhseJBMYABL0GFSZESht5gBoLejor3SqMSo&m=thcH_tBRIMD3fQlav3ogOGyksut5JF_p3v21U7VlKyM&s=SKr88hVIw32a2NGp3MDw-_EUvn0nFf_yISLrwD_ySYk&e=
> >
> >> >*
> >> > > >
> >> > >
> >> >
> >>
> >> --
> >> *Dongjin Lee*
> >>
> >> *A hitchhiker in the mathematical world.*
> >>
> >> *github:
> >> <
> https://urldefense.proofpoint.com/v2/url?u=http-3A__goog-5F969573159_&d=DwMFaQ&c=ncDTmphkJTvjIDPh0hpF_w&r=HrLGT1qWNhseJBMYABL0GFSZESht5gBoLejor3SqMSo&m=thcH_tBRIMD3fQlav3ogOGyksut5JF_p3v21U7VlKyM&s=S55krYhaUwTC-7EGIgllQk3Jl9k8T3fZxfZAS_5nPA4&e=
> >github.com/dongjinleekr
> >> <http://github.com/dongjinleekr>linkedin:
> kr.linkedin.com/in/dongjinleekr
> >> <
> https://urldefense.proofpoint.com/v2/url?u=http-3A__kr.linkedin.com_in_dongjinleekr&d=DwMFaQ&c=ncDTmphkJTvjIDPh0hpF_w&r=HrLGT1qWNhseJBMYABL0GFSZESht5gBoLejor3SqMSo&m=thcH_tBRIMD3fQlav3ogOGyksut5JF_p3v21U7VlKyM&s=G-v0Qp9IHM8nieX8oUBvnQfbqWN_OC_MHzk52goEzqM&e=>slideshare:
> www.slideshare.net/dongjinleekr
> >> <
> https://urldefense.proofpoint.com/v2/url?u=http-3A__www.slideshare.net_dongjinleekr&d=DwMFaQ&c=ncDTmphkJTvjIDPh0hpF_w&r=HrLGT1qWNhseJBMYABL0GFSZESht5gBoLejor3SqMSo&m=thcH_tBRIMD3fQlav3ogOGyksut5JF_p3v21U7VlKyM&s=SKr88hVIw32a2NGp3MDw-_EUvn0nFf_yISLrwD_ySYk&e=
> >*
> >>
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: dev-unsubscribe@xxxxxxxxxxxxxxxx
> > For additional commands, e-mail: dev-help@xxxxxxxxxxxxxxxx
>