OSDir


[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: Problem on druid-s3-extensions


I greatly appreciate your guidance. Now all the tests pass well!

Thanks,
Dongjin


On Thu, Jul 26, 2018 at 3:27 AM Jihoon Son <ghoonson@xxxxxxxxx> wrote:

> Hi Dongjin,
>
> as Charles said, you should set the region for aws sdk. Here is a snippet
> of our Travis configuration.
>
> >       # other modules test
> >    - sudo: false
> >      env:
> >        - NAME="other modules test"
> >        - AWS_REGION=us-east-1 # set a aws region for unit tests
> >      install: echo "MAVEN_OPTS='-Xmx3000m'" > ~/.mavenrc && mvn install
> -q -ff -DskipTests -B
> >      before_script:
> >        - unset _JAVA_OPTIONS
> >      script: echo "MAVEN_OPTS='-Xmx512m'" > ~/.mavenrc && mvn test -B
> -Pparallel-test -Dmaven.fork.count=2 -pl '!processing,!server'
>
> Jihoon
>
> On Wed, Jul 25, 2018 at 8:29 AM Charles Allen
> <charles.allen@xxxxxxxx.invalid> wrote:
>
> > IMHO the best way to do it would be to setup the test so that it sets the
> > region just for the test, and let production systems use whichever region
> > provider fits the way they typically do region configs. That way people
> > running the tests don't have to worry about anything regarding how to
> setup
> > a region, and people deploying cloud services don't have surprise
> behavior
> > for Druid compared to other usages of the java sdk.
> >
> > On Tue, Jul 24, 2018 at 9:48 PM Dongjin Lee <dongjin@xxxxxxxxxx> wrote:
> >
> > > Hello. I encountered a problem building druid. *In short,
> > > `TestAWSCredentialsProvider` fails like the following*:
> > >
> > > ```
> > > mvn -pl extensions-core/s3-extensions test
> > >
> > > ...
> > >
> > >
> > >
> >
> testWithFileSessionCredentials(io.druid.storage.s3.TestAWSCredentialsProvider)
> > > Time elapsed: 6.615 sec  <<< ERROR!
> > > com.amazonaws.SdkClientException: Unable to find a region via the
> region
> > > provider chain. Must provide an explicit region in the builder or setup
> > > environment to supply a region.
> > > at
> > >
> > >
> >
> io.druid.storage.s3.TestAWSCredentialsProvider.testWithFileSessionCredentials(TestAWSCredentialsProvider.java:98)
> > >
> > > testWithFixedAWSKeys(io.druid.storage.s3.TestAWSCredentialsProvider)
> > Time
> > > elapsed: 4.345 sec  <<< ERROR!
> > > com.amazonaws.SdkClientException: Unable to find a region via the
> region
> > > provider chain. Must provide an explicit region in the builder or setup
> > > environment to supply a region.
> > > at
> > >
> > >
> >
> io.druid.storage.s3.TestAWSCredentialsProvider.testWithFixedAWSKeys(TestAWSCredentialsProvider.java:67)
> > > ```
> > >
> > > After digging the code, I found following:
> > >
> > > 1. `S3StorageDruidModule` does not provide a way to explictily set
> > > `Region`, but detects the region automatically with its default
> > > credential/region provider chain.
> `S3StorageDruidModule#getAmazonS3Client
> > > -> AmazonS3Client#builder -> AmazonS3ClientBuilder#standard;`
> > > 2. The default region provider chain[^1][^2] tries to determine region
> in
> > > following orders:[^1][^2]
> > > a. Environment Variable[^5]
> > > b. JVM property[^6]
> > > c. AWS Configuration[^7]
> > > d. EC2 instance metadata service[^8]
> > >
> > > If all of the above fails, it throws an Exception.
> > >
> > > *In short, there is a possibility that `S3StorageDruidModule` can't
> > > determine the Region. It would be better to provide a way to set Region
> > > explicitly.*
> > >
> > > How do you think?
> > >
> > > Best,
> > > Dongjin
> > >
> > > [^1]:
> > >
> > >
> >
> https://github.com/aws/aws-sdk-java/blob/master/aws-java-sdk-core/src/main/java/com/amazonaws/regions/DefaultAwsRegionProviderChain.java
> > > [^2]:
> > >
> > >
> >
> https://github.com/aws/aws-sdk-java/blob/master/aws-java-sdk-core/src/main/java/com/amazonaws/regions/AwsRegionProviderChain.java
> > > [^3]:
> > >
> > >
> >
> https://urldefense.proofpoint.com/v2/url?u=https-3A__docs.aws.amazon.com_sdk-2Dfor-2Djava_v2_developer-2Dguide_java-2Ddg-2Dregion-2Dselection.html&d=DwIBaQ&c=ncDTmphkJTvjIDPh0hpF_w&r=HrLGT1qWNhseJBMYABL0GFSZESht5gBoLejor3SqMSo&m=KVjSJAnGDSKgdNTP-vCFDAUpnrBBYAiix7JzjxvHuwI&s=Ro8j9Jj9lCbFbc0aYDNjwtZuD2a_9fPaYw1lOp7ZQeQ&e=
> > > [^4]: The example client in documentation is `AmazonEC2ClientBuilder`
> but
> > > it also applies to the other clients like `AmazonS3ClientBuilder`.
> > > [^5]:
> > >
> > >
> >
> https://github.com/aws/aws-sdk-java/blob/master/aws-java-sdk-core/src/main/java/com/amazonaws/regions/AwsEnvVarOverrideRegionProvider.java
> > > [^6]:
> > >
> > >
> >
> https://github.com/aws/aws-sdk-java/blob/master/aws-java-sdk-core/src/main/java/com/amazonaws/regions/AwsSystemPropertyRegionProvider.java
> > > [^7]:
> > >
> > >
> >
> https://github.com/aws/aws-sdk-java/blob/master/aws-java-sdk-core/src/main/java/com/amazonaws/regions/AwsProfileRegionProvider.java
> > > [^8]:
> > >
> > >
> >
> https://github.com/aws/aws-sdk-java/blob/master/aws-java-sdk-core/src/main/java/com/amazonaws/regions/InstanceMetadataRegionProvider.java
> > >
> > > --
> > > *Dongjin Lee*
> > >
> > > *A hitchhiker in the mathematical world.*
> > >
> > > *github:  <
> > >
> >
> https://urldefense.proofpoint.com/v2/url?u=http-3A__goog-5F969573159_&d=DwIBaQ&c=ncDTmphkJTvjIDPh0hpF_w&r=HrLGT1qWNhseJBMYABL0GFSZESht5gBoLejor3SqMSo&m=KVjSJAnGDSKgdNTP-vCFDAUpnrBBYAiix7JzjxvHuwI&s=VLpmOvpUoEqDfaI8jAV3WKsttHGasxSzCwmAE-we-CU&e=
> > > >github.com/dongjinleekr
> > > <http://github.com/dongjinleekr>linkedin:
> > kr.linkedin.com/in/dongjinleekr
> > > <
> > >
> >
> https://urldefense.proofpoint.com/v2/url?u=http-3A__kr.linkedin.com_in_dongjinleekr&d=DwIBaQ&c=ncDTmphkJTvjIDPh0hpF_w&r=HrLGT1qWNhseJBMYABL0GFSZESht5gBoLejor3SqMSo&m=KVjSJAnGDSKgdNTP-vCFDAUpnrBBYAiix7JzjxvHuwI&s=LnjNxyq941KadHM075fCkotIkN-U4USFaqm2JsKi08Q&e=
> > > >slideshare:
> > > www.slideshare.net/dongjinleekr
> > > <
> > >
> >
> https://urldefense.proofpoint.com/v2/url?u=http-3A__www.slideshare.net_dongjinleekr&d=DwIBaQ&c=ncDTmphkJTvjIDPh0hpF_w&r=HrLGT1qWNhseJBMYABL0GFSZESht5gBoLejor3SqMSo&m=KVjSJAnGDSKgdNTP-vCFDAUpnrBBYAiix7JzjxvHuwI&s=hoP7U3GI-oMiDEdyIocsIkBAoupeT_QQ52PBHlEeoUQ&e=
> > > >*
> > >
> >
>
> --
> *Dongjin Lee*
>
> *A hitchhiker in the mathematical world.*
>
> *github:  <http://goog_969573159/>github.com/dongjinleekr
> <http://github.com/dongjinleekr>linkedin: kr.linkedin.com/in/dongjinleekr
> <http://kr.linkedin.com/in/dongjinleekr>slideshare: www.slideshare.net/dongjinleekr
> <http://www.slideshare.net/dongjinleekr>*
>