OSDir


[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: Problem on druid-s3-extensions


Hi Dongjin,

as Charles said, you should set the region for aws sdk. Here is a snippet
of our Travis configuration.

>       # other modules test
>    - sudo: false
>      env:
>        - NAME="other modules test"
>        - AWS_REGION=us-east-1 # set a aws region for unit tests
>      install: echo "MAVEN_OPTS='-Xmx3000m'" > ~/.mavenrc && mvn install
-q -ff -DskipTests -B
>      before_script:
>        - unset _JAVA_OPTIONS
>      script: echo "MAVEN_OPTS='-Xmx512m'" > ~/.mavenrc && mvn test -B
-Pparallel-test -Dmaven.fork.count=2 -pl '!processing,!server'

Jihoon

On Wed, Jul 25, 2018 at 8:29 AM Charles Allen
<charles.allen@xxxxxxxx.invalid> wrote:

> IMHO the best way to do it would be to setup the test so that it sets the
> region just for the test, and let production systems use whichever region
> provider fits the way they typically do region configs. That way people
> running the tests don't have to worry about anything regarding how to setup
> a region, and people deploying cloud services don't have surprise behavior
> for Druid compared to other usages of the java sdk.
>
> On Tue, Jul 24, 2018 at 9:48 PM Dongjin Lee <dongjin@xxxxxxxxxx> wrote:
>
> > Hello. I encountered a problem building druid. *In short,
> > `TestAWSCredentialsProvider` fails like the following*:
> >
> > ```
> > mvn -pl extensions-core/s3-extensions test
> >
> > ...
> >
> >
> >
> testWithFileSessionCredentials(io.druid.storage.s3.TestAWSCredentialsProvider)
> > Time elapsed: 6.615 sec  <<< ERROR!
> > com.amazonaws.SdkClientException: Unable to find a region via the region
> > provider chain. Must provide an explicit region in the builder or setup
> > environment to supply a region.
> > at
> >
> >
> io.druid.storage.s3.TestAWSCredentialsProvider.testWithFileSessionCredentials(TestAWSCredentialsProvider.java:98)
> >
> > testWithFixedAWSKeys(io.druid.storage.s3.TestAWSCredentialsProvider)
> Time
> > elapsed: 4.345 sec  <<< ERROR!
> > com.amazonaws.SdkClientException: Unable to find a region via the region
> > provider chain. Must provide an explicit region in the builder or setup
> > environment to supply a region.
> > at
> >
> >
> io.druid.storage.s3.TestAWSCredentialsProvider.testWithFixedAWSKeys(TestAWSCredentialsProvider.java:67)
> > ```
> >
> > After digging the code, I found following:
> >
> > 1. `S3StorageDruidModule` does not provide a way to explictily set
> > `Region`, but detects the region automatically with its default
> > credential/region provider chain. `S3StorageDruidModule#getAmazonS3Client
> > -> AmazonS3Client#builder -> AmazonS3ClientBuilder#standard;`
> > 2. The default region provider chain[^1][^2] tries to determine region in
> > following orders:[^1][^2]
> > a. Environment Variable[^5]
> > b. JVM property[^6]
> > c. AWS Configuration[^7]
> > d. EC2 instance metadata service[^8]
> >
> > If all of the above fails, it throws an Exception.
> >
> > *In short, there is a possibility that `S3StorageDruidModule` can't
> > determine the Region. It would be better to provide a way to set Region
> > explicitly.*
> >
> > How do you think?
> >
> > Best,
> > Dongjin
> >
> > [^1]:
> >
> >
> https://github.com/aws/aws-sdk-java/blob/master/aws-java-sdk-core/src/main/java/com/amazonaws/regions/DefaultAwsRegionProviderChain.java
> > [^2]:
> >
> >
> https://github.com/aws/aws-sdk-java/blob/master/aws-java-sdk-core/src/main/java/com/amazonaws/regions/AwsRegionProviderChain.java
> > [^3]:
> >
> >
> https://urldefense.proofpoint.com/v2/url?u=https-3A__docs.aws.amazon.com_sdk-2Dfor-2Djava_v2_developer-2Dguide_java-2Ddg-2Dregion-2Dselection.html&d=DwIBaQ&c=ncDTmphkJTvjIDPh0hpF_w&r=HrLGT1qWNhseJBMYABL0GFSZESht5gBoLejor3SqMSo&m=KVjSJAnGDSKgdNTP-vCFDAUpnrBBYAiix7JzjxvHuwI&s=Ro8j9Jj9lCbFbc0aYDNjwtZuD2a_9fPaYw1lOp7ZQeQ&e=
> > [^4]: The example client in documentation is `AmazonEC2ClientBuilder` but
> > it also applies to the other clients like `AmazonS3ClientBuilder`.
> > [^5]:
> >
> >
> https://github.com/aws/aws-sdk-java/blob/master/aws-java-sdk-core/src/main/java/com/amazonaws/regions/AwsEnvVarOverrideRegionProvider.java
> > [^6]:
> >
> >
> https://github.com/aws/aws-sdk-java/blob/master/aws-java-sdk-core/src/main/java/com/amazonaws/regions/AwsSystemPropertyRegionProvider.java
> > [^7]:
> >
> >
> https://github.com/aws/aws-sdk-java/blob/master/aws-java-sdk-core/src/main/java/com/amazonaws/regions/AwsProfileRegionProvider.java
> > [^8]:
> >
> >
> https://github.com/aws/aws-sdk-java/blob/master/aws-java-sdk-core/src/main/java/com/amazonaws/regions/InstanceMetadataRegionProvider.java
> >
> > --
> > *Dongjin Lee*
> >
> > *A hitchhiker in the mathematical world.*
> >
> > *github:  <
> >
> https://urldefense.proofpoint.com/v2/url?u=http-3A__goog-5F969573159_&d=DwIBaQ&c=ncDTmphkJTvjIDPh0hpF_w&r=HrLGT1qWNhseJBMYABL0GFSZESht5gBoLejor3SqMSo&m=KVjSJAnGDSKgdNTP-vCFDAUpnrBBYAiix7JzjxvHuwI&s=VLpmOvpUoEqDfaI8jAV3WKsttHGasxSzCwmAE-we-CU&e=
> > >github.com/dongjinleekr
> > <http://github.com/dongjinleekr>linkedin:
> kr.linkedin.com/in/dongjinleekr
> > <
> >
> https://urldefense.proofpoint.com/v2/url?u=http-3A__kr.linkedin.com_in_dongjinleekr&d=DwIBaQ&c=ncDTmphkJTvjIDPh0hpF_w&r=HrLGT1qWNhseJBMYABL0GFSZESht5gBoLejor3SqMSo&m=KVjSJAnGDSKgdNTP-vCFDAUpnrBBYAiix7JzjxvHuwI&s=LnjNxyq941KadHM075fCkotIkN-U4USFaqm2JsKi08Q&e=
> > >slideshare:
> > www.slideshare.net/dongjinleekr
> > <
> >
> https://urldefense.proofpoint.com/v2/url?u=http-3A__www.slideshare.net_dongjinleekr&d=DwIBaQ&c=ncDTmphkJTvjIDPh0hpF_w&r=HrLGT1qWNhseJBMYABL0GFSZESht5gBoLejor3SqMSo&m=KVjSJAnGDSKgdNTP-vCFDAUpnrBBYAiix7JzjxvHuwI&s=hoP7U3GI-oMiDEdyIocsIkBAoupeT_QQ52PBHlEeoUQ&e=
> > >*
> >
>