[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: Problem on druid-s3-extensions


IMHO the best way to do it would be to setup the test so that it sets the
region just for the test, and let production systems use whichever region
provider fits the way they typically do region configs. That way people
running the tests don't have to worry about anything regarding how to setup
a region, and people deploying cloud services don't have surprise behavior
for Druid compared to other usages of the java sdk.

On Tue, Jul 24, 2018 at 9:48 PM Dongjin Lee <dongjin@xxxxxxxxxx> wrote:

> Hello. I encountered a problem building druid. *In short,
> `TestAWSCredentialsProvider` fails like the following*:
>
> ```
> mvn -pl extensions-core/s3-extensions test
>
> ...
>
>
> testWithFileSessionCredentials(io.druid.storage.s3.TestAWSCredentialsProvider)
> Time elapsed: 6.615 sec  <<< ERROR!
> com.amazonaws.SdkClientException: Unable to find a region via the region
> provider chain. Must provide an explicit region in the builder or setup
> environment to supply a region.
> at
>
> io.druid.storage.s3.TestAWSCredentialsProvider.testWithFileSessionCredentials(TestAWSCredentialsProvider.java:98)
>
> testWithFixedAWSKeys(io.druid.storage.s3.TestAWSCredentialsProvider)  Time
> elapsed: 4.345 sec  <<< ERROR!
> com.amazonaws.SdkClientException: Unable to find a region via the region
> provider chain. Must provide an explicit region in the builder or setup
> environment to supply a region.
> at
>
> io.druid.storage.s3.TestAWSCredentialsProvider.testWithFixedAWSKeys(TestAWSCredentialsProvider.java:67)
> ```
>
> After digging the code, I found following:
>
> 1. `S3StorageDruidModule` does not provide a way to explictily set
> `Region`, but detects the region automatically with its default
> credential/region provider chain. `S3StorageDruidModule#getAmazonS3Client
> -> AmazonS3Client#builder -> AmazonS3ClientBuilder#standard;`
> 2. The default region provider chain[^1][^2] tries to determine region in
> following orders:[^1][^2]
> a. Environment Variable[^5]
> b. JVM property[^6]
> c. AWS Configuration[^7]
> d. EC2 instance metadata service[^8]
>
> If all of the above fails, it throws an Exception.
>
> *In short, there is a possibility that `S3StorageDruidModule` can't
> determine the Region. It would be better to provide a way to set Region
> explicitly.*
>
> How do you think?
>
> Best,
> Dongjin
>
> [^1]:
>
> https://github.com/aws/aws-sdk-java/blob/master/aws-java-sdk-core/src/main/java/com/amazonaws/regions/DefaultAwsRegionProviderChain.java
> [^2]:
>
> https://github.com/aws/aws-sdk-java/blob/master/aws-java-sdk-core/src/main/java/com/amazonaws/regions/AwsRegionProviderChain.java
> [^3]:
>
> https://urldefense.proofpoint.com/v2/url?u=https-3A__docs.aws.amazon.com_sdk-2Dfor-2Djava_v2_developer-2Dguide_java-2Ddg-2Dregion-2Dselection.html&d=DwIBaQ&c=ncDTmphkJTvjIDPh0hpF_w&r=HrLGT1qWNhseJBMYABL0GFSZESht5gBoLejor3SqMSo&m=KVjSJAnGDSKgdNTP-vCFDAUpnrBBYAiix7JzjxvHuwI&s=Ro8j9Jj9lCbFbc0aYDNjwtZuD2a_9fPaYw1lOp7ZQeQ&e=
> [^4]: The example client in documentation is `AmazonEC2ClientBuilder` but
> it also applies to the other clients like `AmazonS3ClientBuilder`.
> [^5]:
>
> https://github.com/aws/aws-sdk-java/blob/master/aws-java-sdk-core/src/main/java/com/amazonaws/regions/AwsEnvVarOverrideRegionProvider.java
> [^6]:
>
> https://github.com/aws/aws-sdk-java/blob/master/aws-java-sdk-core/src/main/java/com/amazonaws/regions/AwsSystemPropertyRegionProvider.java
> [^7]:
>
> https://github.com/aws/aws-sdk-java/blob/master/aws-java-sdk-core/src/main/java/com/amazonaws/regions/AwsProfileRegionProvider.java
> [^8]:
>
> https://github.com/aws/aws-sdk-java/blob/master/aws-java-sdk-core/src/main/java/com/amazonaws/regions/InstanceMetadataRegionProvider.java
>
> --
> *Dongjin Lee*
>
> *A hitchhiker in the mathematical world.*
>
> *github:  <
> https://urldefense.proofpoint.com/v2/url?u=http-3A__goog-5F969573159_&d=DwIBaQ&c=ncDTmphkJTvjIDPh0hpF_w&r=HrLGT1qWNhseJBMYABL0GFSZESht5gBoLejor3SqMSo&m=KVjSJAnGDSKgdNTP-vCFDAUpnrBBYAiix7JzjxvHuwI&s=VLpmOvpUoEqDfaI8jAV3WKsttHGasxSzCwmAE-we-CU&e=
> >github.com/dongjinleekr
> <http://github.com/dongjinleekr>linkedin: kr.linkedin.com/in/dongjinleekr
> <
> https://urldefense.proofpoint.com/v2/url?u=http-3A__kr.linkedin.com_in_dongjinleekr&d=DwIBaQ&c=ncDTmphkJTvjIDPh0hpF_w&r=HrLGT1qWNhseJBMYABL0GFSZESht5gBoLejor3SqMSo&m=KVjSJAnGDSKgdNTP-vCFDAUpnrBBYAiix7JzjxvHuwI&s=LnjNxyq941KadHM075fCkotIkN-U4USFaqm2JsKi08Q&e=
> >slideshare:
> www.slideshare.net/dongjinleekr
> <
> https://urldefense.proofpoint.com/v2/url?u=http-3A__www.slideshare.net_dongjinleekr&d=DwIBaQ&c=ncDTmphkJTvjIDPh0hpF_w&r=HrLGT1qWNhseJBMYABL0GFSZESht5gBoLejor3SqMSo&m=KVjSJAnGDSKgdNTP-vCFDAUpnrBBYAiix7JzjxvHuwI&s=hoP7U3GI-oMiDEdyIocsIkBAoupeT_QQ52PBHlEeoUQ&e=
> >*
>