[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Problem on druid-s3-extensions


Hello. I encountered a problem building druid. *In short,
`TestAWSCredentialsProvider` fails like the following*:

```
mvn -pl extensions-core/s3-extensions test

...

testWithFileSessionCredentials(io.druid.storage.s3.TestAWSCredentialsProvider)
Time elapsed: 6.615 sec  <<< ERROR!
com.amazonaws.SdkClientException: Unable to find a region via the region
provider chain. Must provide an explicit region in the builder or setup
environment to supply a region.
at
io.druid.storage.s3.TestAWSCredentialsProvider.testWithFileSessionCredentials(TestAWSCredentialsProvider.java:98)

testWithFixedAWSKeys(io.druid.storage.s3.TestAWSCredentialsProvider)  Time
elapsed: 4.345 sec  <<< ERROR!
com.amazonaws.SdkClientException: Unable to find a region via the region
provider chain. Must provide an explicit region in the builder or setup
environment to supply a region.
at
io.druid.storage.s3.TestAWSCredentialsProvider.testWithFixedAWSKeys(TestAWSCredentialsProvider.java:67)
```

After digging the code, I found following:

1. `S3StorageDruidModule` does not provide a way to explictily set
`Region`, but detects the region automatically with its default
credential/region provider chain. `S3StorageDruidModule#getAmazonS3Client
-> AmazonS3Client#builder -> AmazonS3ClientBuilder#standard;`
2. The default region provider chain[^1][^2] tries to determine region in
following orders:[^1][^2]
a. Environment Variable[^5]
b. JVM property[^6]
c. AWS Configuration[^7]
d. EC2 instance metadata service[^8]

If all of the above fails, it throws an Exception.

*In short, there is a possibility that `S3StorageDruidModule` can't
determine the Region. It would be better to provide a way to set Region
explicitly.*

How do you think?

Best,
Dongjin

[^1]:
https://github.com/aws/aws-sdk-java/blob/master/aws-java-sdk-core/src/main/java/com/amazonaws/regions/DefaultAwsRegionProviderChain.java
[^2]:
https://github.com/aws/aws-sdk-java/blob/master/aws-java-sdk-core/src/main/java/com/amazonaws/regions/AwsRegionProviderChain.java
[^3]:
https://docs.aws.amazon.com/sdk-for-java/v2/developer-guide/java-dg-region-selection.html
[^4]: The example client in documentation is `AmazonEC2ClientBuilder` but
it also applies to the other clients like `AmazonS3ClientBuilder`.
[^5]:
https://github.com/aws/aws-sdk-java/blob/master/aws-java-sdk-core/src/main/java/com/amazonaws/regions/AwsEnvVarOverrideRegionProvider.java
[^6]:
https://github.com/aws/aws-sdk-java/blob/master/aws-java-sdk-core/src/main/java/com/amazonaws/regions/AwsSystemPropertyRegionProvider.java
[^7]:
https://github.com/aws/aws-sdk-java/blob/master/aws-java-sdk-core/src/main/java/com/amazonaws/regions/AwsProfileRegionProvider.java
[^8]:
https://github.com/aws/aws-sdk-java/blob/master/aws-java-sdk-core/src/main/java/com/amazonaws/regions/InstanceMetadataRegionProvider.java

-- 
*Dongjin Lee*

*A hitchhiker in the mathematical world.*

*github:  <http://goog_969573159/>github.com/dongjinleekr
<http://github.com/dongjinleekr>linkedin: kr.linkedin.com/in/dongjinleekr
<http://kr.linkedin.com/in/dongjinleekr>slideshare:
www.slideshare.net/dongjinleekr
<http://www.slideshare.net/dongjinleekr>*