[DISCUSS] -- Need to be able to access multiple s3 buckets with different authentication from within a single job
Currently there doesn’t seem to be a way to do this. Am I correct in that? I guess one could register multiple implementations each with their own scheme but that seems somewhat hacky.
It would be nice if the registration of the filesystem was done when the DataSet (or DataStream) was defined and not have them preregistered in the system. I’ve looked at the code and it doesn’t seem too terribly difficult.
I was wondering if others have a need for this as well?