[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Recommendation for BucketingSink to GZIP compress files to S3

Hi, what is the recommended method for using BucketingSink and compressing files using GZIP before it is uploaded to S3?

I read that one way is to extend the StreamWriterBase class and wrap the stream using GZIPOutputStream. Is there an Flink example for this? If so, what would be the proper way to override the methods (open(), write(), close(), flush())?

Thank you.