osdir.com

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: Dynamic output - Python


Hi Eila,

Python textio currently does not support dynamic writes. So, to achieve this, you'll have to implement a custom DoFn that writes data to dynamic destinations. 

Thanks,
Cham

On Mon, Mar 12, 2018 at 6:26 AM OrielResearch Eila Arich-Landkof <eila@xxxxxxxxxxxxxxxxx> wrote:
Hello all,

I would like to print each PCollection element to a different folder on gs://.../
the folder name is not available in advance, but it is available on the pCollection data (or can be calculated prior to printing).
My question are:
- Is it possible to use the folder path as a parameter to the beam.io.WriteToText?
- Can you refer me to a code example where dynamic output (or similar) is being used in Python?
Thanks,
Eila