osdir.com

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: Should Beam Python throw an error if DoFn returns a string?


Thanks Robert. You raise a good point that this code is performance-critical. 

If the check can be fast, then its worth having. 

> Should we also let Beam error out if users return a string?
> e.g. consider the following pipeline:
> p | Create(['abc']) | ParDo(lambda x: x) | WriteToFile('myfile')
>
> This pipeline would write three separate elements. Is this not a bit awkward?

One can't do ParDo with lambdas (I think, that was the intent). [...]

I tried this recently for someone. That pipeline does work.
 
Best
-P.
--
Got feedback? go/pabloem-feedback