[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: Invite to comment on the @RequiresStableInput design doc

One issue: we definitely have some strong use cases where we want this on ProcessTimer but not on ProcessElement. Since both are on the same DoFn, I'm not sure how you would represent this as a separate transform.

On Mon, Jul 2, 2018 at 5:05 PM Robert Bradshaw <robertwb@xxxxxxxxxx> wrote:
Thanks for the writeup. 

I'm wondering with, rather than phrasing this as an annotation on DoFn methods that gets plumbed down through the portability representation, if it would make more sense to introduce a new, primitive "EnsureStableInput" transform. For those runners whose reshuffle provide stable inputs, they could use that as an implementation, and other runners could provide other suitable implementations. 

On Mon, Jul 2, 2018 at 3:26 PM Robin Qiu <robinyq@xxxxxxxxxx> wrote:
Hi everyone,

Thanks for your feedback on the doc. I have revamped it according to all of the comments. The major changes I have made are:
* The problem description should be more general and accurate now.
* I added more background information, such as details about Reshuffle, so I should be easier to understand now.
* I made it clear what is the scope of my current project and what could be left to future work.
* It now reflects the current progress of my work, and discusses how it should work with the portable pipeline representation (WIP)

Also, I forgot to mention last time that this doc may be interesting to those of you interested in Reshuffle, because Reshuffle is used as a current workaround for the problem described in the doc.

More comments are always welcome.


On Fri, Jun 15, 2018 at 7:34 AM Kenneth Knowles <klk@xxxxxxxxxx> wrote:
Thanks for the write up. It is great to see someone pushing this through.

I wanted to bring Luke's high-level question back to the list for visibility: what about portability and other SDKs?

Portability is probably trivial, but the "other SDKs" question is probably best answered by folks working on them who can have opinions about how it works in their SDKs idioms.