osdir.com

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: Proposal for Beam Python User State and Timer APIs


An update: https://github.com/apache/beam/pull/5691 has been merged.  I hope to send out a reference implementation in the DirectRunner soon.  On the roadmap after that is work on the relevant portability interfaces here so we can get this working on runners like Beam Python on Flink.

On Wed, Jun 20, 2018 at 10:00 AM Charles Chen <ccy@xxxxxxxxxx> wrote:
An update on the implementation: I recently sent out the user-facing pipeline construction part of the API implementation out for review: https://github.com/apache/beam/pull/5691.

On Tue, Jun 5, 2018 at 5:26 PM Charles Chen <ccy@xxxxxxxxxx> wrote:
Thanks everyone for contributing here.  We've reached rough consensus on the approach we should take with this API, and I've summarized this in the new "Community consensus" sections I added to the doc (https://s.apache.org/beam-python-user-state-and-timers).  I will begin initial implementation of this API soon.

On Wed, May 23, 2018 at 8:08 PM Thomas Weise <thw@xxxxxxxxxx> wrote:
Nice proposal; it's exciting to see this about to be added to the SDK as it enables a set of more complex use cases.

I also think that some of the content can later be repurposed as user documentation.

Thanks,
Thomas


On Wed, May 23, 2018 at 11:49 AM, Charles Chen <ccy@xxxxxxxxxx> wrote:
Thanks everyone for the detailed comments and discussions.  It looks like by now, we mostly agree with the requirements and overall direction needed for the API, though there is continuing discussion on specific details.  I want to highlight two new sections of the doc, which address some discussions that have come up:
Let me know any further comments and suggestions.

On Tue, May 22, 2018 at 9:29 AM Kenneth Knowles <klk@xxxxxxxxxx> wrote:
Nice. I know that Java users have found it helpful to have this lower-level way of writing pipelines when the high-level primitives don't quite have the tight control they are looking for. I hope it will be a big draw for Python, too.

(commenting on the doc)

Kenn

On Mon, May 21, 2018 at 5:15 PM Charles Chen <ccy@xxxxxxxxxx> wrote:
I want to share a proposal for adding user state and timer support to the Beam Python SDK and get the community's thoughts on how such an API should look: https://s.apache.org/beam-python-user-state-and-timers

Let me know what you think and please add any comments and suggestions you may have.

Best,
Charles