[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: Beam Samza Runner status update

Thanks for the updating, Xinyu and Hai! Great to see another Running emerging :)

I'm on the FlinkRunner. Looking forward to working together with you to make the Beam Runners even better. Particularly, we should sync on the portability, as some things are still to be fleshed out. In Flink, we are starting to integrate portable State.


On 11.10.18 05:14, Jesse Anderson wrote:

On Wed, Oct 10, 2018, 3:49 PM Kenneth Knowles <kenn@xxxxxxxxxx <mailto:kenn@xxxxxxxxxx>> wrote:

    Welcome, Hai!

    On Wed, Oct 10, 2018 at 3:46 PM Hai Lu <lhaiesp@xxxxxxxxx
    <mailto:lhaiesp@xxxxxxxxx>> wrote:

        Hi, all

        This is Hai from LinkedIn. As Xinyu mentioned, I have been
        working on portable API for Samza runner and made some solid
        progress. It's been a very smooth process (although not
        effortless for sure) and I'm really grateful for the great
        platform that you all have built. I'm very impressed. Bravo!

        Excited to work with everyone on Beam. Do expect more questions
        from me down the road.


        On Wed, Oct 10, 2018 at 12:36 PM Kenneth Knowles
        <kenn@xxxxxxxxxx <mailto:kenn@xxxxxxxxxx>> wrote:

            Clarification: Thomas Groh wrote the fuser, not me!

            Thanks for the sharing all this. Really cool.


            On Wed, Oct 10, 2018 at 11:17 AM Rui Wang <ruwang@xxxxxxxxxx
            <mailto:ruwang@xxxxxxxxxx>> wrote:

                Thanks for sharing! it's so exciting to hear that Beam
                is being used on Samza in production @LinkedIn! Your
                feedback will be helpful to Beam community!

                Besides, Beam supports SQL right now and hopefully Beam
                community could also receive feedback on BeamSQL
                <https://beam.apache.org/documentation/dsls/sql/overview/> in
                the future.


                On Wed, Oct 10, 2018 at 11:10 AM Jean-Baptiste Onofré
                <jb@xxxxxxxxxxxx <mailto:jb@xxxxxxxxxxxx>> wrote:

                    Thanks for sharing and congrats for this great work !

                    Le 10 oct. 2018, à 20:23, Xinyu Liu <xinyuliu.us
                    <mailto:%3Ca>@gmail.com <http://gmail.com>
                    <http://xinyuliu.us>@gmail.com <http://gmail.com>> a

                        Hi, All,

                        It's been over four months since we added the
                        Samza Runner to Beam, and we've been making a
                        lot of progress after that. Here I would like to
                        update your guys and share some really good news
                        happening here at LinkedIn:

                        1) First Beam job in production @LInkedIn!
                        After a few rounds of testing and benchmarking,
                        we finally rolled out our first Beam job here!
                        The job uses quite a few features, such as event
                        time, fixed/session windowing, early triggering,
                        and stateful processing. Our first customer is
                        very happy and they highly appraise the
                        easy-to-use Beam API as well as powerful
                        processing model. Due to the limited resources
                        here, we put our full trust in the work you guys
                        are doing, and we didn't run into any surprises.
                        We see extremely attention to details as well as
                        non-compromise in any user experience everywhere
                        in the code base. We would like to thank
                        everyone in the Beam community to contribute to
                        such an amazing framework!

                        2) A portable Samza Runner prototype
                        We are also starting the work in making Samza
                        Runner portable. So far we just got the python
                        word count example working using portable Samza
                        Runner. Please look out for the PR for this very
                        soon :). Again, this work is not possible
                        without the great Beam portability framework,
                        and the developers like Luke and Ahmet, just to
                        name a few, behind it. The ReferenceRunner has
                        been extremely useful to us to figure out what's
                        needed and how it works. Kudos to Thomas Groh,
                        Ben Sidhom and all the others who makes this
                        available to us. And to Kenn, your fuse work rocks.

                        3) More contributors in Samza Runner
                        The runner has been Chris and my personal
                        project for a while and now it's not the case.
                        We got Hai Lu and Boris Shkolnik from Samza team
                        to contribute. Hai has been focusing on the
                        portability work as mentioned in #2, and Boris
                        will work mostly on supporting our use cases. We
                        will send more emails discussing our use cases,
                        like the "Update state after firing" email I
                        sent out earlier.

                        Finally, a shout-out to our very own Chris
                        Pettitt. Without you, none of the above won't