[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: A new Beam Runner on Apache Nemo

Thanks for the reply and the help!

At the moment, we are thinking about keeping the Runner outside Beam for the time being, as there are a number of extra ongoing developments going regarding a few features for stream processing. I'll submit a PR for the website in a short time with the details for the capability matrix, and on how to use our Runner, with external links and references, etc.

Regarding the portability layer, at the moment we have been focusing on supporting the various features supported by the Java Beam SDK on our system and improving the performance of the system itself, but we will definitely work on the portability layer with the Nemo Runner, as it sounds more than exciting to be able to run Python programs on Apache Nemo. I'll definitely check up on the ValidatesRunner tests. We have been running our tests as well, so we are quite confident that it would run without much problems. Thanks for the tip for the Maven project! 



2018년 11월 17일 (토) 오전 1:48, Kenneth Knowles <kenn@xxxxxxxxxx>님이 작성:
Hi Wonook,

Some more details on what Max said about running the ValidatesRunner tests:

 - if you are planning to contribute the runner to Beam, you can use the other runners as examples and generally the whole community is likely to keep your config up to date

 - if you are planning to keep the runner as part of Apache Nemo, then I see you are using Maven to build so you can use an old snapshot as an example, like this: https://github.com/apache/beam/blob/v2.4.0/runners/gearpump/pom.xml#L55


On Fri, Nov 16, 2018 at 3:16 AM Maximilian Michels <mxm@xxxxxxxxxx> wrote:
Hi Wonook,

First of all, welcome to the Beam community! It is great to see another
Runner emerging.

If you're planning to contribute your Runner to Beam, you should verify
the compatibility with the ValidatesRunner integration tests. Then open
a PR with documentation, a Runner page, and updates to the matrix.

If you're planning to leave the Runner outside Beam for the time being,
please submit a Runner page for the Beam website. The page should
contain information on how to use the Runner and a link to the external
web site with up-to-date information.

Feel free to ask here or in our Slack channel if you have more questions.

I'm also curious, have you looked into integrating portability with the
Nemo Runner?


On 16.11.18 06:51, 송원욱 wrote:
> Hello all!
> I'm a member of the Apache Nemo community, another Apache project for
> processing big data focusing on easy-to-use, flexible optimizations for
> various deployment environments. More information can be seen on our
> website <http://nemo.apache.org>. We've been building the system for
> quite a while now, and we have been using Apache Beam as one of the
> programming layers that we support for writing data processing
> applications. We have already taken a look at the capability matrix
> <https://beam.apache.org/documentation/runners/capability-matrix/> of
> Beam runners, and the runner authoring guide
> <https://beam.apache.org/contribute/runner-guide/>, and we have been
> successful in implementing a large portion of the capability criteria.
> With the progress, we wish to be able to list our runner as one of the
> Beam runners, to  be able to notify the users that our system supports
> Beam, and that Beam users have another option to choose from for running
> their data processing applications. It would be lovely to know the
> details of the process required for it!
> Thanks!
>       Wonook