osdir.com

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: A new Beam Runner on Apache Nemo


Hi Wonook,


Some more details on what Max said about running the ValidatesRunner tests:

 - if you are planning to contribute the runner to Beam, you can use the other runners as examples and generally the whole community is likely to keep your config up to date

 - if you are planning to keep the runner as part of Apache Nemo, then I see you are using Maven to build so you can use an old snapshot as an example, like this: https://github.com/apache/beam/blob/v2.4.0/runners/gearpump/pom.xml#L55

Kenn

On Fri, Nov 16, 2018 at 3:16 AM Maximilian Michels <mxm@xxxxxxxxxx> wrote:
Hi Wonook,

First of all, welcome to the Beam community! It is great to see another
Runner emerging.

If you're planning to contribute your Runner to Beam, you should verify
the compatibility with the ValidatesRunner integration tests. Then open
a PR with documentation, a Runner page, and updates to the matrix.

If you're planning to leave the Runner outside Beam for the time being,
please submit a Runner page for the Beam website. The page should
contain information on how to use the Runner and a link to the external
web site with up-to-date information.

Feel free to ask here or in our Slack channel if you have more questions.

I'm also curious, have you looked into integrating portability with the
Nemo Runner?

Thanks,
Max

On 16.11.18 06:51, 송원욱 wrote:
> Hello all!
>
> I'm a member of the Apache Nemo community, another Apache project for
> processing big data focusing on easy-to-use, flexible optimizations for
> various deployment environments. More information can be seen on our
> website <http://nemo.apache.org>. We've been building the system for
> quite a while now, and we have been using Apache Beam as one of the
> programming layers that we support for writing data processing
> applications. We have already taken a look at the capability matrix
> <https://beam.apache.org/documentation/runners/capability-matrix/> of
> Beam runners, and the runner authoring guide
> <https://beam.apache.org/contribute/runner-guide/>, and we have been
> successful in implementing a large portion of the capability criteria.
>
> With the progress, we wish to be able to list our runner as one of the
> Beam runners, to  be able to notify the users that our system supports
> Beam, and that Beam users have another option to choose from for running
> their data processing applications. It would be lovely to know the
> details of the process required for it!
>
> Thanks!
>
>
>       Wonook
>