OSDir


[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: How to use MapElements.via in a nice way?


Hi Pawel,

I will ignore your mistakes and just mention that it is a known problem - the way lambdas are typed in Java 8 is quite annoying for us. That is why you have to specify the type. But if I recall, this should work to at least be a little more concise:

   MapElements.via(new SimpleFunction(<lambda>){})

Kenn

On Wed, Apr 25, 2018 at 9:52 AM Pawel Szczur <pawelszczur@xxxxxxxxx> wrote:
This email has at least 3 mistakes, please ignore it. I need to rethink it.

2018-04-25 18:45 GMT+02:00 Pawel Szczur <pawelszczur@xxxxxxxxx>:
Hi,

I'm using Beam since Google Dataflow has been made public.
There used to be MapFn. Some time later it was removed.

Now we have MapElements helper which kind of is suppose to do similar things.
Using it with new Java 8 API seems pretty ugly to me, especially when I have something like:

PCollection<KV<Class1, Class2>> input = ...;
input.apply(
  MapElements.into(?).via(v -> KV.of(newKey(v.getKey()), v.getValue()));

I would need to replace ? by KVCoder.of(KeyCoder.of(), ValueCoder.of()).
This is troublesome especially when in a longer transformation chains.

For me, it's became easier to write:
input.apply(DoFn(new ReplaceKey()))
where ReplaceKey is a implementation of SimpleFunction defined somewhere and just replacing KV's key.

I'm wondering, am I missing something?

Cheers, Paweł Szczur