[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: How to use MapElements.via in a nice way?

This email has at least 3 mistakes, please ignore it. I need to rethink it.

2018-04-25 18:45 GMT+02:00 Pawel Szczur <pawelszczur@xxxxxxxxx>:

I'm using Beam since Google Dataflow has been made public.
There used to be MapFn. Some time later it was removed.

Now we have MapElements helper which kind of is suppose to do similar things.
Using it with new Java 8 API seems pretty ugly to me, especially when I have something like:

PCollection<KV<Class1, Class2>> input = ...;
  MapElements.into(?).via(v -> KV.of(newKey(v.getKey()), v.getValue()));

I would need to replace ? by KVCoder.of(KeyCoder.of(), ValueCoder.of()).
This is troublesome especially when in a longer transformation chains.

For me, it's became easier to write:
input.apply(DoFn(new ReplaceKey()))
where ReplaceKey is a implementation of SimpleFunction defined somewhere and just replacing KV's key.

I'm wondering, am I missing something?

Cheers, Paweł Szczur