osdir.com

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Multiple firings on side input


Hi!
I have a side input with streaming updates in a global window. I have tried to approach this several ways but can’t figure out how to do it. What I really need is a side-input Map that should be updated when streaming input change (i.e keys are updated).

I have tried to implement this with a View.asMap transform but got an error that I have duplicate keys in my set (which are there due to multiple triggered updates on the pcollection). I then tried to do it as a singleton using a global CombineFn to build the map and use the a Combine.globally().asSingletonView() instead. But I then got an error java.lang.IllegalArgumentException: PCollection with more than one element accessed as a singleton view.

How are you supposed to do this? In the documentation there is a part that suggests that this should be possible:

If the side input has multiple trigger firings, Beam uses the value from the latest trigger firing. This is particularly useful if you use a side input with a single global window and specify a trigger.

Thanks!

// Vilhelm von Ehrenheim