I have a side input with streaming updates in a global window. I have tried to approach this several ways but can’t figure out how to do it. What I really need is a side-input Map that should be updated when streaming input change (i.e keys are updated).
I have tried to implement this with a
View.asMap transform but got an error that I have duplicate keys in my set (which are there due to multiple triggered updates on the pcollection). I then tried to do it as a singleton using a global CombineFn to build the map and use the a
Combine.globally().asSingletonView() instead. But I then got an error
java.lang.IllegalArgumentException: PCollection with more than one element accessed as a singleton view.
How are you supposed to do this? In the documentation there is a part that suggests that this should be possible:
If the side input has multiple trigger firings, Beam uses the value from the latest trigger firing. This is particularly useful if you use a side input with a single global window and specify a trigger.
// Vilhelm von Ehrenheim