OSDir


[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: Is it possible to fire a trigger per key?


Hi Andre,
Triggers are already per-key (and per-window), as you describe. However, triggers do not give a hard guarantee that it will fire *immediately* when 1000 elements arrive - the trigger *enables* firing, and in practice it can be somewhat more than 1000 (in the limit case, most batch runners will fire triggers trivially after the entire PCollection is processed).

On Thu, Jul 12, 2018 at 9:59 PM Andre Marianiello <andre.marianiello@xxxxxxxxx> wrote:
Based on my current understanding, when a trigger fires, all elements that have been waiting behind a GroupByKey are allowed to continue through the pipeline. Is it possible to create a custom trigger that fires independently for each key? For example, instead of a trigger that fires after 1000 elements total, is it possible to write a trigger that fires for a specific key once 1000 elements with that key have been found?