[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

[Discuss] Upgrade story for Beam's execution engines


Hi Beamers,

In the light of the discussion about Beam LTS releases, I'd like to kick off a thread about how often we upgrade the execution engine of each Runner. By upgrade, I mean major/minor versions which typically break the binary compatibility of Beam pipelines.

For the Flink Runner, we try to track the latest stable version. Some users reported that this can be problematic, as it requires them to potentially upgrade their Flink cluster with a new version of Beam.

From a developer's perspective, it makes sense to migrate as early as possible to the newest version of the execution engine, e.g. to leverage the newest features. From a user's perspective, you don't care about the latest features if your use case still works with Beam.

We have to please both parties. So I'd suggest to upgrade the execution engine whenever necessary (e.g. critical new features, end of life of current version). On the other hand, the upcoming Beam LTS releases will contain a longer-supported version.

Maybe we don't need to discuss much about this but I wanted to hear what the community has to say about it. Particularly, I'd be interested in how the other Runner authors intend to do it.

As far as I understand, with the portability being stable, we could theoretically upgrade the SDK without upgrading the runtime components. That would allow us to defer the upgrade for a longer time.

Best,
Max