[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: Evolving a Coder for an added field

I think the idea is that you would use one coder for paths where you don't need this information and would have FileIO provide a separate path that uses your updated coder.
Existing users would not be impacted and users of the new FileIO that depend on this information would not be able to have updated their pipeline in the first place.

If the feature in FileIO is experimental, we could choose to break it for existing users though since I don't know how feasible my suggestion above is.

On Fri, Nov 2, 2018 at 12:56 PM Jeff Klukas <jklukas@xxxxxxxxxxx> wrote:
Lukasz - Thanks for those links. That's very helpful context.

It sounds like there's no explicit user contract about evolving Coder classes in the Java SDK and users might reasonably assume Coders to be stable between SDK versions. Thus, users of the Dataflow or Flink runners might reasonably expect that they can update the Java SDK version used in their pipeline when performing an update.

Based in that understanding, evolving a class like Metadata might not be possible except in a major version bump where it's obvious to users to expect breaking changes and not to expect an "update" operation to work.

It's not clear to me what changing the "name" of a coder would look like or whether that's a tenable solution here. Would that change be able to happen within the SDK itself, or is it something users would need to specify?