OSDir


[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [DISCUSS] Flink 1.4 and below STOPS writing to Kinesis after June 12th.


I agree that it would be nice to offer a backport for all 1.4 users. Thomas
would it be possible to contribute the backport back to the community?

Cheers,
Till

On Wed, Jun 6, 2018 at 5:25 PM Thomas Weise <thw@xxxxxxxxxx> wrote:

> -->
>
> On Wed, Jun 6, 2018 at 1:06 AM, Bowen Li <bowenli86@xxxxxxxxx> wrote:
>
> > Hi,
> >
> > I think the following email thread might have gone lost.
> >
> > Dyana brought up the attention that AWS has informed users that KPL
> > versions
> > < 0.12.6 will *stop working* starting from the 12th of June. Flink 1.4 is
> > using KPL 0.12.5 and Flink 1.5 uses 0.12.6, *so Flink 1.4 and below (1.3,
> > etc) will be impacted.*
> >
> > I think we probably should try our best to communicate this out to our
> > users in user email alias and all other possible channels, in order not
> to
> > disrupt their production pipeline.
> >
> > A quick solution we can suggest to users is to package and use
> > flink-connector-kinesis in Flink 1.5. Given that the public APIs don't
> > change, flink-connector-kinesis in Flink 1.5 should work with Flink 1.4
> and
> > below, but that needs verification.
> >
>
> The connector might work, but it is going to require a little extra
> juggling for the users to exclude the 1.5 transitive dependencies.
>
> How about just backporting the changes? We internally use the newer SDK
> with a number of other consumer patches on top of 1.4.2 and it works fine.
>
> Normally such changes don't qualify for a patch version, but since it is
> going to stop working anyways, there is probably no harm in this instance.
>
> Thomas
>
>
>
> > What do you think?
> >
> > Thanks, Bowen
> >
> >
> > ---------- Forwarded message ----------
> > From: Bowen Li <bowenli86@xxxxxxxxx>
> > Date: Fri, May 11, 2018 at 10:28 AM
> > Subject: Re: KPL in current stable 1.4.2 and below, upcoming problem
> > To: dev@xxxxxxxxxxxxxxxx, "Tzu-Li (Gordon) Tai" <tzulitai@xxxxxxxxxx>
> >
> >
> > Thanks, this is a great heads-up!  Flink 1.4 is using KPL 0.12.5, *so
> Flink
> > version 1.4 or below will be effected.*
> >
> > Kinesis sink is in flink-kinesis-connector. Good news is that whoever is
> > using flink-kinesis-connector right now is building it themself, because
> > Flink doesn't publish that jar into maven due to licensing issue. So
> these
> > users, like you Dyana,  already have build experience and may try to bump
> > KPL themselves.
> >
> > I think it'll be great if Flink can bump KPL in Flink 1.2/1.3/1.4 and
> > release minor versions for them, as an official support. It also requires
> > checking backward compatibility. This can be done after releasing 1.5.
> > @Gordon may take the final call of how to eventually do it.
> >
> > Thanks,
> > Bowen
> >
> >
> > On Thu, May 10, 2018 at 1:35 AM, Dyana Rose <dyana.rose@xxxxxxxxxxxxx>
> > wrote:
> >
> > > Hello,
> > >
> > > We've received notification from AWS that the Kinesis Producer Library
> > > versions < 0.12.6 will stop working after the 12th of June (assuming
> the
> > > date in the email is in US format...)
> > >
> > > Flink v1.5.0 has the KPL version at 0.12.6 so it will be fine when it's
> > > released. However using the kinesis connector in any previous version
> > looks
> > > like they'll have an issue.
> > >
> > > I'm not sure how/if you want to communicate this. We build Flink
> > ourselves,
> > > so I plan on having a look at any changes done to the Kinesis Sink in
> > > v1.5.0 and then bumpimg the KPL version in our fork and rebuilding.
> > >
> > > Thanks,
> > > Dyana
> > >
> > > below is the email we received (note: we're in eu-west-1):
> > > --------
> > >
> > > Hello,
> > >
> > >
> > >
> > > Your action is required: please update clients running Kinesis Producer
> > > Library 0.12.5 or older or you will experience a breaking change to
> your
> > > application.
> > >
> > >
> > >
> > > We've discovered you have one or more clients writing data to Amazon
> > > Kinesis Data Streams running an outdated version of the Kinesis
> Producer
> > > Library. On 6/12 these clients will be impacted if they are not updated
> > to
> > > Kinesis Producer Library version 0.12.6 or newer. On 06/12 Kinesis Data
> > > Streams will install ATS certificates which will prevent these outdated
> > > clients from writing to a Kinesis Data Stream. The result of this
> change
> > > will break any producer using KPL 0.12.5 or older.
> > >
> > >
> > > * How do I update clients and applications to use the latest version of
> > the
> > > Kinesis Producer Library?
> > >
> > > You will need to ensure producers leveraging the Kinesis Producer
> Library
> > > have upgraded to version 0.12.6 or newer. If you operate older versions
> > > your application will break due untrusted SSL certification.
> > >
> > > Via Maven install Kinesis Producer Library version 0.12.6 or higher [2]
> > >
> > > After you've configured your clients to use the new version, you're
> done.
> > >
> > > * What if I have questions or issues?
> > >
> > > If you have questions or issues, please contact your AWS Technical
> > Account
> > > Manager or AWS support and file a support ticket [3].
> > >
> > > [1] https://docs.aws.amazon.com/streams/latest/dev/kinesis-kpl-u
> > > pgrades.html
> > >
> > > [2] http://search.maven.org/#artifactdetails|com.amazonaws|amazo
> > > n-kinesis-produ...
> > > <http://search.maven.org/#artifactdetails%7Ccom.amazonaws%7C
> > > amazon-kinesis-producer%7C0.12.6%7Cjar>
> > >
> > > [3] https://aws.amazon.com/support
> > >
> > >
> > >
> > > -          Amazon Kinesis Data Streams Team
> > > -----
> > >
> >
>