OSDir


[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

[DISCUSS] Flink 1.4 and below STOPS writing to Kinesis after June 12th.


Hi,

I think the following email thread might have gone lost.

Dyana brought up the attention that AWS has informed users that KPL versions
< 0.12.6 will *stop working* starting from the 12th of June. Flink 1.4 is
using KPL 0.12.5 and Flink 1.5 uses 0.12.6, *so Flink 1.4 and below (1.3,
etc) will be impacted.*

I think we probably should try our best to communicate this out to our
users in user email alias and all other possible channels, in order not to
disrupt their production pipeline.

A quick solution we can suggest to users is to package and use
flink-connector-kinesis in Flink 1.5. Given that the public APIs don't
change, flink-connector-kinesis in Flink 1.5 should work with Flink 1.4 and
below, but that needs verification.

What do you think?

Thanks, Bowen


---------- Forwarded message ----------
From: Bowen Li <bowenli86@xxxxxxxxx>
Date: Fri, May 11, 2018 at 10:28 AM
Subject: Re: KPL in current stable 1.4.2 and below, upcoming problem
To: dev@xxxxxxxxxxxxxxxx, "Tzu-Li (Gordon) Tai" <tzulitai@xxxxxxxxxx>


Thanks, this is a great heads-up!  Flink 1.4 is using KPL 0.12.5, *so Flink
version 1.4 or below will be effected.*

Kinesis sink is in flink-kinesis-connector. Good news is that whoever is
using flink-kinesis-connector right now is building it themself, because
Flink doesn't publish that jar into maven due to licensing issue. So these
users, like you Dyana,  already have build experience and may try to bump
KPL themselves.

I think it'll be great if Flink can bump KPL in Flink 1.2/1.3/1.4 and
release minor versions for them, as an official support. It also requires
checking backward compatibility. This can be done after releasing 1.5.
@Gordon may take the final call of how to eventually do it.

Thanks,
Bowen


On Thu, May 10, 2018 at 1:35 AM, Dyana Rose <dyana.rose@xxxxxxxxxxxxx>
wrote:

> Hello,
>
> We've received notification from AWS that the Kinesis Producer Library
> versions < 0.12.6 will stop working after the 12th of June (assuming the
> date in the email is in US format...)
>
> Flink v1.5.0 has the KPL version at 0.12.6 so it will be fine when it's
> released. However using the kinesis connector in any previous version looks
> like they'll have an issue.
>
> I'm not sure how/if you want to communicate this. We build Flink ourselves,
> so I plan on having a look at any changes done to the Kinesis Sink in
> v1.5.0 and then bumpimg the KPL version in our fork and rebuilding.
>
> Thanks,
> Dyana
>
> below is the email we received (note: we're in eu-west-1):
> --------
>
> Hello,
>
>
>
> Your action is required: please update clients running Kinesis Producer
> Library 0.12.5 or older or you will experience a breaking change to your
> application.
>
>
>
> We've discovered you have one or more clients writing data to Amazon
> Kinesis Data Streams running an outdated version of the Kinesis Producer
> Library. On 6/12 these clients will be impacted if they are not updated to
> Kinesis Producer Library version 0.12.6 or newer. On 06/12 Kinesis Data
> Streams will install ATS certificates which will prevent these outdated
> clients from writing to a Kinesis Data Stream. The result of this change
> will break any producer using KPL 0.12.5 or older.
>
>
> * How do I update clients and applications to use the latest version of the
> Kinesis Producer Library?
>
> You will need to ensure producers leveraging the Kinesis Producer Library
> have upgraded to version 0.12.6 or newer. If you operate older versions
> your application will break due untrusted SSL certification.
>
> Via Maven install Kinesis Producer Library version 0.12.6 or higher [2]
>
> After you've configured your clients to use the new version, you're done.
>
> * What if I have questions or issues?
>
> If you have questions or issues, please contact your AWS Technical Account
> Manager or AWS support and file a support ticket [3].
>
> [1] https://docs.aws.amazon.com/streams/latest/dev/kinesis-kpl-u
> pgrades.html
>
> [2] http://search.maven.org/#artifactdetails|com.amazonaws|amazo
> n-kinesis-produ...
> <http://search.maven.org/#artifactdetails%7Ccom.amazonaws%7C
> amazon-kinesis-producer%7C0.12.6%7Cjar>
>
> [3] https://aws.amazon.com/support
>
>
>
> -          Amazon Kinesis Data Streams Team
> -----
>