OSDir


[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [DISCUSS] Flink 1.4 and below STOPS writing to Kinesis after June 12th.


Thanks for bring this up, Bowen.

I’m not sure that it’ll be straightforward to just use the Kinesis connector in Flink 1.5 to work with 1.4.
So, +1 to have a backport of necessary changes to allow the Kinesis connector to continue to work for Flink 1.4, and we proceed with another bug fix release for the 1.4.x series.

We might want to consider only cherry-picking the Kinesis connector backports on-top of 1.4.2, instead of branching out from 1.4-SNAPSHOT directly, to minimize the test efforts that would be required for releasing.
But this should probably be discussed on a separate thread of its own.

Cheers,
Gordon


On 11 June 2018 at 12:29:10 PM, Till Rohrmann (trohrmann@xxxxxxxxxx) wrote:

I agree that it would be nice to offer a backport for all 1.4 users. Thomas  
would it be possible to contribute the backport back to the community?  

Cheers,  
Till  

On Wed, Jun 6, 2018 at 5:25 PM Thomas Weise <thw@xxxxxxxxxx> wrote:  

> -->  
>  
> On Wed, Jun 6, 2018 at 1:06 AM, Bowen Li <bowenli86@xxxxxxxxx> wrote:  
>  
> > Hi,  
> >  
> > I think the following email thread might have gone lost.  
> >  
> > Dyana brought up the attention that AWS has informed users that KPL  
> > versions  
> > < 0.12.6 will *stop working* starting from the 12th of June. Flink 1.4 is  
> > using KPL 0.12.5 and Flink 1.5 uses 0.12.6, *so Flink 1.4 and below (1.3,  
> > etc) will be impacted.*  
> >  
> > I think we probably should try our best to communicate this out to our  
> > users in user email alias and all other possible channels, in order not  
> to  
> > disrupt their production pipeline.  
> >  
> > A quick solution we can suggest to users is to package and use  
> > flink-connector-kinesis in Flink 1.5. Given that the public APIs don't  
> > change, flink-connector-kinesis in Flink 1.5 should work with Flink 1.4  
> and  
> > below, but that needs verification.  
> >  
>  
> The connector might work, but it is going to require a little extra  
> juggling for the users to exclude the 1.5 transitive dependencies.  
>  
> How about just backporting the changes? We internally use the newer SDK  
> with a number of other consumer patches on top of 1.4.2 and it works fine.  
>  
> Normally such changes don't qualify for a patch version, but since it is  
> going to stop working anyways, there is probably no harm in this instance.  
>  
> Thomas  
>  
>  
>  
> > What do you think?  
> >  
> > Thanks, Bowen  
> >  
> >  
> > ---------- Forwarded message ----------  
> > From: Bowen Li <bowenli86@xxxxxxxxx>  
> > Date: Fri, May 11, 2018 at 10:28 AM  
> > Subject: Re: KPL in current stable 1.4.2 and below, upcoming problem  
> > To: dev@xxxxxxxxxxxxxxxx, "Tzu-Li (Gordon) Tai" <tzulitai@xxxxxxxxxx>  
> >  
> >  
> > Thanks, this is a great heads-up! Flink 1.4 is using KPL 0.12.5, *so  
> Flink  
> > version 1.4 or below will be effected.*  
> >  
> > Kinesis sink is in flink-kinesis-connector. Good news is that whoever is  
> > using flink-kinesis-connector right now is building it themself, because  
> > Flink doesn't publish that jar into maven due to licensing issue. So  
> these  
> > users, like you Dyana, already have build experience and may try to bump  
> > KPL themselves.  
> >  
> > I think it'll be great if Flink can bump KPL in Flink 1.2/1.3/1.4 and  
> > release minor versions for them, as an official support. It also requires  
> > checking backward compatibility. This can be done after releasing 1.5.  
> > @Gordon may take the final call of how to eventually do it.  
> >  
> > Thanks,  
> > Bowen  
> >  
> >  
> > On Thu, May 10, 2018 at 1:35 AM, Dyana Rose <dyana.rose@xxxxxxxxxxxxx>  
> > wrote:  
> >  
> > > Hello,  
> > >  
> > > We've received notification from AWS that the Kinesis Producer Library  
> > > versions < 0.12.6 will stop working after the 12th of June (assuming  
> the  
> > > date in the email is in US format...)  
> > >  
> > > Flink v1.5.0 has the KPL version at 0.12.6 so it will be fine when it's  
> > > released. However using the kinesis connector in any previous version  
> > looks  
> > > like they'll have an issue.  
> > >  
> > > I'm not sure how/if you want to communicate this. We build Flink  
> > ourselves,  
> > > so I plan on having a look at any changes done to the Kinesis Sink in  
> > > v1.5.0 and then bumpimg the KPL version in our fork and rebuilding.  
> > >  
> > > Thanks,  
> > > Dyana  
> > >  
> > > below is the email we received (note: we're in eu-west-1):  
> > > --------  
> > >  
> > > Hello,  
> > >  
> > >  
> > >  
> > > Your action is required: please update clients running Kinesis Producer  
> > > Library 0.12.5 or older or you will experience a breaking change to  
> your  
> > > application.  
> > >  
> > >  
> > >  
> > > We've discovered you have one or more clients writing data to Amazon  
> > > Kinesis Data Streams running an outdated version of the Kinesis  
> Producer  
> > > Library. On 6/12 these clients will be impacted if they are not updated  
> > to  
> > > Kinesis Producer Library version 0.12.6 or newer. On 06/12 Kinesis Data  
> > > Streams will install ATS certificates which will prevent these outdated  
> > > clients from writing to a Kinesis Data Stream. The result of this  
> change  
> > > will break any producer using KPL 0.12.5 or older.  
> > >  
> > >  
> > > * How do I update clients and applications to use the latest version of  
> > the  
> > > Kinesis Producer Library?  
> > >  
> > > You will need to ensure producers leveraging the Kinesis Producer  
> Library  
> > > have upgraded to version 0.12.6 or newer. If you operate older versions  
> > > your application will break due untrusted SSL certification.  
> > >  
> > > Via Maven install Kinesis Producer Library version 0.12.6 or higher [2]  
> > >  
> > > After you've configured your clients to use the new version, you're  
> done.  
> > >  
> > > * What if I have questions or issues?  
> > >  
> > > If you have questions or issues, please contact your AWS Technical  
> > Account  
> > > Manager or AWS support and file a support ticket [3].  
> > >  
> > > [1] https://docs.aws.amazon.com/streams/latest/dev/kinesis-kpl-u  
> > > pgrades.html  
> > >  
> > > [2] http://search.maven.org/#artifactdetails|com.amazonaws|amazo  
> > > n-kinesis-produ...  
> > > <http://search.maven.org/#artifactdetails%7Ccom.amazonaws%7C  
> > > amazon-kinesis-producer%7C0.12.6%7Cjar>  
> > >  
> > > [3] https://aws.amazon.com/support  
> > >  
> > >  
> > >  
> > > - Amazon Kinesis Data Streams Team  
> > > -----  
> > >  
> >  
>