osdir.com


[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: Python 3.6 Support for Airflow 1.10.0


Okay, a worrying (to me) number of people are still on Python 2.7 :(

AIP created anyway https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-3+Drop+support+for+Python+2

If you are adding support for a Py3 to your platform I'd skip 3.5 and go straight to 3.6/3.7

Let's not forget that supporting Python 2 is not free for us - both in terms of test time on Travis, and in terms of code complexity in having to use six/backports all over the place.


> On 29 Aug 2018, at 09:46, Sumit Maheshwari <msumit@xxxxxxxxxx> wrote:
> 
> Strong +1 for keeping 2.7 for now. We've 30+ big and small enterprises
> running Airflow as a service on our platform and they all are on py 2.7 for
> now. Soon we are adding support for py 3.5 as well, but still, not everyone
> gonna switch to that immediately.
> 
> So IMO we should keep the support of py 2.7 until mid-2019.
> 
> On Wed, Aug 29, 2018 at 6:01 AM Tao Feng <fengtao04@xxxxxxxxx> wrote:
> 
>> +1 for keeping 2.7 as well. Lyft runs airflow on 2.7 internally.
>> 
>> On Tue, Aug 28, 2018 at 1:12 PM, Feng Lu <fenglu@xxxxxxxxxx.invalid>
>> wrote:
>> 
>>> +1 for keeping 2.7 as long as we can so people have time to plan and
>>> migrate away from it.
>>> 
>>> On Tue, Aug 28, 2018, 10:35 Arthur Wiedmer <arthur.wiedmer@xxxxxxxxx>
>>> wrote:
>>> 
>>>> Given that Python 2.7 EOL is slated for January 1st 2020, we should
>>>> probably ensure that the early releases of 2019 are still 2.7
>> compatible.
>>>> 
>>>> Beyond this, I think we can also be responsible security wise and help
>>>> nudge people towards 3.
>>>> 
>>>> Best,
>>>> Arthur
>>>> 
>>>> On Tue, Aug 28, 2018 at 10:28 AM Bolke de Bruin <bdbruin@xxxxxxxxx>
>>> wrote:
>>>> 
>>>>> Let’s not drop 2.7 too quickly but maybe mark it deprecated. I’m
>> pretty
>>>>> sure Airbnb still runs on 2.7.
>>>>> 
>>>>> Also RedHat does not deliver python 3 in its enterprise edition yet
>> by
>>>>> default so it will put enterprise users in a bit of an awkward spot.
>>>>> 
>>>>> B.
>>>>> 
>>>>> Verstuurd vanaf mijn iPad
>>>>> 
>>>>>> Op 28 aug. 2018 om 19:00 heeft Sid Anand <sanand@xxxxxxxxxx> het
>>>>> volgende geschreven:
>>>>>> 
>>>>>> I'm +1 on going to 3.7 -- I'm running 3.6 myself.
>>>>>> 
>>>>>> Regarding dropping Python2 support, with almost 200 companies using
>>>>>> Airflow, I'd want to be very careful that we don't put any of them
>>> at a
>>>>>> disadvantage. For example, my former employer (a small startup) is
>>>>> running
>>>>>> on Python2 -- after I left, they don't have anyone actively
>>> maintaining
>>>>> it
>>>>>> at the company. Easing upgrades for such cases will keep them using
>>>>> Airflow.
>>>>>> 
>>>>>> It would be good to hold a survey that we promote beyond daily
>>> readers
>>>> of
>>>>>> this mailing list and raise this as an AIP, since it's a major
>>> change.
>>>>>> Let's not rush it.
>>>>>> 
>>>>>> -s
>>>>>> 
>>>>>>> On Tue, Aug 28, 2018 at 9:24 AM Naik Kaxil <k.naik@xxxxxxxxx>
>>> wrote:
>>>>>>> 
>>>>>>> We should definitely support 3.7. I left comments on the PR
>>> @tedmiston
>>>>>>> regarding the same. Python 2.7 will be dropped in 2020, so I guess
>>> we
>>>>>>> should start planning about it. Not really 100% sure though that
>> we
>>>>> should
>>>>>>> drop it in Airflow 2.0
>>>>>>> 
>>>>>>> On 28/08/2018, 17:08, "Taylor Edmiston" <tedmiston@xxxxxxxxx>
>>> wrote:
>>>>>>> 
>>>>>>>   I am onboard with dropping Python 2.x support.  Django
>> officially
>>>>>>> dropped
>>>>>>>   Python 2.x support with their 2.0 release since December 2017.
>>>>>>> 
>>>>>>>   *Taylor Edmiston*
>>>>>>>   Blog <https://blog.tedmiston.com/> | CV
>>>>>>>   <https://stackoverflow.com/cv/taylor> | LinkedIn
>>>>>>>   <https://www.linkedin.com/in/tedmiston/> | AngelList
>>>>>>>   <https://angel.co/taylor> | Stack Overflow
>>>>>>>   <https://stackoverflow.com/users/149428/taylor-edmiston>
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>>   On Tue, Aug 28, 2018 at 12:03 PM Ash Berlin-Taylor <
>>> ash@xxxxxxxxxx
>>>>> 
>>>>>>> wrote:
>>>>>>> 
>>>>>>>> Supporting 3.7 is absolutely something we should do - it just got
>>>>>>> released
>>>>>>>> while we were already mid-way through the release process of 1.10
>>> and
>>>>>>>> didn't want the scope creep.
>>>>>>>> 
>>>>>>>> I'm happy to release a 1.10.1 that supports Py 3.7. The only
>> issue
>>>>>>> I've
>>>>>>>> seen so far is around the use of `async` as a keyword. both in
>>>>>>>> 
>>>>>>>> A perhaps bigger question: What are people's thoughts on dropping
>>>>>>> support
>>>>>>>> for Python2? This wouldn't happen before 2.0 at the earliest if
>> we
>>>>>>> did it.
>>>>>>>> Probably something to raise an AIP for.
>>>>>>>> 
>>>>>>>> -ash
>>>>>>>> 
>>>>>>>>> 
>>>>>>> 
>>>>>>> Kaxil Naik
>>>>>>> 
>>>>>>> Data Reply
>>>>>>> 2nd Floor, Nova South
>>>>>>> 160 Victoria Street, Westminster
>>>>>>> London SW1E 5LB - UK
>>>>>>> phone: +44 (0)20 7730 6000
>>>>>>> k.naik@xxxxxxxxx
>>>>>>> www.reply.com
>>>>>>> On 28 Aug 2018, at 16:56, Taylor Edmiston <tedmiston@xxxxxxxxx>
>>>> wrote:
>>>>>>>>> 
>>>>>>>>> We are also running on 3.6 for some time.
>>>>>>>>> 
>>>>>>>>> I put a quick branch together adding / upgrading to 3.6 in all
>> of
>>>>>>> the
>>>>>>>>> places.  CI is still running so I may expect some test failures
>>> but
>>>>>>>>> hopefully nothing major.  I would be happy to merge this into
>>>>>>> Kaxil's
>>>>>>>>> current #3815 or as a follow-on PR.  I'll paste this back onto
>> his
>>>>>>> PR as
>>>>>>>>> well.
>>>>>>>>> 
>>>>>>>>> https://github.com/apache/incubator-airflow/pull/3816
>>>>>>>>> 
>>>>>>>>> I think it's important for the project to officially support
>>>>>>> Python 3.6
>>>>>>>>> latest especially since 3.7 is out now.  While we're on the
>> topic,
>>>>>>> does
>>>>>>>>> anyone else have thoughts on supporting 3.7 (perhaps
>> unofficially
>>>>>>> to
>>>>>>>>> start)?  I wouldn't mind starting to get that ball rolling.
>>>>>>>>> 
>>>>>>>>> *Taylor Edmiston*
>>>>>>>>> Blog <https://blog.tedmiston.com/> | CV
>>>>>>>>> <https://stackoverflow.com/cv/taylor> | LinkedIn
>>>>>>>>> <https://www.linkedin.com/in/tedmiston/> | AngelList
>>>>>>>>> <https://angel.co/taylor> | Stack Overflow
>>>>>>>>> <https://stackoverflow.com/users/149428/taylor-edmiston>
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> On Tue, Aug 28, 2018 at 9:29 AM Adam Boscarino
>>>>>>>>> <aboscarino@xxxxxxxxxxxxxxxx.invalid> wrote:
>>>>>>>>> 
>>>>>>>>>> fwiw, we run Airflow on Python 3.6.
>>>>>>>>>> 
>>>>>>>>>> On Tue, Aug 28, 2018 at 8:30 AM Naik Kaxil <k.naik@xxxxxxxxx>
>>>>>>> wrote:
>>>>>>>>>> 
>>>>>>>>>>> To provide more context to the issue:
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> PyPI shows that Airflow is supported on Py2.7, 3.4 and 3.5 :
>>>>>>>>>>> https://pypi.org/project/apache-airflow/
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> This is picked from setup.py:
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>> 
>>>>>>> 
>>>>> 
>>>> https://github.com/apache/incubator-airflow/blob/
>>> 26e0d449737e8671000f671d820a9537f23f345a/setup.py#L367
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> So, should we update setup.py to include 3.6 as well?
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> @bolke – Thughts?
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> Kaxil Naik
>>>>>>>>>>> 
>>>>>>>>>>> Data Reply
>>>>>>>>>>> 2nd Floor, Nova South
>>>>>>>>>>> 160 Victoria Street, Westminster
>>>>>>>>>>> London SW1E 5LB - UK
>>>>>>>>>>> phone: +44 (0)20 7730 6000
>>>>>>>>>>> k.naik@xxxxxxxxx
>>>>>>>>>>> www.reply.com
>>>>>>>>>>> 
>>>>>>>>>>> [image: Data Reply]
>>>>>>>>>>> 
>>>>>>>>>>> *From: *Naik Kaxil <k.naik@xxxxxxxxx>
>>>>>>>>>>> *Reply-To: *"dev@xxxxxxxxxxxxxxxxxxxxxxxxxxxx" <
>>>>>>>>>>> dev@xxxxxxxxxxxxxxxxxxxxxxxxxxxx>
>>>>>>>>>>> *Date: *Tuesday, 28 August 2018 at 13:27
>>>>>>>>>>> *To: *"dev@xxxxxxxxxxxxxxxxxxxxxxxxxxxx" <
>>>>>>>>>> dev@xxxxxxxxxxxxxxxxxxxxxxxxxxxx
>>>>>>>>>>>> 
>>>>>>>>>>> *Subject: *Python 3.6 Support for Airflow 1.10.0
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> Hi all,
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> @fokko – I remember that you had test Airflow on 3.6 . Can we
>>>>>>> include
>>>>>>>> 3.6
>>>>>>>>>>> in setup.py then ?
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> Regards,
>>>>>>>>>>> 
>>>>>>>>>>> Kaxil
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> Kaxil Naik
>>>>>>>>>>> 
>>>>>>>>>>> *Data Reply*
>>>>>>>>>>> 2nd Floor, Nova South
>>>>>>>>>>> 160 Victoria Street, Westminster
>>>>>>>>>>> London SW1E 5LB - UK
>>>>>>>>>>> phone: +44 (0)20 7730 6000
>>>>>>>>>>> k.naik@xxxxxxxxx
>>>>>>>>>>> www.reply.com
>>>>>>>>>>> 
>>>>>>>>>>> [image: Data Reply]
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> --
>>>>>>>>>> Adam Boscarino
>>>>>>>>>> Senior Data Engineer
>>>>>>>>>> 
>>>>>>>>>> aboscarino@xxxxxxxxxxxxxxxx
>>>>>>>>>> ------------------------------
>>>>>>>>>> We're Hiring! <https://www.digitalocean.com/company/careers/>
>> |
>>>>>>>>>> @digitalocean <https://twitter.com/digitalocean>
>>>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>> 
>>>> 
>>> 
>>