OSDir


[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: Pinning dependencies for Apache Airflow


One thing to point out here.

Right now if you `pip install apache-airflow=1.10.0` in a clean environment it will fail.

This is because we pin flask-login to 0.2.1 but flask-appbuilder is >= 1.11.1, so that pulls in 1.12.0 which requires flask-login >= 0.3.

So I do think there is maybe something to be said about pinning for releases. The down side to that is that if there are updates to a module that we want then we have to make a point release to let people get it

Both methods have draw-backs

-ash

> On 4 Oct 2018, at 17:13, Arthur Wiedmer <arthur.wiedmer@xxxxxxxxx> wrote:
> 
> Hi Jarek,
> 
> I will +1 the discussion Dan is referring to and George's advice.
> 
> I just want to double check we are talking about pinning in
> requirements.txt only.
> 
> This offers the ability to
> pip install -r requirements.txt
> pip install --no-deps airflow
> For a guaranteed install which works.
> 
> Several different requirement files can be provided for specific use cases,
> like a stable dev one for instance for people wanting to work on operators
> and non-core functions.
> 
> However, I think we should proactively test in CI against unpinned
> dependencies (though it might be a separate case in the matrix) , so that
> we get advance warning if possible that things will break.
> CI downtime is not a bad thing here, it actually caught a problem :)
> 
> We should unpin as possible in setup.py to only maintain minimum required
> compatibility. The process of pinning in setup.py is extremely detrimental
> when you have a large number of python libraries installed with different
> pinned versions.
> 
> Best,
> Arthur
> 
> On Thu, Oct 4, 2018 at 8:36 AM Dan Davydov <ddavydov@xxxxxxxxxxx.invalid>
> wrote:
> 
>> Relevant discussion about this:
>> 
>> https://github.com/apache/incubator-airflow/pull/1809#issuecomment-257502174
>> 
>> On Thu, Oct 4, 2018 at 11:25 AM Jarek Potiuk <Jarek.Potiuk@xxxxxxxxxxx>
>> wrote:
>> 
>>> TL;DR; A change is coming in the way how dependencies/requirements are
>>> specified for Apache Airflow - they will be fixed rather than flexible
>> (==
>>> rather than >=).
>>> 
>>> This is follow up after Slack discussion we had with Ash and Kaxil -
>>> summarising what we propose we'll do.
>>> 
>>> *Problem:*
>>> During last few weeks we experienced quite a few downtimes of TravisCI
>>> builds (for all PRs/branches including master) as some of the transitive
>>> dependencies were automatically upgraded. This because in a number of
>>> dependencies we have  >= rather than == dependencies.
>>> 
>>> Whenever there is a new release of such dependency, it might cause chain
>>> reaction with upgrade of transitive dependencies which might get into
>>> conflict.
>>> 
>>> An example was Flask-AppBuilder vs flask-login transitive dependency with
>>> click. They started to conflict once AppBuilder has released version
>>> 1.12.0.
>>> 
>>> *Diagnosis:*
>>> Transitive dependencies with "flexible" versions (where >= is used
>> instead
>>> of ==) is a reason for "dependency hell". We will sooner or later hit
>> other
>>> cases where not fixed dependencies cause similar problems with other
>>> transitive dependencies. We need to fix-pin them. This causes problems
>> for
>>> both - released versions (cause they stop to work!) and for development
>>> (cause they break master builds in TravisCI and prevent people from
>>> installing development environment from the scratch.
>>> 
>>> *Solution:*
>>> 
>>>   - Following the old-but-good post
>>>   https://nvie.com/posts/pin-your-packages/ we are going to fix the
>>> pinned
>>>   dependencies to specific versions (so basically all dependencies are
>>>   "fixed").
>>>   - We will introduce mechanism to be able to upgrade dependencies with
>>>   pip-tools (https://github.com/jazzband/pip-tools). We might also
>> take a
>>>   look at pipenv: https://pipenv.readthedocs.io/en/latest/
>>>   - People who would like to upgrade some dependencies for their PRs
>> will
>>>   still be able to do it - but such upgrades will be in their PR thus
>> they
>>>   will go through TravisCI tests and they will also have to be specified
>>> with
>>>   pinned fixed versions (==). This should be part of review process to
>>> make
>>>   sure new/changed requirements are pinned.
>>>   - In release process there will be a point where an upgrade will be
>>>   attempted for all requirements (using pip-tools) so that we are not
>>> stuck
>>>   with older releases. This will be in controlled PR environment where
>>> there
>>>   will be time to fix all dependencies without impacting others and
>> likely
>>>   enough time to "vet" such changes (this can be done for alpha/beta
>>> releases
>>>   for example).
>>>   - As a side effect dependencies specification will become far simpler
>>>   and straightforward.
>>> 
>>> Happy to hear community comments to the proposal. I am happy to take a
>> lead
>>> on that, open JIRA issue and implement if this is something community is
>>> happy with.
>>> 
>>> J.
>>> 
>>> --
>>> 
>>> *Jarek Potiuk, Principal Software Engineer*
>>> Mobile: +48 660 796 129
>>> 
>>