[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: Pinning dependencies for Apache Airflow

Relevant discussion about this:

On Thu, Oct 4, 2018 at 11:25 AM Jarek Potiuk <Jarek.Potiuk@xxxxxxxxxxx>

> TL;DR; A change is coming in the way how dependencies/requirements are
> specified for Apache Airflow - they will be fixed rather than flexible (==
> rather than >=).
> This is follow up after Slack discussion we had with Ash and Kaxil -
> summarising what we propose we'll do.
> *Problem:*
> During last few weeks we experienced quite a few downtimes of TravisCI
> builds (for all PRs/branches including master) as some of the transitive
> dependencies were automatically upgraded. This because in a number of
> dependencies we have  >= rather than == dependencies.
> Whenever there is a new release of such dependency, it might cause chain
> reaction with upgrade of transitive dependencies which might get into
> conflict.
> An example was Flask-AppBuilder vs flask-login transitive dependency with
> click. They started to conflict once AppBuilder has released version
> 1.12.0.
> *Diagnosis:*
> Transitive dependencies with "flexible" versions (where >= is used instead
> of ==) is a reason for "dependency hell". We will sooner or later hit other
> cases where not fixed dependencies cause similar problems with other
> transitive dependencies. We need to fix-pin them. This causes problems for
> both - released versions (cause they stop to work!) and for development
> (cause they break master builds in TravisCI and prevent people from
> installing development environment from the scratch.
> *Solution:*
>    - Following the old-but-good post
>    https://nvie.com/posts/pin-your-packages/ we are going to fix the
> pinned
>    dependencies to specific versions (so basically all dependencies are
>    "fixed").
>    - We will introduce mechanism to be able to upgrade dependencies with
>    pip-tools (https://github.com/jazzband/pip-tools). We might also take a
>    look at pipenv: https://pipenv.readthedocs.io/en/latest/
>    - People who would like to upgrade some dependencies for their PRs will
>    still be able to do it - but such upgrades will be in their PR thus they
>    will go through TravisCI tests and they will also have to be specified
> with
>    pinned fixed versions (==). This should be part of review process to
> make
>    sure new/changed requirements are pinned.
>    - In release process there will be a point where an upgrade will be
>    attempted for all requirements (using pip-tools) so that we are not
> stuck
>    with older releases. This will be in controlled PR environment where
> there
>    will be time to fix all dependencies without impacting others and likely
>    enough time to "vet" such changes (this can be done for alpha/beta
> releases
>    for example).
>    - As a side effect dependencies specification will become far simpler
>    and straightforward.
> Happy to hear community comments to the proposal. I am happy to take a lead
> on that, open JIRA issue and implement if this is something community is
> happy with.
> J.
> --
> *Jarek Potiuk, Principal Software Engineer*
> Mobile: +48 660 796 129