OSDir


[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Pinning dependencies for Apache Airflow


TL;DR; A change is coming in the way how dependencies/requirements are
specified for Apache Airflow - they will be fixed rather than flexible (==
rather than >=).

This is follow up after Slack discussion we had with Ash and Kaxil -
summarising what we propose we'll do.

*Problem:*
During last few weeks we experienced quite a few downtimes of TravisCI
builds (for all PRs/branches including master) as some of the transitive
dependencies were automatically upgraded. This because in a number of
dependencies we have  >= rather than == dependencies.

Whenever there is a new release of such dependency, it might cause chain
reaction with upgrade of transitive dependencies which might get into
conflict.

An example was Flask-AppBuilder vs flask-login transitive dependency with
click. They started to conflict once AppBuilder has released version
1.12.0.

*Diagnosis:*
Transitive dependencies with "flexible" versions (where >= is used instead
of ==) is a reason for "dependency hell". We will sooner or later hit other
cases where not fixed dependencies cause similar problems with other
transitive dependencies. We need to fix-pin them. This causes problems for
both - released versions (cause they stop to work!) and for development
(cause they break master builds in TravisCI and prevent people from
installing development environment from the scratch.

*Solution:*

   - Following the old-but-good post
   https://nvie.com/posts/pin-your-packages/ we are going to fix the pinned
   dependencies to specific versions (so basically all dependencies are
   "fixed").
   - We will introduce mechanism to be able to upgrade dependencies with
   pip-tools (https://github.com/jazzband/pip-tools). We might also take a
   look at pipenv: https://pipenv.readthedocs.io/en/latest/
   - People who would like to upgrade some dependencies for their PRs will
   still be able to do it - but such upgrades will be in their PR thus they
   will go through TravisCI tests and they will also have to be specified with
   pinned fixed versions (==). This should be part of review process to make
   sure new/changed requirements are pinned.
   - In release process there will be a point where an upgrade will be
   attempted for all requirements (using pip-tools) so that we are not stuck
   with older releases. This will be in controlled PR environment where there
   will be time to fix all dependencies without impacting others and likely
   enough time to "vet" such changes (this can be done for alpha/beta releases
   for example).
   - As a side effect dependencies specification will become far simpler
   and straightforward.

Happy to hear community comments to the proposal. I am happy to take a lead
on that, open JIRA issue and implement if this is something community is
happy with.

J.

-- 

*Jarek Potiuk, Principal Software Engineer*
Mobile: +48 660 796 129