OSDir


[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: Pinning dependencies for Apache Airflow


Great that this discussion already happened :). Lots of useful things in
it. And yes - it means pinning in requirement.txt - this is how pip-tools
work.

J.

Principal Software Engineer
Phone: +48660796129

On Thu, 4 Oct 2018, 18:14 Arthur Wiedmer, <arthur.wiedmer@xxxxxxxxx> wrote:

> Hi Jarek,
>
> I will +1 the discussion Dan is referring to and George's advice.
>
> I just want to double check we are talking about pinning in
> requirements.txt only.
>
> This offers the ability to
> pip install -r requirements.txt
> pip install --no-deps airflow
> For a guaranteed install which works.
>
> Several different requirement files can be provided for specific use cases,
> like a stable dev one for instance for people wanting to work on operators
> and non-core functions.
>
> However, I think we should proactively test in CI against unpinned
> dependencies (though it might be a separate case in the matrix) , so that
> we get advance warning if possible that things will break.
> CI downtime is not a bad thing here, it actually caught a problem :)
>
> We should unpin as possible in setup.py to only maintain minimum required
> compatibility. The process of pinning in setup.py is extremely detrimental
> when you have a large number of python libraries installed with different
> pinned versions.
>
> Best,
> Arthur
>
> On Thu, Oct 4, 2018 at 8:36 AM Dan Davydov <ddavydov@xxxxxxxxxxx.invalid>
> wrote:
>
> > Relevant discussion about this:
> >
> >
> https://github.com/apache/incubator-airflow/pull/1809#issuecomment-257502174
> >
> > On Thu, Oct 4, 2018 at 11:25 AM Jarek Potiuk <Jarek.Potiuk@xxxxxxxxxxx>
> > wrote:
> >
> > > TL;DR; A change is coming in the way how dependencies/requirements are
> > > specified for Apache Airflow - they will be fixed rather than flexible
> > (==
> > > rather than >=).
> > >
> > > This is follow up after Slack discussion we had with Ash and Kaxil -
> > > summarising what we propose we'll do.
> > >
> > > *Problem:*
> > > During last few weeks we experienced quite a few downtimes of TravisCI
> > > builds (for all PRs/branches including master) as some of the
> transitive
> > > dependencies were automatically upgraded. This because in a number of
> > > dependencies we have  >= rather than == dependencies.
> > >
> > > Whenever there is a new release of such dependency, it might cause
> chain
> > > reaction with upgrade of transitive dependencies which might get into
> > > conflict.
> > >
> > > An example was Flask-AppBuilder vs flask-login transitive dependency
> with
> > > click. They started to conflict once AppBuilder has released version
> > > 1.12.0.
> > >
> > > *Diagnosis:*
> > > Transitive dependencies with "flexible" versions (where >= is used
> > instead
> > > of ==) is a reason for "dependency hell". We will sooner or later hit
> > other
> > > cases where not fixed dependencies cause similar problems with other
> > > transitive dependencies. We need to fix-pin them. This causes problems
> > for
> > > both - released versions (cause they stop to work!) and for development
> > > (cause they break master builds in TravisCI and prevent people from
> > > installing development environment from the scratch.
> > >
> > > *Solution:*
> > >
> > >    - Following the old-but-good post
> > >    https://nvie.com/posts/pin-your-packages/ we are going to fix the
> > > pinned
> > >    dependencies to specific versions (so basically all dependencies are
> > >    "fixed").
> > >    - We will introduce mechanism to be able to upgrade dependencies
> with
> > >    pip-tools (https://github.com/jazzband/pip-tools). We might also
> > take a
> > >    look at pipenv: https://pipenv.readthedocs.io/en/latest/
> > >    - People who would like to upgrade some dependencies for their PRs
> > will
> > >    still be able to do it - but such upgrades will be in their PR thus
> > they
> > >    will go through TravisCI tests and they will also have to be
> specified
> > > with
> > >    pinned fixed versions (==). This should be part of review process to
> > > make
> > >    sure new/changed requirements are pinned.
> > >    - In release process there will be a point where an upgrade will be
> > >    attempted for all requirements (using pip-tools) so that we are not
> > > stuck
> > >    with older releases. This will be in controlled PR environment where
> > > there
> > >    will be time to fix all dependencies without impacting others and
> > likely
> > >    enough time to "vet" such changes (this can be done for alpha/beta
> > > releases
> > >    for example).
> > >    - As a side effect dependencies specification will become far
> simpler
> > >    and straightforward.
> > >
> > > Happy to hear community comments to the proposal. I am happy to take a
> > lead
> > > on that, open JIRA issue and implement if this is something community
> is
> > > happy with.
> > >
> > > J.
> > >
> > > --
> > >
> > > *Jarek Potiuk, Principal Software Engineer*
> > > Mobile: +48 660 796 129
> > >
> >
>