OSDir


[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: Pinning dependencies for Apache Airflow


You should run `pip check` to ensure no conflicts. Pip does not do this on
its own.

On Thu, Oct 4, 2018 at 9:20 AM Jarek Potiuk <Jarek.Potiuk@xxxxxxxxxxx>
wrote:

> Great that this discussion already happened :). Lots of useful things in
> it. And yes - it means pinning in requirement.txt - this is how pip-tools
> work.
>
> J.
>
> Principal Software Engineer
> Phone: +48660796129
>
> On Thu, 4 Oct 2018, 18:14 Arthur Wiedmer, <arthur.wiedmer@xxxxxxxxx>
> wrote:
>
> > Hi Jarek,
> >
> > I will +1 the discussion Dan is referring to and George's advice.
> >
> > I just want to double check we are talking about pinning in
> > requirements.txt only.
> >
> > This offers the ability to
> > pip install -r requirements.txt
> > pip install --no-deps airflow
> > For a guaranteed install which works.
> >
> > Several different requirement files can be provided for specific use
> cases,
> > like a stable dev one for instance for people wanting to work on
> operators
> > and non-core functions.
> >
> > However, I think we should proactively test in CI against unpinned
> > dependencies (though it might be a separate case in the matrix) , so that
> > we get advance warning if possible that things will break.
> > CI downtime is not a bad thing here, it actually caught a problem :)
> >
> > We should unpin as possible in setup.py to only maintain minimum required
> > compatibility. The process of pinning in setup.py is extremely
> detrimental
> > when you have a large number of python libraries installed with different
> > pinned versions.
> >
> > Best,
> > Arthur
> >
> > On Thu, Oct 4, 2018 at 8:36 AM Dan Davydov <ddavydov@xxxxxxxxxxx.invalid
> >
> > wrote:
> >
> > > Relevant discussion about this:
> > >
> > >
> >
> https://github.com/apache/incubator-airflow/pull/1809#issuecomment-257502174
> > >
> > > On Thu, Oct 4, 2018 at 11:25 AM Jarek Potiuk <Jarek.Potiuk@xxxxxxxxxxx
> >
> > > wrote:
> > >
> > > > TL;DR; A change is coming in the way how dependencies/requirements
> are
> > > > specified for Apache Airflow - they will be fixed rather than
> flexible
> > > (==
> > > > rather than >=).
> > > >
> > > > This is follow up after Slack discussion we had with Ash and Kaxil -
> > > > summarising what we propose we'll do.
> > > >
> > > > *Problem:*
> > > > During last few weeks we experienced quite a few downtimes of
> TravisCI
> > > > builds (for all PRs/branches including master) as some of the
> > transitive
> > > > dependencies were automatically upgraded. This because in a number of
> > > > dependencies we have  >= rather than == dependencies.
> > > >
> > > > Whenever there is a new release of such dependency, it might cause
> > chain
> > > > reaction with upgrade of transitive dependencies which might get into
> > > > conflict.
> > > >
> > > > An example was Flask-AppBuilder vs flask-login transitive dependency
> > with
> > > > click. They started to conflict once AppBuilder has released version
> > > > 1.12.0.
> > > >
> > > > *Diagnosis:*
> > > > Transitive dependencies with "flexible" versions (where >= is used
> > > instead
> > > > of ==) is a reason for "dependency hell". We will sooner or later hit
> > > other
> > > > cases where not fixed dependencies cause similar problems with other
> > > > transitive dependencies. We need to fix-pin them. This causes
> problems
> > > for
> > > > both - released versions (cause they stop to work!) and for
> development
> > > > (cause they break master builds in TravisCI and prevent people from
> > > > installing development environment from the scratch.
> > > >
> > > > *Solution:*
> > > >
> > > >    - Following the old-but-good post
> > > >    https://nvie.com/posts/pin-your-packages/ we are going to fix the
> > > > pinned
> > > >    dependencies to specific versions (so basically all dependencies
> are
> > > >    "fixed").
> > > >    - We will introduce mechanism to be able to upgrade dependencies
> > with
> > > >    pip-tools (https://github.com/jazzband/pip-tools). We might also
> > > take a
> > > >    look at pipenv: https://pipenv.readthedocs.io/en/latest/
> > > >    - People who would like to upgrade some dependencies for their PRs
> > > will
> > > >    still be able to do it - but such upgrades will be in their PR
> thus
> > > they
> > > >    will go through TravisCI tests and they will also have to be
> > specified
> > > > with
> > > >    pinned fixed versions (==). This should be part of review process
> to
> > > > make
> > > >    sure new/changed requirements are pinned.
> > > >    - In release process there will be a point where an upgrade will
> be
> > > >    attempted for all requirements (using pip-tools) so that we are
> not
> > > > stuck
> > > >    with older releases. This will be in controlled PR environment
> where
> > > > there
> > > >    will be time to fix all dependencies without impacting others and
> > > likely
> > > >    enough time to "vet" such changes (this can be done for alpha/beta
> > > > releases
> > > >    for example).
> > > >    - As a side effect dependencies specification will become far
> > simpler
> > > >    and straightforward.
> > > >
> > > > Happy to hear community comments to the proposal. I am happy to take
> a
> > > lead
> > > > on that, open JIRA issue and implement if this is something community
> > is
> > > > happy with.
> > > >
> > > > J.
> > > >
> > > > --
> > > >
> > > > *Jarek Potiuk, Principal Software Engineer*
> > > > Mobile: +48 660 796 129
> > > >
> > >
> >
>