python package management confusion
On Sunday, 20 January 2019 20:38:30 UTC, Oscar Benjamin wrote:
> On Sun, 20 Jan 2019 at 16:22, dcs3spp via Python-list
> <python-list at python.org> wrote:
> > On Saturday, 19 January 2019 11:17:19 UTC, dcs3spp wrote:
> > >
> > > My question is, can setuptools be configured to pull in child from a separate git repository when running python setup.py develop from parent folder? I have since found and tried this approach at https://stackoverflow.com/a/53706140/8325270
> > > It appears that later versions of setuptools can install via a PEP508 url. I currently trying to investigate this approach?..
> > After trying PEP508 url approach my conclusions are as follows.
> > A PEP508 url for a git repository can be used in *install_requires* of *setup.py*. An example is listed below.
> > ```
> > requires = [
> > 'parent',
> > 'kinto-http at git+https://github.com/Kinto/kinto-http.py',
> > ]
> > ...
> > install_requires=requires
> > ```
> > The package can then be installed with pip, using ```pip install -e . or pip install .```
> > However, installation with setuptools is then broken, i.e. ```python setup.py develop``` and ```python setup.py install``` fails. setuptools looks for packages in pypi indexes. To install using setuptools a devpi index would have to be installed and configured or packages would have to installed from a paid for pypi repository in the cloud. Alternatively, developers could manually install each private package dependency individually, prior to running ```python setup.py develop``` for the source package. Unless, there are other alternative(s) such as zc.buildout with mr developer plugin etc.....
> > If I want to have a Python private project, referencing other private project(s), available under source control and CI via gitlab.com, it seems that I can use the pip approach with PEP508 or use a requirements.txt file containing the git projects referenced as PEP508 urls, i.e. ```pip install -r requirements.txt```.
> > Confusion, stems from the fact that pip and setuptools dependencies are then not synchronised, i.e. setuptools will break if PEP508 urls are listed for install_requires. Presumably the approach is to use either pip or setuptools but not both?
> I'm not sure what you mean by pip and setuptools not being
> synchronised. Pip depends on setuptools and cannot be used without it.
> Both setuptools and pip are maintained under the PyPA. They are
> intended to work together. in fact if your setup.py uses distutils
> instead of setuptools then pip will "inject" setuptools into it to
> ensure that meets pip's needs.
> You will need to be more specific about which commands you are using
> and what it is that becomes unsynchronised.
Have since done further testing and figured out how I can install from a setup.py file using both pip and python setup.py develop. Confusion was caused by trying to install using both pip and setuptools with PEP508 urls in install_requires, see note at end of post....
>From what I understand setuptools offers *dependency_links* as a list of dependency urls. In the example below, the *pyramid_core* package is a private dependency that I have written.
I am currently using pip 18.1. pip has an option *--process-dependencies* that issues a deprecation warning. The following *setup.py* example works with both setuptools (python setup develop etc.) and pip (pip install -e . and pip install .).
The example *setup.py* below can be installed using both setuptools and pip as follows:
python setup.py develop
python setup.py install
pip install -e . --process-dependency-links
pip install .
**setup.py that is compatible with both setuptools and pip 18.1**
from setuptools import setup, find_packages
here = os.path.abspath(os.path.dirname(__file__))
with open(os.path.join(here, 'README.md')) as f:
README = f.read()
with open(os.path.join(here, 'CHANGES.md')) as f:
CHANGES = f.read()
dependencies = [
'git+ssh://git at gitlab.com/dcs3spp/plantoeducate_core.git#egg=pyramid_core-0',
requires = [
setup_requires = [
tests_require = [
long_description=README + '\n\n' + CHANGES,
"Programming Language :: Python",
author_email='myemail at outlook.com',
Pip 18.1 supports reading pep508 direct urls from install_requires. In future release there are plans to deprecate the --process-dependency-links pip install option:
Will setuptools provide ability to use direct pep508 urls in install_requires in the future also?