Bug#958764: python3-pip: debundled _vendor packaging approach breaks usage of pip in environments

classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view

Bug#958764: python3-pip: debundled _vendor packaging approach breaks usage of pip in environments

Paul Ivanov
Package: python3-pip
Version: 20.0.2-5
Severity: normal

Dear Maintainer,

Some tools that use pip rely on being able to use it  by adding its installed
location to the path of their environments.

One example is pipx.

$ pip3 --version
pip 20.0.2 from /usr/lib/python3/dist-packages/pip (python 3.8)

$ pip3 install pipx

$ export PATH=~/.local/bin:$PATH
$ pipx --version

$ pipx install cowsay
Traceback (most recent call last):
  File "/usr/lib/python3.8/runpy.py", line 193, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/usr/lib/python3.8/runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "/home/dummy/.local/pipx/shared/lib/python3.8/site-
packages/pip/__main__.py", line 16, in <module>
    from pip._internal.cli.main import main as _main  # isort:skip # noqa
  File "/home/dummy/.local/pipx/shared/lib/python3.8/site-
packages/pip/_internal/cli/main.py", line 10, in <module>
    from pip._internal.cli.autocompletion import autocomplete
  File "/home/dummy/.local/pipx/shared/lib/python3.8/site-
packages/pip/_internal/cli/autocompletion.py", line 9, in <module>
    from pip._internal.cli.main_parser import create_main_parser
  File "/home/dummy/.local/pipx/shared/lib/python3.8/site-
packages/pip/_internal/cli/main_parser.py", line 7, in <module>
    from pip._internal.cli import cmdoptions
  File "/home/dummy/.local/pipx/shared/lib/python3.8/site-
packages/pip/_internal/cli/cmdoptions.py", line 24, in <module>
    from pip._internal.exceptions import CommandError
  File "/home/dummy/.local/pipx/shared/lib/python3.8/site-
packages/pip/_internal/exceptions.py", line 10, in <module>
    from pip._vendor.six import iteritems
ModuleNotFoundError: No module named 'pip._vendor.six'

This was raised in a pipx issue [0] and in investigating it, I tracked it
down to the DEBUNDLE logic that gets patched to point to the python-wheels

Whereas with pip's code upstream (both vendored and debundled), adding the
site-packages folder containing a working version of pip will result in a being
able to use pip, this is not the case for Debian.

For pip as it is currently packaged in Debian, one has to also include all of
the wheels that get added to the path after importing pip._vendor. See an
example of such an approach in [1].

Would it be possible to keep the expected functionality provided by pip as
released upstream, perhaps by symlinking the wheels into the pip._vendor

0. https://github.com/pipxproject/pipx/issues/386
1. https://github.com/pipxproject/pipx/pull/388


-- System Information:
Debian Release: bullseye/sid
  APT prefers unstable
  APT policy: (500, 'unstable'), (500, 'stable')
Architecture: amd64 (x86_64)

Kernel: Linux 5.5.0-2-amd64 (SMP w/8 CPU cores)
Locale: LANG=en_US.UTF-8, LC_CTYPE=en_US.UTF-8 (charmap=UTF-8), LANGUAGE=en_US.UTF-8 (charmap=UTF-8)
Shell: /bin/sh linked to /usr/bin/dash
Init: systemd (via /run/systemd/system)
LSM: AppArmor: enabled

Versions of packages python3-pip depends on:
ii  ca-certificates     20190110
ii  python-pip-whl      20.0.2-5
ii  python3             3.8.2-3
ii  python3-distutils   3.8.2-2
ii  python3-setuptools  45.2.0-1
ii  python3-wheel       0.34.2-1

Versions of packages python3-pip recommends:
ii  build-essential  12.8
ii  python3-dev      3.8.2-3

python3-pip suggests no packages.

-- no debconf information

Reply | Threaded
Open this post in threaded view

Bug#958816: re Bug 958816: pipx makes unsupported use of pip internals

Scott Kitterman-5
Additional feedback from pip upstream:

> <dstufft> what pipx is doing is actually a bit more interesting than I
> thought <dstufft> they're actually running pip via the command line as
> expected, but they're sort of creating a weird custom installed version in
> a way
> <dstufft> they're creating a virtual environment without pip
> installed into it, and then using a .pth file it appears to magic it up so
> that "import pip" inside that virtualenv, actually pulls from the base
> python install
> <dstufft> it doesn't really change my answer, I wouldn't
> call that supported by upstream and if that ticket had opened up on pip's
> repo due to a change we made, I would have shrugged and said sorry and
> closed it
That said, while I don't think this is a pip bug, we'll take a look and see if
we can resolve the issue as he had some suggestions for that as well.

Scott K

signature.asc (849 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view

Bug#958764: closed by Scott Kitterman <debian@kitterman.com> (re: python3-pip: debundled _vendor packaging approach breaks usage of pip in environments)

Paul Ivanov
In reply to this post by Paul Ivanov
Thanks for looking into this, Scott!

> Maintaining symlink farms is inherently difficult and unreliable.  Debian used
> to use a similar approach to support multiple python2 versions and it never
> worked well.  I would be reluctant to use such an approach.

Ok, makes sense.

> I checked with a pip upstream developer on IRC and his view is that what pipx
> is doing is not supported by upstream:
> > <ScottK> dstufft: If an program external to pip tries to use one of pip's
> > vendored libraries directsly (e.g. from pip._vendor.six import iteritems)
> > this won't work on Debian because we don't put the wheels in the pip
> > package tree (they are in /usr/share/python-wheels).  From your point of
> > view upstream, if another package is doing such a thing are they
> > unreasonably mucking about in pip internals and if it breaks it's their
> > problem or is it something reasonable that they should be able to rely on
> > and we should try to support?
> > <dstufft> ScottK: upstrem does not support people importing pip internals
> > /vendored libs
> > <dstufft> we moved *everything* into a _ directory just to make this obvious

Pipx is not importing pip internals. Pipx is simply using pip.
The way pipx is doing this is by adding the site-packages of a
working pip installation in it's ~/.local/pipx/shared folder to
the sys.path of the environments that it manages in various
~/.local/pipx/venvs folders

Pipx is trying to save from having a copy of pip in each of its
venvs. This works if the pip on the path is the  wheel as
published on PyPI, or as the vendored code with the debundled
wheels in the  pip/_vendor directory, with the debundled wheels
in the pip/_vendor directory. But this does not work with the
logic patched in Debian's packaging that points the debundled
wheels to the sys.prefix + "/share/python-wheels" , because the
sys.prefix for each of the venvs is different and will not have
copies of the vendored code.

So the question to follow up with Donald and PyPA folks is
something like: "should pip be usable by adding the site packages
of a working installation to the sys.path of another
environment?" And similarly, "should a repackaged pip wheel work
standalone" - because the pip wheel in /usr/share/python-wheels
cannot be used the same way as a regular pip wheel, it is not
standalone, and it requires making the debundled wheels which are
assumed to be on the sys.prefix + share/python-wheels

> It may be that this pipx bug is only apparent on Debian and its derivatives
> because we rely on the pip unbundle logic, but what pipx is doing is
> unsupported upstream, so I don't think we can support it either.

Again, I believe it's the additional modification to the unbundle
logic, namely that the vendored wheels will be in sys.prefix +
share/python-wheels directory *instead* of the pip/_vendor
directory that is causing this behaviour.

> Sorry I didn't have a more positive answer.

I appreciate that, thank you for your time and efforts.


Reply | Threaded
Open this post in threaded view

Bug#958764: [Python-modules-team] Bug#958764: closed by Scott Kitterman <debian@kitterman.com> (re: python3-pip: debundled _vendor packaging approach breaks usage of pip in environments)

Scott Kitterman-5
I do have something that might address this, but I'm reluctant to promise anything until I test it.

Scott K