Python3 modules not built for all supported Python versions

classic Classic list List threaded Threaded
8 messages Options
Reply | Threaded
Open this post in threaded view
|

Python3 modules not built for all supported Python versions

Emilio Pozuelo Monfort-4
Hi,

We've just finished the transition to python3.8 as the default python3
interpreter, which was a bit difficult due to some autopkgtest regressions in a
few rdeps, and to the fact that many modules only build their extensions for the
default python version, which means they have a strict dependency on the python3
version[1] and they need to be rebuilt and migrated in lockstep with
python3-defaults.

I have analyzed this based on current sid amd64 contents and have come up with
the following packages that don't ship extensions for both py3.7 and 3.8 (which
are the currently supported versions). Note that pure python packages that don't
build C extensions are not affected.

It would be great if this situation can be improved in order to help with future
python transitions. Building for all the supported python versions can be done
by build-depending on python3-all-dev and compiling your package (or just the
python bits) with PYTHON pointing to each version. Depending on your package's
build system, this could be largely automated using some helper, such as
pybuild. If you don't know how to add support for your package, feel free to ask.

Cheers,
Emilio

[1] e.g. python3 (>= 3.7), python3 (<< 3.8)



dd-list.txt (17K) Download Attachment
packages.txt (7K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Python3 modules not built for all supported Python versions

Matthias Klose
On 3/30/20 1:24 PM, Emilio Pozuelo Monfort wrote:

> Hi,
>
> We've just finished the transition to python3.8 as the default python3
> interpreter, which was a bit difficult due to some autopkgtest regressions in a
> few rdeps, and to the fact that many modules only build their extensions for the
> default python version, which means they have a strict dependency on the python3
> version[1] and they need to be rebuilt and migrated in lockstep with
> python3-defaults.
>
> I have analyzed this based on current sid amd64 contents and have come up with
> the following packages that don't ship extensions for both py3.7 and 3.8 (which
> are the currently supported versions). Note that pure python packages that don't
> build C extensions are not affected.
>
> It would be great if this situation can be improved in order to help with future
> python transitions. Building for all the supported python versions can be done
> by build-depending on python3-all-dev and compiling your package (or just the
> python bits) with PYTHON pointing to each version. Depending on your package's
> build system, this could be largely automated using some helper, such as
> pybuild. If you don't know how to add support for your package, feel free to ask.

Yes, that would be useful.  However there are only limited time windows when you
can do *and* check such work, i.e. when the python3-defaults package has two
supported python3 versions.  Filing and fixing these issues would be nice.

Plus for most of those packages you have to modify the build system to build the
package for each python3 version, or to just build the extension for the
non-default version.

I assume we still could keep 3.7 as a supported python3 version for some time,
if people want to work on those issues.  But people probably won't have the time
to work on that when we introduce 3.9.

Matthias

Reply | Threaded
Open this post in threaded view
|

Re: Python3 modules not built for all supported Python versions

Johannes Schauer-3
In reply to this post by Emilio Pozuelo Monfort-4
Hi,

Quoting Emilio Pozuelo Monfort (2020-03-30 13:24:01)

> We've just finished the transition to python3.8 as the default python3
> interpreter, which was a bit difficult due to some autopkgtest regressions in
> a few rdeps, and to the fact that many modules only build their extensions
> for the default python version, which means they have a strict dependency on
> the python3 version[1] and they need to be rebuilt and migrated in lockstep
> with python3-defaults.
>
> I have analyzed this based on current sid amd64 contents and have come up with
> the following packages that don't ship extensions for both py3.7 and 3.8 (which
> are the currently supported versions). Note that pure python packages that don't
> build C extensions are not affected.
>
> It would be great if this situation can be improved in order to help with future
> python transitions. Building for all the supported python versions can be done
> by build-depending on python3-all-dev and compiling your package (or just the
> python bits) with PYTHON pointing to each version. Depending on your package's
> build system, this could be largely automated using some helper, such as
> pybuild. If you don't know how to add support for your package, feel free to ask.
does this mean that build-depending on python3-dev is wrong in general and
should instead be replaced by build-depending on python3-all-dev?

Could wrong build dependencies or binary packages which have a strict
dependency on a python3 version not easily be detected by lintian?

I'm not an expert in Python module packaging -- it just so happens that some
packages I maintain offer Python bindings. Do you have a diff of a source
package that successfully made the transition so that I know what I would have
to change? For example the package src:ros-geometry2 has a super simple
dh-style rules file, basically just doing:

%:
        dh $@ --buildsystem=cmake --with python3

What would I have to change to successfully fix this problem?

Thanks!

cheers, josch

signature.asc (849 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Python3 modules not built for all supported Python versions

Simon McVittie-7
On Mon, 30 Mar 2020 at 15:30:01 +0200, Johannes Schauer wrote:
> does this mean that build-depending on python3-dev is wrong in general and
> should instead be replaced by build-depending on python3-all-dev?

It is only wrong for packages that build Python 3 extensions (binary
modules) that are intended to be loadable by all supported Python
3 versions (roughly: `find /usr/lib/python3/dist-packages -name '*.so'`).

For packages that embed Python 3, like the versions of vim that
have Python scripting support, or packages that use a Python 3
extension as an internal implementation detail of some tool, like
gobject-introspection, my understanding is that build-depending
on python3-dev continues to be appropriate. These extensions would
ideally be installed in a private directory, like gobject-introspection's
/usr/lib/x86_64-linux-gnu/gobject-introspection/giscanner/_giscanner.cpython-38-x86_64-linux-gnu.so
- but I know some upstreams and some downstream maintainers (arguably
incorrectly) package private extensions as though they were public
extensions, because the mechanics of doing so are much simpler.

> For example the package src:ros-geometry2 has a super simple
> dh-style rules file, basically just doing:
>
> %:
> dh $@ --buildsystem=cmake --with python3
>
> What would I have to change to successfully fix this problem?

The general answer is that you would have to build it repeatedly in a
loop, with each supported version of Python 3 in turn. I am not aware
of a way to do this in a similarly simple rules file.

    smcv

Reply | Threaded
Open this post in threaded view
|

Re: Python3 modules not built for all supported Python versions

Emilio Pozuelo Monfort-4
On 30/03/2020 16:08, Simon McVittie wrote:

> On Mon, 30 Mar 2020 at 15:30:01 +0200, Johannes Schauer wrote:
>> does this mean that build-depending on python3-dev is wrong in general and
>> should instead be replaced by build-depending on python3-all-dev?
>
> It is only wrong for packages that build Python 3 extensions (binary
> modules) that are intended to be loadable by all supported Python
> 3 versions (roughly: `find /usr/lib/python3/dist-packages -name '*.so'`).
>
> For packages that embed Python 3, like the versions of vim that
> have Python scripting support, or packages that use a Python 3
> extension as an internal implementation detail of some tool, like
> gobject-introspection, my understanding is that build-depending
> on python3-dev continues to be appropriate. These extensions would
> ideally be installed in a private directory, like gobject-introspection's
> /usr/lib/x86_64-linux-gnu/gobject-introspection/giscanner/_giscanner.cpython-38-x86_64-linux-gnu.so
> - but I know some upstreams and some downstream maintainers (arguably
> incorrectly) package private extensions as though they were public
> extensions, because the mechanics of doing so are much simpler.
>
>> For example the package src:ros-geometry2 has a super simple
>> dh-style rules file, basically just doing:
>>
>> %:
>> dh $@ --buildsystem=cmake --with python3
>>
>> What would I have to change to successfully fix this problem?
>
> The general answer is that you would have to build it repeatedly in a
> loop, with each supported version of Python 3 in turn. I am not aware
> of a way to do this in a similarly simple rules file.
I've heard pybuild now has a cmake backend, so theoretically you could do
something like

%:
        dh $@ --buildsystem=pybuild --system=cmake

Since I couldn't find any package in the archive that used pybuild with the
cmake backend, I have looked at this specific package and came up with the
attached debdiff. I added the dh_auto_install hack because the cmake build
system creates the python extension as _tf2.so, and is only renamed by
dh_python3 (as can be seen in current build logs). However when building for
both python3.7 and python3.8, the last installation will override the former,
and only one _tf2.so will be available.

However that workaround proved to not be enough, because dh_install3 renames
both the 3.7 and 3.8 versions as 3.8:

make[2]: Leaving directory '/build/ros-geometry2-0.6.6/.pybuild/cpython3_3.7/build'
I: pybuild pybuild:310: dh_python3
I: dh_python3 fs:343: renaming _tf2.so to _tf2.cpython-38-x86_64-linux-gnu.so
[...]
make[2]: Leaving directory '/build/ros-geometry2-0.6.6/.pybuild/cpython3_3.8/build'
I: pybuild pybuild:310: dh_python3
W: dh_python3 fs:340: destination file exist, cannot rename _tf2.so to
_tf2.cpython-38-x86_64-linux-gnu.so

Resulting in these installed files:

-rw-r--r-- root/root     69112 2020-03-30 14:56
./usr/lib/python3/dist-packages/tf2_py/_tf2.cpython-38-x86_64-linux-gnu.so
-rw-r--r-- root/root     69128 2020-03-30 14:56
./usr/lib/python3/dist-packages/tf2_py/_tf2.so

I don't know if I'm missing an argument to dh_python3 so that it knows the
python version, or even if there's a better workaround. But perhaps pybuild
should be doing this automatically between the dh_auto_install calls so that
this kind of workarounds aren't necessary.

Piotr, is this a bug in pybuild, or am I doing something wrong?

Cheers,
Emilio

ros-geometry2.debdiff (2K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Python3 modules not built for all supported Python versions

Piotr Ozarowski-3
> I don't know if I'm missing an argument to dh_python3 so that it knows the
> python version, or even if there's a better workaround. But perhaps pybuild

better workaround is to force cmake to install into versioned
dist-packages (that's what I do in distutils) as I didn't find a
reliable way to read Python version/architecture/tag/etc. from .so file

> should be doing this automatically between the dh_auto_install calls so that
> this kind of workarounds aren't necessary.

it's actually a nice idea - pybuild can check if
/usr/lib/python3.X/dist-packages/ is empty and there are .so files in
/usr/lib/python3/dist-packages/ and move them temporarily to
/usr/lib/python3.X/dist-packages/ (it knows which interpreter is used at
this point) and let dh_python3 rename them later

> Piotr, is this a bug in pybuild, or am I doing something wrong?

well, cmake is not really easy to work with, they provide
-DPYTHON_LIBRARY:FILEPATH… but they don't use it or use completely
different name… it basically is different for each library I checked
(and in different official documentation versions I read, sic!)

Reply | Threaded
Open this post in threaded view
|

Re: Python3 modules not built for all supported Python versions

Jochen Sprickerhof-5
In reply to this post by Emilio Pozuelo Monfort-4
Hi Emilio,

* Emilio Pozuelo Monfort <[hidden email]> [2020-03-30 17:47]:
>I've heard pybuild now has a cmake backend, so theoretically you could do
>something like
>
>%:
> dh $@ --buildsystem=pybuild --system=cmake
[..]
>I don't know if I'm missing an argument to dh_python3 so that it knows the
>python version, or even if there's a better workaround. But perhaps pybuild
>should be doing this automatically between the dh_auto_install calls so that
>this kind of workarounds aren't necessary.

Thanks for looking into this and providing a patch!
I polished it a little:

https://salsa.debian.org/science-team/ros-geometry2/-/commit/11739091abe58007485772492fddbdc1a12a59c6

and uploaded a working version to the archive. Will fix all ros-*
packages now.

Cheers Jochen

signature.asc (849 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Python3 modules not built for all supported Python versions

Scott Kitterman-5
On Monday, April 6, 2020 9:24:35 AM EDT Jochen Sprickerhof wrote:

> Hi Emilio,
>
> * Emilio Pozuelo Monfort <[hidden email]> [2020-03-30 17:47]:
> >I've heard pybuild now has a cmake backend, so theoretically you could do
> >something like
> >
> >%:
> > dh $@ --buildsystem=pybuild --system=cmake
>
> [..]
>
> >I don't know if I'm missing an argument to dh_python3 so that it knows the
> >python version, or even if there's a better workaround. But perhaps pybuild
> >should be doing this automatically between the dh_auto_install calls so
> >that this kind of workarounds aren't necessary.
>
> Thanks for looking into this and providing a patch!
> I polished it a little:
>
> https://salsa.debian.org/science-team/ros-geometry2/-/commit/11739091abe5800
> 7485772492fddbdc1a12a59c6
>
> and uploaded a working version to the archive. Will fix all ros-*
> packages now.
You may have noticed that the pybuild cmake plug-in is somewhat under
documented.  While this is fresh in your mind, it might be useful for you to
consider what documentation would have made this easier for you to figure out.

https://salsa.debian.org/python-team/tools/dh-python

Scott K

signature.asc (849 bytes) Download Attachment