Re: Bug#929296: libopenblas-base: is libopenblas.so needed?

classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

Re: Bug#929296: libopenblas-base: is libopenblas.so needed?

Mo Zhou
Hi Drew,

I didn't closely investigate into the scipy bug, but I can answer
some of your questions. BTW, does anything break in a clean chroot?
I mean, making sure a thing works fine in an unclean environment
is difficult.

On 2019-05-21 04:57, Drew Parsons wrote:
> Why is /usr/lib/x86_64-linux-gnu/libopenblasp-r0.3.5.so provided by
> libopenblas-base?

Julia links against libopenblas.so.0 instead of libblas.so /
liblapack.so .
Julia's linear algebra standard library is not quite ready for
library switching.

> The problem is that libopenblasp-r0.3.5.so contains exactly the same
> symbols as libblas.so and liblapack.so.  This seems to be confusing
> linkers, e.g. see Bug#914655 for python-scipy.

The Debian dependency template resolving of the BLAS / LAPACK family
is designed to work like this:

- linked against libblas.so.3 -> enable alternative,
  resolve to libblas.so.3 | libblas.so.3

- linked against libopenblas.so.0 -> tight requirement on openblas,
  resolve to libopenblas-base

Even if libopenblas.so.0 contains the same set of symbols as
(libblas.so.3 + liblapack.so.3 + some extended routines),
the SONAME matters for our dependency tree.

> If libopenblasp-r0.3.5.so is present then it is used preferentially to
> provide BLAS and LAPACK symbols, so the shared library
> (_flapack.x86_64-linux-gnu.so in the case of python-scipy) ends up with
>   NEEDED libopenblas.so.0
> instead of
>   NEEDED liblapack.so.3
>   NEEDED libblas.so.3

libopenblasp-r0.3.5.so should never be used as BLAS / LAPACK provider
as it's SONAME unmatches the depencency template and would incur
breakage to our dependency tree.

If a program is really linked against libopenblas.so.0, liblapack.so.3,
and libblas.so.3, it is still problematic as symbol clashes between
(possibly) different implementations may lead to unexpected results.

> The official builds of scipy are fine since they build against the
> generic BLAS. But local builds can become confused, which is what I
> think happened with Bug#914655.  The package Depends: libblas3 |
> libblas.so.3 where in this case it would need Depends: libopenblas-base.
>
> So, is libopenblasp-r0.3.5.so really supposed to be provided in
> /usr/lib/x86_64-linux-gnu by libopenblas-base, or this a bug which
> should be fixed when the various BLAS flavours are being expanded?

I'm sure libopenblasp-r0.3.5.so cannot be removed. Every OpenBLAS
flavor will ship 3 shared objects:

   openblas-flavor/libopenblas*.so.*  SONAME: libopenblas.so.0
   openblas-flavor/libblas.so.3       SONAME: libblas.so.3
   openblas-flavor/liblapack.so.3     SONAME: liblapack.so.3

Reply | Threaded
Open this post in threaded view
|

Re: Bug#929296: libopenblas-base: is libopenblas.so needed?

Drew Parsons
On 2019-05-21 14:41, Mo Zhou wrote:
> Hi Drew,
>
> I didn't closely investigate into the scipy bug, but I can answer
> some of your questions. BTW, does anything break in a clean chroot?
> I mean, making sure a thing works fine in an unclean environment
> is difficult.

It seems fine in a clean chroot. In that case tt links against
liblapack.so.3 as it should.

> On 2019-05-21 04:57, Drew Parsons wrote:
>> The problem is that libopenblasp-r0.3.5.so contains exactly the same
>> symbols as libblas.so and liblapack.so.  This seems to be confusing
>> linkers, e.g. see Bug#914655 for python-scipy.
>
> The Debian dependency template resolving of the BLAS / LAPACK family
> is designed to work like this:
>
> - linked against libblas.so.3 -> enable alternative,
>   resolve to libblas.so.3 | libblas.so.3
>
> - linked against libopenblas.so.0 -> tight requirement on openblas,
>   resolve to libopenblas-base
>
> Even if libopenblas.so.0 contains the same set of symbols as
> (libblas.so.3 + liblapack.so.3 + some extended routines),
> the SONAME matters for our dependency tree.
...
> libopenblasp-r0.3.5.so should never be used as BLAS / LAPACK provider
> as it's SONAME unmatches the depencency template and would incur
> breakage to our dependency tree.

This seems to be the problem. libopenblas.so.0 is used to resolve
symbols instead of liblapack.so.3.  The symbol in question in Bug#914655
is ilaver_ which is part of lapack, not specific to libopenblas.

Perhaps our scipy build should explicitly avoid libopenblas.so by
setting
export BLAS=/path/to/libblas.so
export LAPACK=/path/to/liblapack.so
as suggested at
http://scipy.github.io/devdocs/building/linux.html#specific-instructions

Reply | Threaded
Open this post in threaded view
|

Re: Bug#929296: libopenblas-base: is libopenblas.so needed?

Mo Zhou
On 2019-05-21 09:13, Drew Parsons wrote:
>
> This seems to be the problem. libopenblas.so.0 is used to resolve
> symbols instead of liblapack.so.3.  The symbol in question in
> Bug#914655 is ilaver_ which is part of lapack, not specific to
> libopenblas.

ilaver_ is indeed a standard fortran routine provided in lapack.
The scipy package in archive looks good:

dpkg -L python3-scipy | rg '.*.so$' | xargs readelf -d | grep NEEDED |
sort | uniq

> Perhaps our scipy build should explicitly avoid libopenblas.so by setting
> export BLAS=/path/to/libblas.so
> export LAPACK=/path/to/liblapack.so
> as suggested at
> http://scipy.github.io/devdocs/building/linux.html#specific-instructions

Sounds good to me if exporting these environment variables solves the
problem.
But what happens during run time? Will scipy mix the usage of
(libopenblas.so) and (libblas.so + liblapack.so) ?

Reply | Threaded
Open this post in threaded view
|

Re: Bug#929296: libopenblas-base: is libopenblas.so needed?

Drew Parsons
On 2019-05-21 19:55, Mo Zhou wrote:

> On 2019-05-21 09:13, Drew Parsons wrote:
>
>> Perhaps our scipy build should explicitly avoid libopenblas.so by
>> setting
>> export BLAS=/path/to/libblas.so
>> export LAPACK=/path/to/liblapack.so
>> as suggested at
>> http://scipy.github.io/devdocs/building/linux.html#specific-instructions
>
> Sounds good to me if exporting these environment variables solves the
> problem.
> But what happens during run time? Will scipy mix the usage of
> (libopenblas.so) and (libblas.so + liblapack.so) ?

I don't think so. Once its built and linked, it's got the NEEDED
liblapack.so headers so no more fuss about libopenblas.so

It looks like there's something in the configuration pattern which
explicitly checks for openblas and links to it if its found.  The weird
things is there is no such explicit check in the scipy code. I wonder if
it's grabbing some setup from numpy.  Digging further, by the look of it
/usr/lib/python3/dist-packages/numpy/distutils/system_info.py is the
root of the problem.

Drew