dpkg-shlibdeps couldn't find library ... warning vs error

classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

dpkg-shlibdeps couldn't find library ... warning vs error

Crim, Christopher
I am trying to understand the regex rules in the function split_soname from
dkpg-shlibdeps.pl.  Working with an old corporate code base that has yet to
adopt any versioning, all of the shared objects are named <object>.so.  When
attempting to package on a box that does not yet have the dependent packages
installed dpkg-shlibdeps will throw warnings, "dpkg-shlibdeps: warning:
couldn't find library <lib>.so needed by ..." but packaging will still succeed.  
We have few shared objects that have ".so." as part of their name.  When dpkg-
shlibdeps encounters these dependencies the warning becomes an error, "dpkg-
shlibdeps: error: couldn't find library <lib>.so needed by ..." and packaging
fails.  I traced this down to the split_soname function which checks 2 regex
comparisons.  The shared objects that only throw warnings fall into the "else"
clause of the function.  The ones that fail fall into the first if clause which
as regex 1.  What were these regexs meant to match?

regex 1 (/^(.*)\.so\.(.*)$/):
was this meant to capture shared objects with so number and revisons such as
libc.6 or libbz2.so.1.0.4?  If so, should the values after "so" be restricted
to numbers?

regex 2 (/^(.*)-(\d.*)\.so$/)
The terminating so on this regex helps limit its capture but again should the
groups be limited to digits only?

 I have tried searching for inforamtion related to to naming conventions for
shared library names but the best I can come up with is that they need to be
prefixed with "lib" and suffixed with ".so" and possibly the so number and
revision.  A name such as "libcom.debian.foo.bar.so" should be valid.  If not
I would appreciate being pointed to such requirements.

I am willing to create a patch but before I assume I know what it "should" be
doing I wanted to see how the maintainers felt.


Reply | Threaded
Open this post in threaded view
|

Re: dpkg-shlibdeps couldn't find library ... warning vs error

Guillem Jover
Hi!

On Wed, 2019-06-26 at 14:58:34 -0400, Crim, Christopher wrote:

> I am trying to understand the regex rules in the function split_soname from
> dkpg-shlibdeps.pl.  Working with an old corporate code base that has yet to
> adopt any versioning, all of the shared objects are named <object>.so.  When
> attempting to package on a box that does not yet have the dependent packages
> installed dpkg-shlibdeps will throw warnings, "dpkg-shlibdeps: warning:
> couldn't find library <lib>.so needed by ..." but packaging will still succeed.  
> We have few shared objects that have ".so." as part of their name.  When dpkg-
> shlibdeps encounters these dependencies the warning becomes an error, "dpkg-
> shlibdeps: error: couldn't find library <lib>.so needed by ..." and packaging
> fails.  I traced this down to the split_soname function which checks 2 regex
> comparisons.  The shared objects that only throw warnings fall into the "else"
> clause of the function.  The ones that fail fall into the first if clause which
> as regex 1.  What were these regexs meant to match?

If the installed packages have no shlibs or symbols file information
then dpkg-shlibdeps cannot know what dependency information to inject,
which means the generated binaries will end up with missing/incorrect
dependencies. See below for the distinction.

> regex 1 (/^(.*)\.so\.(.*)$/):
> was this meant to capture shared objects with so number and revisons such as
> libc.6 or libbz2.so.1.0.4?  If so, should the values after "so" be restricted
> to numbers?

It cannot be restricted to numbers only because the version might
include other characters.

This format is the one used commonly for public and stable shared
libraries, where the format is <name>.so.<version>, and then there's
at least one symlink of the form <name>.so pointing to the former and
used by other objects to link to, that gets them the latest <version>.

The key here is that the SONAME might only include part of the
SOVERSION, depending on where the <name>.so symlink points to, so you
could have:

  libfoo.so.1.2.3
  libfoo.so.1
  libfoo.so → libfoo.1

Because these are public interfaces they are expected to have
dependency information available.

> regex 2 (/^(.*)-(\d.*)\.so$/)
> The terminating so on this regex helps limit its capture but again should the
> groups be limited to digits only?

This is the format commonly used for private and/or unstable shared
libraries, where the whole version is encoded in the SONAME. So you
could have:

  libbar-1.2.3.so
  libbar.so → libbar-1.2.3.so (optionally, otherwise you might be
                               required to specify the whole SONAME)

As these are private/unstable interfaces they are to be used at your
own risk, so that's why they are just warnings.

>  I have tried searching for inforamtion related to to naming conventions for
> shared library names but the best I can come up with is that they need to be
> prefixed with "lib" and suffixed with ".so" and possibly the so number and
> revision.  A name such as "libcom.debian.foo.bar.so" should be valid.  If not
> I would appreciate being pointed to such requirements.

I'm not sure now, there's anywhere properly documenting these details.
I don't think the debian-policy manual contains rationale for these.
I've for now locallt documented the split_soname() function, willcheck
whether one of the man pages can also be improved.

Hope that helps. Even though I'm not sure it resolved your issue at
hand? :)

Thanks,
Guillem

Reply | Threaded
Open this post in threaded view
|

Re: dpkg-shlibdeps couldn't find library ... warning vs error

Russ Allbery-2
Guillem Jover <[hidden email]> writes:
> On Wed, 2019-06-26 at 14:58:34 -0400, Crim, Christopher wrote:

>> I have tried searching for inforamtion related to to naming conventions
>> for shared library names but the best I can come up with is that they
>> need to be prefixed with "lib" and suffixed with ".so" and possibly the
>> so number and revision.  A name such as "libcom.debian.foo.bar.so"
>> should be valid.  If not I would appreciate being pointed to such
>> requirements.

> I'm not sure now, there's anywhere properly documenting these details.
> I don't think the debian-policy manual contains rationale for these.
> I've for now locallt documented the split_soname() function, willcheck
> whether one of the man pages can also be improved.

Policy does not document these details either.  It mentions the two normal
conventions (libfoo.so.<version> and libfoo-<version>.so), but only in the
context of helping packagers understand what to look for when figuring out
the SONAME of a library.  It sounds like packaging may actually fail if a
package ships shared libraries that are too far afield of the normal
conventions because dpkg-shlibdeps imposes some restrictions, so maybe it
would be worth adding some details there.  But I'm not sure how often this
comes up.

Usually I refer people to the Libtool documentation on maintaining the
SONAME, which documents two specific and reasonable conventions in the
context of how Libtool manages the version information.

--
Russ Allbery ([hidden email])               <http://www.eyrie.org/~eagle/>

Reply | Threaded
Open this post in threaded view
|

Re: dpkg-shlibdeps couldn't find library ... warning vs error

Crim, Christopher
In reply to this post by Guillem Jover
Thank you Guillem for your response.  I have a few followups that I have put
inline below.

On Thursday, June 27, 2019 9:09:34 PM EDT Guillem Jover wrote:

> Hi!
>
> On Wed, 2019-06-26 at 14:58:34 -0400, Crim, Christopher wrote:
> > I am trying to understand the regex rules in the function split_soname
> > from
> > dkpg-shlibdeps.pl.  Working with an old corporate code base that has yet
> > to
> > adopt any versioning, all of the shared objects are named <object>.so.
> > When attempting to package on a box that does not yet have the dependent
> > packages installed dpkg-shlibdeps will throw warnings, "dpkg-shlibdeps:
> > warning: couldn't find library <lib>.so needed by ..." but packaging will
> > still succeed. We have few shared objects that have ".so." as part of
Just to be clear, an example name would be "libcom.foo.so.system.utility.so".  
The reason we had to add the domain prefix was that we were running into name
collisions.  "libcom.bar.ss.pos.utility.so" would have conflicted with the
above without the domains.

> > their name.  When dpkg- shlibdeps encounters these dependencies the
> > warning becomes an error, "dpkg- shlibdeps: error: couldn't find library
> > <lib>.so needed by ..." and packaging fails.  I traced this down to the
> > split_soname function which checks 2 regex comparisons.  The shared
> > objects that only throw warnings fall into the "else" clause of the
> > function.  The ones that fail fall into the first if clause which as
> > regex 1.  What were these regexs meant to match?
>
> If the installed packages have no shlibs or symbols file information
> then dpkg-shlibdeps cannot know what dependency information to inject,
> which means the generated binaries will end up with missing/incorrect
> dependencies. See below for the distinction.
>
> > regex 1 (/^(.*)\.so\.(.*)$/):
> > was this meant to capture shared objects with so number and revisons such
> > as libc.6 or libbz2.so.1.0.4?  If so, should the values after "so" be
> > restricted to numbers?
>
> It cannot be restricted to numbers only because the version might
> include other characters.
I agree now, I did find some examples on my system:
libtidy.so.5deb1.6.0
libKPimSMTP.so.5.9.3.abi1

>
> This format is the one used commonly for public and stable shared
> libraries, where the format is <name>.so.<version>, and then there's
> at least one symlink of the form <name>.so pointing to the former and
> used by other objects to link to, that gets them the latest <version>.
>
> The key here is that the SONAME might only include part of the
> SOVERSION, depending on where the <name>.so symlink points to, so you
> could have:
>
>   libfoo.so.1.2.3
>   libfoo.so.1
>   libfoo.so → libfoo.1
>
> Because these are public interfaces they are expected to have
> dependency information available.
>
> > regex 2 (/^(.*)-(\d.*)\.so$/)
> > The terminating so on this regex helps limit its capture but again should
> > the groups be limited to digits only?
>
> This is the format commonly used for private and/or unstable shared
> libraries, where the whole version is encoded in the SONAME. So you
> could have:
I was not aware of this distinction.  A quick search did not point me to
anything stating that.  Does this mean that libc on my machine is unstable?
(libc-2.28.so)

>
>   libbar-1.2.3.so
>   libbar.so → libbar-1.2.3.so (optionally, otherwise you might be
>                                required to specify the whole SONAME)
>
> As these are private/unstable interfaces they are to be used at your
> own risk, so that's why they are just warnings.
These do not throw warnings though, they throw errors.  At link time, if the
library depended on had an soname that name would be put in the NEEDED field.  
If the library depended on did not have an soname or any version info then the
just the name would be in the NEEDED field.  
libfoo.so.1 will fail
libbar-1.2.3.so will fail
libbaz.so will just be a warning

So it can't be that dpkg-shlibdeps only wants to throw warnings for private/
unstable libraries and errors for everything else.  I guess this gets to the
heart of my question, why is there a distinction.  If it cannot find a
dependency should it just always fail?

>
> >  I have tried searching for inforamtion related to to naming conventions
> >  for
> >
> > shared library names but the best I can come up with is that they need to
> > be prefixed with "lib" and suffixed with ".so" and possibly the so number
> > and revision.  A name such as "libcom.debian.foo.bar.so" should be valid.
> >  If not I would appreciate being pointed to such requirements.
>
> I'm not sure now, there's anywhere properly documenting these details.
> I don't think the debian-policy manual contains rationale for these.
> I've for now locallt documented the split_soname() function, willcheck
> whether one of the man pages can also be improved.
>
> Hope that helps. Even though I'm not sure it resolved your issue at
> hand? :)
>
> Thanks,
> Guillem


signature.asc (235 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: dpkg-shlibdeps couldn't find library ... warning vs error

Guillem Jover
Hi!

[ Just rereading my reply, I should stop sending mail with a sleep deprived
  brain, sorry! :) ]

On Fri, 2019-06-28 at 08:09:11 -0400, Crim, Christopher wrote:

> On Thursday, June 27, 2019 9:09:34 PM EDT Guillem Jover wrote:
> > On Wed, 2019-06-26 at 14:58:34 -0400, Crim, Christopher wrote:
> > > I am trying to understand the regex rules in the function split_soname
> > > from
> > > dkpg-shlibdeps.pl.  Working with an old corporate code base that has yet
> > > to
> > > adopt any versioning, all of the shared objects are named <object>.so.
> > > When attempting to package on a box that does not yet have the dependent
> > > packages installed dpkg-shlibdeps will throw warnings, "dpkg-shlibdeps:
> > > warning: couldn't find library <lib>.so needed by ..." but packaging will
> > > still succeed. We have few shared objects that have ".so." as part of

> Just to be clear, an example name would be "libcom.foo.so.system.utility.so".  
> The reason we had to add the domain prefix was that we were running into name
> collisions.  "libcom.bar.ss.pos.utility.so" would have conflicted with the
> above without the domains.

I think this is a pretty uncommon practice. I'd probably recommend
using «-» instead of «.» to avoid possible matches if there's a «.so.»
inbetween. And to add a version too.

> > > their name.  When dpkg- shlibdeps encounters these dependencies the
> > > warning becomes an error, "dpkg- shlibdeps: error: couldn't find library
> > > <lib>.so needed by ..." and packaging fails.  I traced this down to the
> > > split_soname function which checks 2 regex comparisons.  The shared
> > > objects that only throw warnings fall into the "else" clause of the
> > > function.  The ones that fail fall into the first if clause which as
> > > regex 1.  What were these regexs meant to match?
> >
> > If the installed packages have no shlibs or symbols file information
> > then dpkg-shlibdeps cannot know what dependency information to inject,
> > which means the generated binaries will end up with missing/incorrect
> > dependencies. See below for the distinction.
> >
> > > regex 1 (/^(.*)\.so\.(.*)$/):
> > > was this meant to capture shared objects with so number and revisons such
> > > as libc.6 or libbz2.so.1.0.4?  If so, should the values after "so" be
> > > restricted to numbers?
> >
> > It cannot be restricted to numbers only because the version might
> > include other characters.
>
> I agree now, I did find some examples on my system:
> libtidy.so.5deb1.6.0
> libKPimSMTP.so.5.9.3.abi1

Right.

> > This format is the one used commonly for public and stable shared
> > libraries, where the format is <name>.so.<version>, and then there's
> > at least one symlink of the form <name>.so pointing to the former and
> > used by other objects to link to, that gets them the latest <version>.

This part is correct.

> > The key here is that the SONAME might only include part of the
> > SOVERSION, depending on where the <name>.so symlink points to, so you
> > could have:
> >
> >   libfoo.so.1.2.3
> >   libfoo.so.1
> >   libfoo.so → libfoo.1

This is very confused. The unversioned symlink (or what it resolves to)
does not need to have a direct relation to the SONAME, which is what gets
encoded in the shared library at link time usually via a command-line
argument.

So for example (following the one you queried about):

  $ realpath /lib/x86_64-linux-gnu/libc.so.6
  /lib/x86_64-linux-gnu/libc-2.28.so
  $ objdump -p /lib/x86_64-linux-gnu/libc.so.6 | grep SONAME
  SONAME               libc.so.6

In this case libc uses (for historical reasons) a filename encoding
the project version name (which is distinct to its SOVERSION), but its
SONAME encodes the arch-specific SOVERSION. The actual requirement
here is that the unversioned «<name>.so» used during compile/link-time
refers to the version intended to be linked against, and that the
SONAME can then be found at run-time, directly or via symlinks.

> > Because these are public interfaces they are expected to have
> > dependency information available.

More or less I guess.

> > > regex 2 (/^(.*)-(\d.*)\.so$/)
> > > The terminating so on this regex helps limit its capture but again should
> > > the groups be limited to digits only?

> > This is the format commonly used for private and/or unstable shared
> > libraries, where the whole version is encoded in the SONAME. So you
> > could have:

I think this is also correct in general.

> I was not aware of this distinction.  A quick search did not point me to
> anything stating that.  Does this mean that libc on my machine is unstable?
> (libc-2.28.so)

> >   libbar-1.2.3.so
> >   libbar.so → libbar-1.2.3.so (optionally, otherwise you might be
> >                                required to specify the whole SONAME)

An example of this could be:

  $ realpath /usr/lib/x86_64-linux-gnu/libbfd-2.31.1-system.so
  /usr/lib/x86_64-linux-gnu/libbfd-2.31.1-system.so
  $ objdump -p /usr/lib/x86_64-linux-gnu/libbfd-2.31.1-system.so | grep SONAME
  SONAME               libbfd-2.31.1-system.so

In this case there's no unversioned «<name>.so» symlink, because this
is a private interface, so projects need to know the exact version to
link to.

> > As these are private/unstable interfaces they are to be used at your
> > own risk, so that's why they are just warnings.

This is partial non-sense, sorry. :) The first part holds, but not the
second.

> These do not throw warnings though, they throw errors.  At link time, if the
> library depended on had an soname that name would be put in the NEEDED field.  
> If the library depended on did not have an soname or any version info then the
> just the name would be in the NEEDED field.  
> libfoo.so.1 will fail
> libbar-1.2.3.so will fail
> libbaz.so will just be a warning

Yes, sorry. The recognized formats throw errors, the unknown ones
warnings. It is expected that both recognized formats provide
dependency information in the form of shlibs or symbols files.

> So it can't be that dpkg-shlibdeps only wants to throw warnings for private/
> unstable libraries and errors for everything else.  I guess this gets to the
> heart of my question, why is there a distinction.  If it cannot find a
> dependency should it just always fail?

Yes, sorry for the confusion. :)

> > I'm not sure now, there's anywhere properly documenting these details.
> > I don't think the debian-policy manual contains rationale for these.
> > I've for now locallt documented the split_soname() function, willcheck
> > whether one of the man pages can also be improved.

(Some heavy typos in there :)

Thanks,
Guillem