Changes in dpkg Pre-Depends

classic Classic list List threaded Threaded
33 messages Options
12
Reply | Threaded
Open this post in threaded view
|

Changes in dpkg Pre-Depends

Guillem Jover
Hi!

As I'd like to change some Pre-Depends in dpkg, I'm bringing this up
here for discussion, as per policy §3.5 and given dpkg “Essential: yes”
nature.


First, I'd like to change the dpkg Pre-Depends from lzma to xz-utils,
the latter is a bit bigger in size (lzma 172 KiB; xz-utils 504 KiB,
160 KiB in share/doc/ and liblzma2 304 KiB, 124 KiB in share/doc/) but
it supports both “.lzma” and “.xz” formats, in the imminent future
dpkg-deb will get switched to directly use liblzma-dev so that one will
disappear, Jonathan Nieder has already sent some patches for that, they
need some changes first, so probably after next release (after 1.15.6).


Second, I'd like to switch from statically to dynamically linking
against zlib and libbz2, eventually liblzma too (affecting dpkg-deb)
and libselinux (affecting dpkg itself only on Linux). Here's the
arguments I know of against and in favour, with rebuttals:

Robustness (against)
~~~~~~~~~~

It's one of the few native packages (if not the only one) that is still
statically linking. One of the reasons for that has been for robustness
for several scenarios of brokenness. Let's see:

  * If dynamic linking does not work in general for whatever reason,
    then libc would be affected, and mostly nothing would work anyway.
  * If one of the specific shared libraries breaks due to the file
    being disappeared/broken/corrupted then executing dpkg or dpkg-deb
    would fail, but so would lots of other things, and installing a
    package might be the lesser of your problems.
  * If one of the specific shared libraries breaks due to broken ABI,
    or an implementation bug, then it might only affect the code using
    those libraries, which might not trigger at all (say one does not
    have SE Linux enabled, or does not install a bzip2 compressed
    package).

Statically linking might allow an easier recovery of the system, but
with enough skills most broken systems can be recovered, and the less
skilled users will not be able to w/o assistance anyway, and most
probably their GUI environments will not work either. This is though
the strongest argument about keeping status quo, and personally is the
one which has made me hesitate for a long time in proposing such change,
but if we cannot safely use shared libraries at this point in time,
then maybe we are doing something wrong.

There's also the argument that dpkg internally uses other commands to
perform operations (namely tar, find, rm and sh) and if any of those
breaks, dpkg will not be able to operate at all in some circumstances.

Increase in dpkg dependencies (against)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Statically linking might make backporting or just installing a newer
dpkg.deb in an previous Debian release easier. Except for liblzma which
is not yet in a stable release zlib, libbz2 and libselinux have long
been there, and it would be a matter of not using symbols too new to
avoid getting a versioned dependency on a package not in stable.

Increase in the pseudo-essential set (neutral)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

zlib and libselinux are already part of the pseudo-essential set.
Dynamically linking against libbz2 and liblzma would pull those as new
pseudo-essential dependencies, but then the size just gets shifted from
dpkg to the actual shared library packages, and in addition it allows
others to make use of them, and not only dpkg (by way of having them
embedded while statically linked).

  zlib       ← util-linux
  libselinux ← coreutils, sed, util-linux, libpam-modules, sysvinit

Size reduction (favour)
~~~~~~~~~~~~~~

It will make dpkg.deb slightly smaller, although not much noticeable
given the huge percentage taken presently by l10n files, but it is
somehow significant for a stripped binary (with locales and docs
removed, but not man pages). This will make dpkg.deb get closer to be
usable in really space constrained environments, the next reduction
will be when switching to use libdpkg.so. After some more culling it
might even be a possible candidate to replace stuff like udpkg, ipkg,
opkg and friends. For the actual size changes, taking numbers from
current master:

  w/ static linking:

  -rwxr-xr-x root/root    408488 2010-02-19 20:04 ./usr/bin/dpkg
  -rwxr-xr-x root/root    257608 2010-02-19 20:04 ./usr/bin/dpkg-deb

  -rw-r--r--  1 guillem guillem 533642 Feb 19 20:30 dpkg-strip-static.deb

  w/ dynamic linking:

  -rwxr-xr-x root/root    219488 2010-02-19 20:13 ./usr/bin/dpkg
  -rwxr-xr-x root/root    126824 2010-02-19 20:13 ./usr/bin/dpkg-deb

  -rw-r--r--  1 guillem guillem 380694 Feb 19 20:30 dpkg-strip-dynamic.deb

Security (favour)
~~~~~~~~

Avoids the usual security problems when dealing with statically linked
code.

Avoiding static PIC libraries (favour)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

When dpkg switches to use a shared libdpkg, those libraries will need
to be linked dynamically anyway, or we'd have to produce PIC enabled
static libraries (ugh), which I'd strongly be against.


So if there's no strong reason not to, I'd like to do these changes for
next dpkg 1.15.6 release. Note though that they are not interdependent,
and if we decided so, one could be done while not the other.


thanks,
guillem


--
To UNSUBSCRIBE, email to [hidden email]
with a subject of "unsubscribe". Trouble? Contact [hidden email]
Archive: http://lists.debian.org/20100219231510.GA7308@...

Reply | Threaded
Open this post in threaded view
|

Re: Changes in dpkg Pre-Depends

Keegan Quinn-4
Guillem Jover wrote:
> It's one of the few native packages (if not the only one) that is still
> statically linking.

dpkg appears to be dynamically linked on my unstable/amd64 box:

keegan@keegan:~$ file /usr/bin/dpkg
/usr/bin/dpkg: ELF 64-bit LSB executable, x86-64, version 1 (SYSV),
dynamically linked (uses shared libs), for GNU/Linux 2.6.18, stripped

Am I missing something? Apologies in advance if so.

  - Keegan

--
Keegan Quinn  <[hidden email]>
Software Support Engineer
Storix, Inc.
(619) 543-0200


--
To UNSUBSCRIBE, email to [hidden email]
with a subject of "unsubscribe". Trouble? Contact [hidden email]
Archive: http://lists.debian.org/4B7F3376.9050409@...

Reply | Threaded
Open this post in threaded view
|

Re: Changes in dpkg Pre-Depends

Peter Samuelson

[Keegan Quinn]
> Guillem Jover wrote:
> >It's one of the few native packages (if not the only one) that is still
> >statically linking.
>
> dpkg appears to be dynamically linked on my unstable/amd64 box:

It is dynamically linked to some libraries, staticly linked to others.
Guillem is proposing to dynamically link to all of them.

I agree with everything in the original proposal.  The main reason not
to link dpkg dynamically to libbz2 etc. is robustness - but I haven't
worried about Debian's library handling robustness for a long time.
(When was the last time I installed 'sash'?  8 years ago?)
--
Peter Samuelson | org-tld!p12n!peter | http://p12n.org/


--
To UNSUBSCRIBE, email to [hidden email]
with a subject of "unsubscribe". Trouble? Contact [hidden email]
Archive: http://lists.debian.org/20100220015259.GE18278@...

Reply | Threaded
Open this post in threaded view
|

Re: Changes in dpkg Pre-Depends

Goswin von Brederlow-2
Peter Samuelson <[hidden email]> writes:

> [Keegan Quinn]
>> Guillem Jover wrote:
>> >It's one of the few native packages (if not the only one) that is still
>> >statically linking.
>>
>> dpkg appears to be dynamically linked on my unstable/amd64 box:
>
> It is dynamically linked to some libraries, staticly linked to others.
> Guillem is proposing to dynamically link to all of them.
>
> I agree with everything in the original proposal.  The main reason not
> to link dpkg dynamically to libbz2 etc. is robustness - but I haven't
> worried about Debian's library handling robustness for a long time.
> (When was the last time I installed 'sash'?  8 years ago?)

It might be worth noting that the most recent critical failure of dpkg,
failing to work with kernel < 2.6.22, wasn't prevented by having static
libs. The code blew up, not the linking.

And since you mention sash there could be a sdpkg. That would avoid
needing *_pic.a libs for libdpkg.so.

MfG
        Goswin


--
To UNSUBSCRIBE, email to [hidden email]
with a subject of "unsubscribe". Trouble? Contact [hidden email]
Archive: http://lists.debian.org/878waozfaw.fsf@...

Reply | Threaded
Open this post in threaded view
|

Re: Changes in dpkg Pre-Depends

Stefano Zacchiroli
In reply to this post by Guillem Jover
On Sat, Feb 20, 2010 at 12:15:10AM +0100, Guillem Jover wrote:
> First, I'd like to change the dpkg Pre-Depends from lzma to xz-utils,
> the latter is a bit bigger in size (lzma 172 KiB; xz-utils 504 KiB,
> 160 KiB in share/doc/ and liblzma2 304 KiB, 124 KiB in share/doc/) but

This just seems the obvious right thing™ to do, given the fate of lzma
itself.

> Second, I'd like to switch from statically to dynamically linking
> against zlib and libbz2, eventually liblzma too (affecting dpkg-deb)
> and libselinux (affecting dpkg itself only on Linux). Here's the
> arguments I know of against and in favour, with rebuttals:

I'm personally convinced by your arguments. Still, I'd like if you
consider the transitional idea of having---say, for a release---two
different binary packages shipping dpkg: "dpkg" (essential: yes) and
"dpkg-static" (essential: no), the latter containing a fully statically
linked version of dpkg, coming as /usr/bin/dpkg-static.

I've seen this for other safety-critical tools, e.g. the dar backup tool
which comes both as "dar" and "dar-static". I personally don't believe
there would be *much* use of "dpkg-static", but having it around for a
release would enable to see if/how many (paranoid) people actually
install it. Would that make sense in your opinion? Would it be worth?

Cheers.

--
Stefano Zacchiroli -o- PhD in Computer Science \ PostDoc @ Univ. Paris 7
zack@{upsilon.cc,pps.jussieu.fr,debian.org} -<>- http://upsilon.cc/zack/
Dietro un grande uomo c'è ..|  .  |. Et ne m'en veux pas si je te tutoie
sempre uno zaino ...........| ..: |.... Je dis tu à tous ceux que j'aime

signature.asc (197 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Changes in dpkg Pre-Depends

Marco d'Itri
On Feb 20, Stefano Zacchiroli <[hidden email]> wrote:

> I've seen this for other safety-critical tools, e.g. the dar backup tool
> which comes both as "dar" and "dar-static". I personally don't believe
> there would be *much* use of "dpkg-static", but having it around for a
> release would enable to see if/how many (paranoid) people actually
> install it. Would that make sense in your opinion? Would it be worth?
I don't think so. Can you think of some real life disaster scenarios
which would benefit from a static dpkg?
And in that case, why it would not be simpler to copy the dpkg binary
and the few libraries it depends on from another system?

--
ciao,
Marco

signature.asc (205 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Changes in dpkg Pre-Depends

Guillem Jover
In reply to this post by Stefano Zacchiroli
On Sat, 2010-02-20 at 10:07:45 +0100, Stefano Zacchiroli wrote:

> On Sat, Feb 20, 2010 at 12:15:10AM +0100, Guillem Jover wrote:
> > Second, I'd like to switch from statically to dynamically linking
> > against zlib and libbz2, eventually liblzma too (affecting dpkg-deb)
> > and libselinux (affecting dpkg itself only on Linux). Here's the
> > arguments I know of against and in favour, with rebuttals:
>
> I'm personally convinced by your arguments. Still, I'd like if you
> consider the transitional idea of having---say, for a release---two
> different binary packages shipping dpkg: "dpkg" (essential: yes) and
> "dpkg-static" (essential: no), the latter containing a fully statically
> linked version of dpkg, coming as /usr/bin/dpkg-static.
>
> I've seen this for other safety-critical tools, e.g. the dar backup tool
> which comes both as "dar" and "dar-static". I personally don't believe
> there would be *much* use of "dpkg-static", but having it around for a
> release would enable to see if/how many (paranoid) people actually
> install it. Would that make sense in your opinion? Would it be worth?

I don't think this would be worth it, as Marco has also said, if the
system is hosed but you can still get to the point of obtaining a
package to install you might as well just obtain the broken files.
Of course you might have it already on apt's cache, but that's not
usually the case when you need it. :)

Also I'm guessing if it's there some people will end up installing it,
either on purpose or by accident, and trying to remove it later might
find some resistance (from the former).

Something that came to me after having sent the mail is that we have
programs way more critical that are dynamically linking with all used
libraries, like the ones involved in the boot sequence: init (libc,
libsepol, libselinux), mount (libc, libblkid, libuuid, libsepol,
libselinux), udevd (libc, libselinux) and there's not been the need
to provide static versions for them, and I would tend to think an
unbootable system is trickier to recover than one where you cannot
install new packages.

thanks,
guillem


--
To UNSUBSCRIBE, email to [hidden email]
with a subject of "unsubscribe". Trouble? Contact [hidden email]
Archive: http://lists.debian.org/20100223042013.GB29243@...

Reply | Threaded
Open this post in threaded view
|

Re: Changes in dpkg Pre-Depends

Guillem Jover
In reply to this post by Guillem Jover
On Sat, 2010-02-20 at 00:15:10 +0100, Guillem Jover wrote:
> First, I'd like to change the dpkg Pre-Depends from lzma to xz-utils,
> the latter is a bit bigger in size (lzma 172 KiB; xz-utils 504 KiB,
> 160 KiB in share/doc/ and liblzma2 304 KiB, 124 KiB in share/doc/)

Regarding xz-utils' size, I was just checking and it seems not all
programs are linking against liblzma, by passing --enable-dynamic to
both configure lines the package gets reduced to 396 KiB. Also by
not shipping xzdec and lzmadec the package gets down to 324 KiB.

Jonathan, any reason why those are not dynamically linked? Also what's
the point of shipping the xzdec and lzmadec tools in the main package?
It seems they would make sense as part of another package as tiny
decompressors-only, but no in addition to the main tools as those
already provide the whole functionality?

thanks,
guillem


--
To UNSUBSCRIBE, email to [hidden email]
with a subject of "unsubscribe". Trouble? Contact [hidden email]
Archive: http://lists.debian.org/20100223044512.GC29243@...

Reply | Threaded
Open this post in threaded view
|

Re: Changes in dpkg Pre-Depends

Jonathan Nieder
Guillem Jover wrote:

> Regarding xz-utils' size, I was just checking and it seems not all
> programs are linking against liblzma, by passing --enable-dynamic to
> both configure lines the package gets reduced to 396 KiB. Also by
> not shipping xzdec and lzmadec the package gets down to 324 KiB.
>
> Jonathan, any reason why those are not dynamically linked?

The usual i386-centric reason: the PIC version is (~5%) slower than
the non-PIC version.  See PACKAGERS in the source, section 4.1.
I considered doing this only on i386, but since I only have an i386 to
test on, I would worry about missing packaging bugs.

I could very easily be convinced to switch to dynamic linking.  Please
file a bug (with what you just said about size and any other reasons).


--
To UNSUBSCRIBE, email to [hidden email]
with a subject of "unsubscribe". Trouble? Contact [hidden email]
Archive: http://lists.debian.org/20100223050612.GA8060@...

Reply | Threaded
Open this post in threaded view
|

Bug#571060: xz-utils: give xzdec and lzmadec their own package

Jonathan Nieder
In reply to this post by Guillem Jover
Package: xz-utils
Version: 4.999.9beta+20100212-1
Severity: wishlist

Guillem Jover wrote:

> Also what's
> the point of shipping the xzdec and lzmadec tools in the main package?
> It seems they would make sense as part of another package as tiny
> decompressors-only, but no in addition to the main tools as those
> already provide the whole functionality?

Indeed.

xzdec and lzmadec are meant mostly for copying to small media with
lots of xz- or lzma-compressed files (for distribution).  They don’t
belong in the xz-utils package, so let’s move them.

Thanks,
Jonathan



--
To UNSUBSCRIBE, email to [hidden email]
with a subject of "unsubscribe". Trouble? Contact [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Changes in dpkg Pre-Depends

Robert Collins
In reply to this post by Guillem Jover
On Tue, 2010-02-23 at 05:20 +0100, Guillem Jover wrote:

> I don't think this would be worth it, as Marco has also said, if the
> system is hosed but you can still get to the point of obtaining a
> package to install you might as well just obtain the broken files.
> Of course you might have it already on apt's cache, but that's not
> usually the case when you need it. :)

dpkg is useful for more than just installing new packages; a hosed
system might be fixed by removing a package, running maintainer scripts,
both of which dpkg is the right tool for.

Not arguing for or against, just noting that you're being overly
restrictive in your analysis of what a sdpkg might be used for.

-Rob

signature.asc (205 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Changes in dpkg Pre-Depends

Don Armstrong
In reply to this post by Guillem Jover
On Sat, 20 Feb 2010, Guillem Jover wrote:
> First, I'd like to change the dpkg Pre-Depends from lzma to xz-utils,

[...]

> Second, I'd like to switch from statically to dynamically linking
> against zlib and libbz2, eventually liblzma too (affecting dpkg-deb)
> and libselinux (affecting dpkg itself only on Linux). Here's the
> arguments I know of against and in favour, with rebuttals:

[...]
 
> So if there's no strong reason not to, I'd like to do these changes
> for next dpkg 1.15.6 release. Note though that they are not
> interdependent, and if we decided so, one could be done while not
> the other.

I don't have any objections to this, but I'd strongly suggest that
this get a run-through experimental with an announcement on
-devel-announce to request testing so that any really bad problems are
caught before it gets deployed more widely.


Don Armstrong

--
Who is thinking this?
I am.
 -- Greg Egan _Diaspora_ p38

http://www.donarmstrong.com              http://rzlab.ucr.edu


--
To UNSUBSCRIBE, email to [hidden email]
with a subject of "unsubscribe". Trouble? Contact [hidden email]
Archive: http://lists.debian.org/20100223065155.GD3437@...

Reply | Threaded
Open this post in threaded view
|

Re: Changes in dpkg Pre-Depends

Marco d'Itri
In reply to this post by Jonathan Nieder
On Feb 23, Jonathan Nieder <[hidden email]> wrote:

> The usual i386-centric reason: the PIC version is (~5%) slower than
> the non-PIC version.  See PACKAGERS in the source, section 4.1.
> I considered doing this only on i386, but since I only have an i386 to
> test on, I would worry about missing packaging bugs.
Using non-PIC code for a 5% speed up looks like an acceptable trade off
to me, but it really must be restricted only to architectures which
need it.
But there is no reason to statically link the packages: you can still
build the library non-PIC and dinamically link programs with it: this
will waste RAM as if they were statically linked, but at least will
save disk space.
Beware: this does not work on all architectures, so I think should be
enabled only for i386.

If this will cause bugs then somebody will report them, this is what
users are for... :-)

--
ciao,
Marco

signature.asc (205 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Changes in dpkg Pre-Depends

Josselin Mouette
Le mardi 23 février 2010 à 14:43 +0100, Marco d'Itri a écrit :
> Using non-PIC code for a 5% speed up looks like an acceptable trade off
> to me, but it really must be restricted only to architectures which
> need it.

Those who worry about a 5% speedup should use amd64. Which is an
architecture that doesn’t need it.

--
 .''`.      Josselin Mouette
: :' :
`. `'   “I recommend you to learn English in hope that you in
  `-     future understand things”  -- Jörg Schilling


--
To UNSUBSCRIBE, email to [hidden email]
with a subject of "unsubscribe". Trouble? Contact [hidden email]
Archive: http://lists.debian.org/1266935046.2655.12.camel@tomoyo

Reply | Threaded
Open this post in threaded view
|

Re: Changes in dpkg Pre-Depends

Marco d'Itri
On Feb 23, Josselin Mouette <[hidden email]> wrote:

> Le mardi 23 février 2010 à 14:43 +0100, Marco d'Itri a écrit :
> > Using non-PIC code for a 5% speed up looks like an acceptable trade off
> > to me, but it really must be restricted only to architectures which
> > need it.
> Those who worry about a 5% speedup should use amd64. Which is an
> architecture that doesn???t need it.
Cool, where can I send you the invoice for the replacement hardware?

--
ciao,
Marco

signature.asc (205 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Changes in dpkg Pre-Depends

Thomas Weber-8
On Tue, Feb 23, 2010 at 04:01:51PM +0100, Marco d'Itri wrote:
> On Feb 23, Josselin Mouette <[hidden email]> wrote:
>
> > Le mardi 23 février 2010 à 14:43 +0100, Marco d'Itri a écrit :
> > > Using non-PIC code for a 5% speed up looks like an acceptable trade off
> > > to me, but it really must be restricted only to architectures which
> > > need it.
> > Those who worry about a 5% speedup should use amd64. Which is an
> > architecture that doesn???t need it.
> Cool, where can I send you the invoice for the replacement hardware?

You have x86 hardware that is so old that it doesn't run amd64, but at
the same moment you care about speed?

        Thomas


--
To UNSUBSCRIBE, email to [hidden email]
with a subject of "unsubscribe". Trouble? Contact [hidden email]
Archive: http://lists.debian.org/20100223175800.GA12391@atlan

Reply | Threaded
Open this post in threaded view
|

Optimization for slow platforms (Re: Changes in dpkg Pre-Depends)

Jonathan Nieder
Thomas Weber wrote:
>>> Le mardi 23 février 2010 à 14:43 +0100, Marco d'Itri a écrit :
>>>> Using non-PIC code for a 5% speed up looks like an acceptable trade off
>>>> to me, but it really must be restricted only to architectures which
>>>> need it.
[...]
> You have x86 hardware that is so old that it doesn't run amd64, but at
> the same moment you care about speed?

Yes, it is when you are stuck on a computer like this one that slow
programs really start to feel painful.

IMHO the choice of what to optimize should be based on what code is
used most often (or whose running time is particularly harmful for
other reasons), not what happens to be running on the most expensive
hardware.

In this case the speed difference from using non-PIC code is
noticeable.  But the memory pressure from not sharing code between
processes might mean it is not worth it --- I am really torn.

Jonathan


--
To UNSUBSCRIBE, email to [hidden email]
with a subject of "unsubscribe". Trouble? Contact [hidden email]
Archive: http://lists.debian.org/20100223183137.GA7962@...

Reply | Threaded
Open this post in threaded view
|

Re: Changes in dpkg Pre-Depends

Marco d'Itri
In reply to this post by Thomas Weber-8
On Feb 23, Thomas Weber <[hidden email]> wrote:

> You have x86 hardware that is so old that it doesn't run amd64, but at
> the same moment you care about speed?
Why should I not care about speed if the hardware is slow?
Anyway, there are often good reasons to use x86 on modern hardware
(think about laptops and smaller VPSes).

--
ciao,
Marco

signature.asc (205 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Optimization for slow platforms (Re: Changes in dpkg Pre-Depends)

Marco d'Itri
In reply to this post by Jonathan Nieder
On Feb 23, Jonathan Nieder <[hidden email]> wrote:

> In this case the speed difference from using non-PIC code is
> noticeable.  But the memory pressure from not sharing code between
> processes might mean it is not worth it --- I am really torn.
If the programs are linked statically then they will have the same
memory footprint of programs linked with non-PIC libraries, so the
situation will not be worse in this sense.

--
ciao,
Marco

signature.asc (205 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Changes in dpkg Pre-Depends

Josselin Mouette
In reply to this post by Marco d'Itri
Le mardi 23 février 2010 à 20:32 +0100, Marco d'Itri a écrit :
> Anyway, there are often good reasons to use x86 on modern hardware
> (think about laptops and smaller VPSes).

You mean, like saving memory?

Wait… wouldn’t you save more memory by using shared libraries and PIC
code?

--
 .''`.      Josselin Mouette
: :' :
`. `'   “I recommend you to learn English in hope that you in
  `-     future understand things”  -- Jörg Schilling


--
To UNSUBSCRIBE, email to [hidden email]
with a subject of "unsubscribe". Trouble? Contact [hidden email]
Archive: http://lists.debian.org/1266956184.8746.0.camel@tomoyo

12