Removing bzip2 support from apt due to rustification

classic Classic list List threaded Threaded
10 messages Options
Reply | Threaded
Open this post in threaded view
|

Removing bzip2 support from apt due to rustification

Julian Andres Klode-4
Hi folks,

seeing that Federico Mena Quintero has taken over bzip2 development
and is in the process of porting it to Rust[1], we should consider
removing bzip2 support from apt, dpkg, etc. following the release
of buster.

My understanding is that having APT depend on a library written in
Rust severily hurts its portability, and makes it hard to support
for stable releases, as Rust is a fairly fast moving target.

I do not believe that bzip2 is a useful algorithm in todays world,
and we should look at migrating any remaining bzip2-only things
(translation files I think) to xz or zstd.

Thanks,
Julian

[1] https://people.gnome.org/~federico/blog/maintaining-bzip2.html
--
debian developer - deb.li/jak | jak-linux.org - free software dev
ubuntu core developer                              i speak de, en

Reply | Threaded
Open this post in threaded view
|

Re: Removing bzip2 support from apt due to rustification

Boyuan Yang-2
在 2019-06-06四的 23:35 +0200,Julian Andres Klode写道:

> Hi folks,
>
> seeing that Federico Mena Quintero has taken over bzip2 development
> and is in the process of porting it to Rust[1], we should consider
> removing bzip2 support from apt, dpkg, etc. following the release
> of buster.
>
> My understanding is that having APT depend on a library written in
> Rust severily hurts its portability, and makes it hard to support
> for stable releases, as Rust is a fairly fast moving target.
>
> I do not believe that bzip2 is a useful algorithm in todays world,
> and we should look at migrating any remaining bzip2-only things
> (translation files I think) to xz or zstd.
I do remember there's still some source packages / binary packages in Debian
using the bzip2 format. If we are going to do that (which looks reasonable,
BTW), a serious archive-wide scan should be made in advance to see how great
the impact is and we need to deal with each occurrence.

Another issue is that the new toolchain (apt/dpkg/...) will not be able to
handle old packages using bzip2. Why not make the bzip2 support optional (like
a plugin or something similar)?

Regards,
Boyuan Yang

signature.asc (875 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Removing bzip2 support from apt due to rustification

Adam Borowski-3
On Thu, Jun 06, 2019 at 05:44:27PM -0400, Boyuan Yang wrote:
> 在 2019-06-06四的 23:35 +0200,Julian Andres Klode写道:
> > seeing that Federico Mena Quintero has taken over bzip2 development
> > and is in the process of porting it to Rust[1], we should consider
> > removing bzip2 support from apt, dpkg, etc. following the release
> > of buster.

How long are versions written in a sane language going to be security
supported?  It's not like bzip2 is a fast moving target.  Vulnerability to
malicious data is the only real concern -- a small code base can otherwise
be kept afloat for decades by just fixing FTBFSes on new compilers/archs.

> > My understanding is that having APT depend on a library written in
> > Rust severily hurts its portability, and makes it hard to support
> > for stable releases, as Rust is a fairly fast moving target.
> >
> > I do not believe that bzip2 is a useful algorithm in todays world,
> > and we should look at migrating any remaining bzip2-only things
> > (translation files I think) to xz or zstd.

Aye, per the recent thread.

> I do remember there's still some source packages / binary packages in Debian
> using the bzip2 format. If we are going to do that (which looks reasonable,
> BTW), a serious archive-wide scan should be made in advance to see how great
> the impact is and we need to deal with each occurrence.

I did such a scan just weeks before.  There's not a single _binary_ package
that uses bz2 for either the control or data tarball in stretch nor buster
(in amd64 but I doubt other archs would be different).  I did not test
jessie -- but dropping support in bullseye would mean the user needs to mix
packages 3 releases apart.  Disallowing that sounds fine to me.

On the other hand, there's a massive number of _source_ packages with bz2
components.  There's 3621 referenced bz2 files in sid.  We're not getting
rid of them anytime soon.

> Another issue is that the new toolchain (apt/dpkg/...) will not be able to
> handle old packages using bzip2. Why not make the bzip2 support optional (like
> a plugin or something similar)?

pipe|exec("bzip2") or dlopen('libbzip2') may work.

Please add support for zstd and improve either 0.939 debs or a new format
while doing so. :)


Meow!
--
⢀⣴⠾⠻⢶⣦⠀
⣾⠁⢠⠒⠀⣿⡁ Sometimes you benefit from delegating stuff.  For example,
⢿⡄⠘⠷⠚⠋⠀ this way I get to be a vegetarian.
⠈⠳⣄⠀⠀⠀⠀

Reply | Threaded
Open this post in threaded view
|

Potentially forking bzip2 (was: Re: Removing bzip2 support from apt due to rustification)

Guillem Jover
In reply to this post by Julian Andres Klode-4
Hi!

On Thu, 2019-06-06 at 23:35:28 +0200, Julian Andres Klode wrote:
> seeing that Federico Mena Quintero has taken over bzip2 development
> and is in the process of porting it to Rust[1], we should consider
> removing bzip2 support from apt, dpkg, etc. following the release
> of buster.

Argh. :( I've stated this many times in the past, but just to make it
clear, I'm not planning on removing support for extraction using
previously supported compression algorithms. bzip2 is already
considered obsolete (and disallowed) at «dpkg-deb --build» time, but
dpkg-deb will happily extract old packages using it.

> My understanding is that having APT depend on a library written in
> Rust severily hurts its portability, and makes it hard to support
> for stable releases, as Rust is a fairly fast moving target.

Yes, this would pretty much make bzip2 a (currently) non-portable
project.

So I really, really, do not want to do it, and I'd rather spend my
time elsewhere, but if it comes to it, and no one else does it before,
I guess I'll have to consider forking prior to its rustification. :(

> [1] https://people.gnome.org/~federico/blog/maintaining-bzip2.html

Thanks,
Guillem

Reply | Threaded
Open this post in threaded view
|

Re: Removing bzip2 support from apt due to rustification

Julian Andres Klode-4
In reply to this post by Boyuan Yang-2
On Thu, Jun 06, 2019 at 05:44:27PM -0400, Boyuan Yang wrote:

> 在 2019-06-06四的 23:35 +0200,Julian Andres Klode写道:
> > Hi folks,
> >
> > seeing that Federico Mena Quintero has taken over bzip2 development
> > and is in the process of porting it to Rust[1], we should consider
> > removing bzip2 support from apt, dpkg, etc. following the release
> > of buster.
> >
> > My understanding is that having APT depend on a library written in
> > Rust severily hurts its portability, and makes it hard to support
> > for stable releases, as Rust is a fairly fast moving target.
> >
> > I do not believe that bzip2 is a useful algorithm in todays world,
> > and we should look at migrating any remaining bzip2-only things
> > (translation files I think) to xz or zstd.
>
> I do remember there's still some source packages / binary packages in Debian
> using the bzip2 format. If we are going to do that (which looks reasonable,
> BTW), a serious archive-wide scan should be made in advance to see how great
> the impact is and we need to deal with each occurrence.
>
> Another issue is that the new toolchain (apt/dpkg/...) will not be able to
> handle old packages using bzip2. Why not make the bzip2 support optional (like
> a plugin or something similar)?

Both apt and dpkg are capable of falling back to executing the bzip2 binary
where available. Well, in theory anyway. I'm not sure it works reliably.

For APT, the impact of removing bzip2 support is a bit more limited
than for dpkg; it "only" breaks apt-extracttemplates, apt-ftparchive,
support for Translation-*.bz2 indexes (which are the only ones we ship
right now (Bug#930103), and python-apt clients using apt_inst to inspect
bzip2 debs.

The risk there seems limited enough, and one thing you can expect from
a 2.0 version bump.

--
debian developer - deb.li/jak | jak-linux.org - free software dev
ubuntu core developer                              i speak de, en

Reply | Threaded
Open this post in threaded view
|

Re: Removing bzip2 support from apt due to rustification

Mo Zhou
In reply to this post by Boyuan Yang-2
Hi Boyuan,

On 2019-06-06 21:44, Boyuan Yang wrote:
> I do remember there's still some source packages / binary packages in Debian
> using the bzip2 format. If we are going to do that (which looks reasonable,
> BTW), a serious archive-wide scan should be made in advance to see how great
> the impact is and we need to deal with each occurrence.

Correction: s/some packages/massive amount of packages/

I quickly scanned my local Sid (amd64+source) mirror:

/l/M/debian ❯❯❯ find . -type f -name '*.orig.tar.bz2' | wc -l
1338
/l/M/debian ❯❯❯ find . -type f -name '*.debian.tar.bz2' | wc -l
57

Reply | Threaded
Open this post in threaded view
|

Re: Removing bzip2 support from apt due to rustification

Federico Mena Quintero-2
In reply to this post by Julian Andres Klode-4
On Fri, 2019-06-07 at 11:48 +0800, Julian Andres Klode wrote:

Why am I getting BCCed on this?  Is this low-key unsolicited
complaining like in https://mastodon.social/@juliank/102226793499538013
 ?

> seeing that Federico Mena Quintero has taken over bzip2 development
> and is in the process of porting it to Rust[1], we should consider
> removing bzip2 support from apt, dpkg, etc. following the release
> of buster.

What Debian decides to do is up to them.  Please keep me out of this
discussion.

Also, may I suggest that you monitor
https://gitlab.com/federicomenaquintero/bzip2/issues/6 and especially
read my replies to that issue.

Julian, do not contact me again regarding bzip2 and Rust.  A compatible
C version is going to be available by default.

  Federico

Reply | Threaded
Open this post in threaded view
|

Re: Removing bzip2 support from apt due to rustification

Julian Andres Klode-4
On Fri, Jun 07, 2019 at 03:36:08PM -0500, Federico Mena Quintero wrote:
> On Fri, 2019-06-07 at 11:48 +0800, Julian Andres Klode wrote:
>
> Why am I getting BCCed on this?  Is this low-key unsolicited
> complaining like in https://mastodon.social/@juliank/102226793499538013

I did not BCC you, someone must have bounced it to you. You might
want to look at your mail headers to see how it got to you. I don't
even know your email address (well now I do...). I hope whoever did
it will step forward.

The mastodon post was an odd thing of sorts, as it reads like it
was addressed to you, but was more a general post, missing the
introductory "With". The highlighting there was supposed to allow
people to see more context, not to direct the message at you.

Sorry about that.

--
debian developer - deb.li/jak | jak-linux.org - free software dev
ubuntu core developer                              i speak de, en

Reply | Threaded
Open this post in threaded view
|

Re: Removing bzip2 support from apt due to rustification

Federico Mena Quintero-2
On Fri, 2019-06-07 at 22:48 +0200, Julian Andres Klode wrote:
> I did not BCC you, someone must have bounced it to you. You might
> want to look at your mail headers to see how it got to you. I don't
> even know your email address (well now I do...). I hope whoever did
> it will step forward.

You are correct.  I apologize for assuming it was you who BCCed me and
for making that accusation.

  Federico

Reply | Threaded
Open this post in threaded view
|

Re: Removing bzip2 support from apt due to rustification

Paul Wise via nm
In reply to this post by Julian Andres Klode-4
On Sat, Jun 8, 2019 at 4:48 AM Julian Andres Klode wrote:

> On Fri, Jun 07, 2019 at 03:36:08PM -0500, Federico Mena Quintero wrote:
> > On Fri, 2019-06-07 at 11:48 +0800, Julian Andres Klode wrote:
> >
> > Why am I getting BCCed on this?  Is this low-key unsolicited
> > complaining like in https://mastodon.social/@juliank/102226793499538013
>
> I did not BCC you, someone must have bounced it to you. You might
> want to look at your mail headers to see how it got to you. I don't
> even know your email address (well now I do...). I hope whoever did
> it will step forward.

That was me. I apologise for the confusion I generated. I get annoyed
at people who don't CC the right people when sending mail. In this
case I probably should have sent you Federico direct mail pointing at
the archives instead.

--
bye,
pabs

https://wiki.debian.org/PaulWise