Removing bzip2 support from apt due to rustification

classic Classic list List threaded Threaded
17 messages Options
Reply | Threaded
Open this post in threaded view
|

Removing bzip2 support from apt due to rustification

Julian Andres Klode-4
Hi folks,

seeing that Federico Mena Quintero has taken over bzip2 development
and is in the process of porting it to Rust[1], we should consider
removing bzip2 support from apt, dpkg, etc. following the release
of buster.

My understanding is that having APT depend on a library written in
Rust severily hurts its portability, and makes it hard to support
for stable releases, as Rust is a fairly fast moving target.

I do not believe that bzip2 is a useful algorithm in todays world,
and we should look at migrating any remaining bzip2-only things
(translation files I think) to xz or zstd.

Thanks,
Julian

[1] https://people.gnome.org/~federico/blog/maintaining-bzip2.html
--
debian developer - deb.li/jak | jak-linux.org - free software dev
ubuntu core developer                              i speak de, en

Reply | Threaded
Open this post in threaded view
|

Re: Removing bzip2 support from apt due to rustification

Boyuan Yang-2
在 2019-06-06四的 23:35 +0200,Julian Andres Klode写道:

> Hi folks,
>
> seeing that Federico Mena Quintero has taken over bzip2 development
> and is in the process of porting it to Rust[1], we should consider
> removing bzip2 support from apt, dpkg, etc. following the release
> of buster.
>
> My understanding is that having APT depend on a library written in
> Rust severily hurts its portability, and makes it hard to support
> for stable releases, as Rust is a fairly fast moving target.
>
> I do not believe that bzip2 is a useful algorithm in todays world,
> and we should look at migrating any remaining bzip2-only things
> (translation files I think) to xz or zstd.
I do remember there's still some source packages / binary packages in Debian
using the bzip2 format. If we are going to do that (which looks reasonable,
BTW), a serious archive-wide scan should be made in advance to see how great
the impact is and we need to deal with each occurrence.

Another issue is that the new toolchain (apt/dpkg/...) will not be able to
handle old packages using bzip2. Why not make the bzip2 support optional (like
a plugin or something similar)?

Regards,
Boyuan Yang

signature.asc (875 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Removing bzip2 support from apt due to rustification

Adam Borowski-3
On Thu, Jun 06, 2019 at 05:44:27PM -0400, Boyuan Yang wrote:
> 在 2019-06-06四的 23:35 +0200,Julian Andres Klode写道:
> > seeing that Federico Mena Quintero has taken over bzip2 development
> > and is in the process of porting it to Rust[1], we should consider
> > removing bzip2 support from apt, dpkg, etc. following the release
> > of buster.

How long are versions written in a sane language going to be security
supported?  It's not like bzip2 is a fast moving target.  Vulnerability to
malicious data is the only real concern -- a small code base can otherwise
be kept afloat for decades by just fixing FTBFSes on new compilers/archs.

> > My understanding is that having APT depend on a library written in
> > Rust severily hurts its portability, and makes it hard to support
> > for stable releases, as Rust is a fairly fast moving target.
> >
> > I do not believe that bzip2 is a useful algorithm in todays world,
> > and we should look at migrating any remaining bzip2-only things
> > (translation files I think) to xz or zstd.

Aye, per the recent thread.

> I do remember there's still some source packages / binary packages in Debian
> using the bzip2 format. If we are going to do that (which looks reasonable,
> BTW), a serious archive-wide scan should be made in advance to see how great
> the impact is and we need to deal with each occurrence.

I did such a scan just weeks before.  There's not a single _binary_ package
that uses bz2 for either the control or data tarball in stretch nor buster
(in amd64 but I doubt other archs would be different).  I did not test
jessie -- but dropping support in bullseye would mean the user needs to mix
packages 3 releases apart.  Disallowing that sounds fine to me.

On the other hand, there's a massive number of _source_ packages with bz2
components.  There's 3621 referenced bz2 files in sid.  We're not getting
rid of them anytime soon.

> Another issue is that the new toolchain (apt/dpkg/...) will not be able to
> handle old packages using bzip2. Why not make the bzip2 support optional (like
> a plugin or something similar)?

pipe|exec("bzip2") or dlopen('libbzip2') may work.

Please add support for zstd and improve either 0.939 debs or a new format
while doing so. :)


Meow!
--
⢀⣴⠾⠻⢶⣦⠀
⣾⠁⢠⠒⠀⣿⡁ Sometimes you benefit from delegating stuff.  For example,
⢿⡄⠘⠷⠚⠋⠀ this way I get to be a vegetarian.
⠈⠳⣄⠀⠀⠀⠀

Reply | Threaded
Open this post in threaded view
|

Potentially forking bzip2 (was: Re: Removing bzip2 support from apt due to rustification)

Guillem Jover
In reply to this post by Julian Andres Klode-4
Hi!

On Thu, 2019-06-06 at 23:35:28 +0200, Julian Andres Klode wrote:
> seeing that Federico Mena Quintero has taken over bzip2 development
> and is in the process of porting it to Rust[1], we should consider
> removing bzip2 support from apt, dpkg, etc. following the release
> of buster.

Argh. :( I've stated this many times in the past, but just to make it
clear, I'm not planning on removing support for extraction using
previously supported compression algorithms. bzip2 is already
considered obsolete (and disallowed) at «dpkg-deb --build» time, but
dpkg-deb will happily extract old packages using it.

> My understanding is that having APT depend on a library written in
> Rust severily hurts its portability, and makes it hard to support
> for stable releases, as Rust is a fairly fast moving target.

Yes, this would pretty much make bzip2 a (currently) non-portable
project.

So I really, really, do not want to do it, and I'd rather spend my
time elsewhere, but if it comes to it, and no one else does it before,
I guess I'll have to consider forking prior to its rustification. :(

> [1] https://people.gnome.org/~federico/blog/maintaining-bzip2.html

Thanks,
Guillem

Reply | Threaded
Open this post in threaded view
|

Re: Removing bzip2 support from apt due to rustification

Julian Andres Klode-4
In reply to this post by Boyuan Yang-2
On Thu, Jun 06, 2019 at 05:44:27PM -0400, Boyuan Yang wrote:

> 在 2019-06-06四的 23:35 +0200,Julian Andres Klode写道:
> > Hi folks,
> >
> > seeing that Federico Mena Quintero has taken over bzip2 development
> > and is in the process of porting it to Rust[1], we should consider
> > removing bzip2 support from apt, dpkg, etc. following the release
> > of buster.
> >
> > My understanding is that having APT depend on a library written in
> > Rust severily hurts its portability, and makes it hard to support
> > for stable releases, as Rust is a fairly fast moving target.
> >
> > I do not believe that bzip2 is a useful algorithm in todays world,
> > and we should look at migrating any remaining bzip2-only things
> > (translation files I think) to xz or zstd.
>
> I do remember there's still some source packages / binary packages in Debian
> using the bzip2 format. If we are going to do that (which looks reasonable,
> BTW), a serious archive-wide scan should be made in advance to see how great
> the impact is and we need to deal with each occurrence.
>
> Another issue is that the new toolchain (apt/dpkg/...) will not be able to
> handle old packages using bzip2. Why not make the bzip2 support optional (like
> a plugin or something similar)?

Both apt and dpkg are capable of falling back to executing the bzip2 binary
where available. Well, in theory anyway. I'm not sure it works reliably.

For APT, the impact of removing bzip2 support is a bit more limited
than for dpkg; it "only" breaks apt-extracttemplates, apt-ftparchive,
support for Translation-*.bz2 indexes (which are the only ones we ship
right now (Bug#930103), and python-apt clients using apt_inst to inspect
bzip2 debs.

The risk there seems limited enough, and one thing you can expect from
a 2.0 version bump.

--
debian developer - deb.li/jak | jak-linux.org - free software dev
ubuntu core developer                              i speak de, en

Reply | Threaded
Open this post in threaded view
|

Re: Removing bzip2 support from apt due to rustification

Michael Biebl-3
In reply to this post by Julian Andres Klode-4
Am 06.06.19 um 23:35 schrieb Julian Andres Klode:
> Hi folks,
>
> seeing that Federico Mena Quintero has taken over bzip2 development
> and is in the process of porting it to Rust[1], we should consider
> removing bzip2 support from apt, dpkg, etc. following the release
> of buster.

Reading https://gitlab.com/federicomenaquintero/bzip2/issues/6 I don't
think there is a need to panic.


--
Why is it that all of the instruments seeking intelligent life in the
universe are pointed away from Earth?

Reply | Threaded
Open this post in threaded view
|

Re: Removing bzip2 support from apt due to rustification

Mo Zhou
In reply to this post by Boyuan Yang-2
Hi Boyuan,

On 2019-06-06 21:44, Boyuan Yang wrote:
> I do remember there's still some source packages / binary packages in Debian
> using the bzip2 format. If we are going to do that (which looks reasonable,
> BTW), a serious archive-wide scan should be made in advance to see how great
> the impact is and we need to deal with each occurrence.

Correction: s/some packages/massive amount of packages/

I quickly scanned my local Sid (amd64+source) mirror:

/l/M/debian ❯❯❯ find . -type f -name '*.orig.tar.bz2' | wc -l
1338
/l/M/debian ❯❯❯ find . -type f -name '*.debian.tar.bz2' | wc -l
57

Reply | Threaded
Open this post in threaded view
|

Re: Removing bzip2 support from apt due to rustification

Xavier Guimard-3
Le 07/06/2019 à 09:48, Mo Zhou a écrit :

> Hi Boyuan,
>
> On 2019-06-06 21:44, Boyuan Yang wrote:
>> I do remember there's still some source packages / binary packages in Debian
>> using the bzip2 format. If we are going to do that (which looks reasonable,
>> BTW), a serious archive-wide scan should be made in advance to see how great
>> the impact is and we need to deal with each occurrence.
>
> Correction: s/some packages/massive amount of packages/
>
> I quickly scanned my local Sid (amd64+source) mirror:
>
> /l/M/debian ❯❯❯ find . -type f -name '*.orig.tar.bz2' | wc -l
> 1338
> /l/M/debian ❯❯❯ find . -type f -name '*.debian.tar.bz2' | wc -l
> 57

Hello,

uscan can be modified to repackage all tar.bz2 upstream archives to
tar.xz ones (as for .zip files). If we decide to remove bz2, please open
a BTS against devscripts and I'll do the work.

Cheers,
Xavier

Reply | Threaded
Open this post in threaded view
|

Re: Removing bzip2 support from apt due to rustification

Federico Mena Quintero-2
In reply to this post by Julian Andres Klode-4
On Fri, 2019-06-07 at 11:48 +0800, Julian Andres Klode wrote:

Why am I getting BCCed on this?  Is this low-key unsolicited
complaining like in https://mastodon.social/@juliank/102226793499538013
 ?

> seeing that Federico Mena Quintero has taken over bzip2 development
> and is in the process of porting it to Rust[1], we should consider
> removing bzip2 support from apt, dpkg, etc. following the release
> of buster.

What Debian decides to do is up to them.  Please keep me out of this
discussion.

Also, may I suggest that you monitor
https://gitlab.com/federicomenaquintero/bzip2/issues/6 and especially
read my replies to that issue.

Julian, do not contact me again regarding bzip2 and Rust.  A compatible
C version is going to be available by default.

  Federico

Reply | Threaded
Open this post in threaded view
|

Re: Removing bzip2 support from apt due to rustification

Julian Andres Klode-4
On Fri, Jun 07, 2019 at 03:36:08PM -0500, Federico Mena Quintero wrote:
> On Fri, 2019-06-07 at 11:48 +0800, Julian Andres Klode wrote:
>
> Why am I getting BCCed on this?  Is this low-key unsolicited
> complaining like in https://mastodon.social/@juliank/102226793499538013

I did not BCC you, someone must have bounced it to you. You might
want to look at your mail headers to see how it got to you. I don't
even know your email address (well now I do...). I hope whoever did
it will step forward.

The mastodon post was an odd thing of sorts, as it reads like it
was addressed to you, but was more a general post, missing the
introductory "With". The highlighting there was supposed to allow
people to see more context, not to direct the message at you.

Sorry about that.

--
debian developer - deb.li/jak | jak-linux.org - free software dev
ubuntu core developer                              i speak de, en

Reply | Threaded
Open this post in threaded view
|

Re: Removing bzip2 support from apt due to rustification

Michael Banck
In reply to this post by Xavier Guimard-3
Hi,

On Fri, Jun 07, 2019 at 09:54:01AM +0200, Xavier wrote:

> Le 07/06/2019 à 09:48, Mo Zhou a écrit :
> > On 2019-06-06 21:44, Boyuan Yang wrote:
> >> I do remember there's still some source packages / binary packages in Debian
> >> using the bzip2 format. If we are going to do that (which looks reasonable,
> >> BTW), a serious archive-wide scan should be made in advance to see how great
> >> the impact is and we need to deal with each occurrence.
> >
> > Correction: s/some packages/massive amount of packages/
> >
> > I quickly scanned my local Sid (amd64+source) mirror:
> >
> > /l/M/debian ❯❯❯ find . -type f -name '*.orig.tar.bz2' | wc -l
> > 1338
> > /l/M/debian ❯❯❯ find . -type f -name '*.debian.tar.bz2' | wc -l
> > 57
>
> Hello,
>
> uscan can be modified to repackage all tar.bz2 upstream archives to
> tar.xz ones (as for .zip files). If we decide to remove bz2, please open
> a BTS against devscripts and I'll do the work.

If upstream ships a .bz2 I wouldn't want uscan to unilaterally repackage
it to something else, so -1 from to do that automatically.


Michael

Reply | Threaded
Open this post in threaded view
|

Re: Removing bzip2 support from apt due to rustification

Federico Mena Quintero-2
In reply to this post by Julian Andres Klode-4
On Fri, 2019-06-07 at 22:48 +0200, Julian Andres Klode wrote:
> I did not BCC you, someone must have bounced it to you. You might
> want to look at your mail headers to see how it got to you. I don't
> even know your email address (well now I do...). I hope whoever did
> it will step forward.

You are correct.  I apologize for assuming it was you who BCCed me and
for making that accusation.

  Federico

Reply | Threaded
Open this post in threaded view
|

Re: Removing bzip2 support from apt due to rustification

Paul Wise via nm
In reply to this post by Julian Andres Klode-4
On Sat, Jun 8, 2019 at 4:48 AM Julian Andres Klode wrote:

> On Fri, Jun 07, 2019 at 03:36:08PM -0500, Federico Mena Quintero wrote:
> > On Fri, 2019-06-07 at 11:48 +0800, Julian Andres Klode wrote:
> >
> > Why am I getting BCCed on this?  Is this low-key unsolicited
> > complaining like in https://mastodon.social/@juliank/102226793499538013
>
> I did not BCC you, someone must have bounced it to you. You might
> want to look at your mail headers to see how it got to you. I don't
> even know your email address (well now I do...). I hope whoever did
> it will step forward.

That was me. I apologise for the confusion I generated. I get annoyed
at people who don't CC the right people when sending mail. In this
case I probably should have sent you Federico direct mail pointing at
the archives instead.

--
bye,
pabs

https://wiki.debian.org/PaulWise

Reply | Threaded
Open this post in threaded view
|

Re: Removing bzip2 support from apt due to rustification

Russ Allbery-2
Paul Wise <[hidden email]> writes:

> That was me. I apologise for the confusion I generated. I get annoyed at
> people who don't CC the right people when sending mail. In this case I
> probably should have sent you Federico direct mail pointing at the
> archives instead.

This is called snitch-tagging in other forums like Twitter, and there's a
reason why it's considered extremely rude.

Please let us have internal conversations without using them to
prematurely pick fights with external maintainers.  If those maintainers
really want to see every Debian discussion about their work, they can
search or read the archives; they're all open.  But, quite wisely, they
generally don't.  That's because any group says all sorts of things in the
heat of the moment that the whole project may or may not agree with, or
that may change over time as we discuss them in more depth, or that, even
if we would stand by the sentiment, aren't phrased in a constructive or
productive way because it was an internal discussion (which can include
some blowing off of steam).

Dragging external developers into every Debian discussion concerning their
work doesn't lead to higher-quality and more inclusive discussion.  It
leads to people not talking on public lists at all and taking the same
conversation private, which is a loss for everyone.

--
Russ Allbery ([hidden email])               <http://www.eyrie.org/~eagle/>

Reply | Threaded
Open this post in threaded view
|

Re: Removing bzip2 support from apt due to rustification

Paul Wise via nm
On Sat, Jun 8, 2019 at 10:09 AM Russ Allbery wrote:

> Please let us have internal conversations without using them to
> prematurely pick fights with external maintainers.

It definitely wasn't my intention to pick a fight.

I strongly disagree with hiding upstream problems from upstream
developers. I wonder why our first reaction should be to have internal
discussions instead of upstream discussions. To me that also seems
rude, especially as we promised to communicate with upstream as part
of the Social Contract.

Thanks for your advice on this.

--
bye,
pabs

https://wiki.debian.org/PaulWise

Reply | Threaded
Open this post in threaded view
|

Re: Removing bzip2 support from apt due to rustification

Russ Allbery-2
Paul Wise <[hidden email]> writes:
> On Sat, Jun 8, 2019 at 10:09 AM Russ Allbery wrote:

>> Please let us have internal conversations without using them to
>> prematurely pick fights with external maintainers.

> It definitely wasn't my intention to pick a fight.

Yeah, I didn't think that part was intentional since it seemed contrary to
everything I know about you, but that is the effect of pulling them into a
discussion that's negative about their work.

> I strongly disagree with hiding upstream problems from upstream
> developers. I wonder why our first reaction should be to have internal
> discussions instead of upstream discussions. To me that also seems rude,
> especially as we promised to communicate with upstream as part of the
> Social Contract.

It's a human psychology thing, I think, and it's specifically about the
critical or negative threads.

While it's a common maxim that you should only say positive things about
people and you should never say something that you wouldn't say to
someone's face, it's pretty unrealistic with real people in real
situations that we're going to achieve this 100% of the time.  Asking
everything anyone says in debian-devel be phrased in such a way that it
would be a productive thing to relay directly to an upstream project is
asking a lot.  It might be a lovely utopia if we all lived up to that
ideal, but, well, we won't.

I think it's helpful to give people a bit of space to acknowledge that the
first thing we say out of frustration or surprise or dismay may not be the
thoughtful and constructive thing we *want* to say.  There's a lot of
initial noise in reactions.

This isn't just for us; it's also for the upstreams involved.  It can be
demoralizing and frustrating to be pulled into negative discussions about
something you care about, and people have limited capacity to react
gracefully to such things.  Part of the goal is to be respectful of that
energy.  They will pick and choose what they read and what they choose to
react to in order to manage their energy, and in turn we, when reaching
out with constructive criticisms, will (or at least should) put more
effort into phrasing than we might when complaining about something to our
fellow project members.

I completely agree with you that if we have some piece of concrete
feedback for upstream, we should send that to upstream, particularly
before we act our beliefs or understandings.  It all may be a
misunderstanding, or we may be able to convince them to do something
differently, and this should be a collaboration and partnership.  I'm just
saying let's not do that as a raw feed of discussions that weren't written
with them as the intended audience.  The effort we put into reframing our
consensus or discussion into something more readily consumable by upstream
than a complaint thread is important and increases the chances that
discussion will go well.

We're not hiding anything -- the archives are public and if they really
want to read it all, they can.  We're creating social space for
politeness, which is, in part, maintaining the mutual grace of not
immediately telling other people everything negative that we think about
their work, only the things that are important and that survive some
further reflection.

--
Russ Allbery ([hidden email])               <http://www.eyrie.org/~eagle/>

Reply | Threaded
Open this post in threaded view
|

Re: Removing bzip2 support from apt due to rustification

Colin Watson
On Fri, Jun 07, 2019 at 09:03:18PM -0700, Russ Allbery wrote:
> This isn't just for us; it's also for the upstreams involved.  It can be
> demoralizing and frustrating to be pulled into negative discussions about
> something you care about, and people have limited capacity to react
> gracefully to such things.  Part of the goal is to be respectful of that
> energy.

Thank you for saying this.  I'd go a little further and add that this
may very well be true even if the discussion isn't negative!  There are
very many distributions out there, and a popular upstream project can
easily end up being the subject of discussions in quite a number of them
at once.  If copied on all of them, this can become quite a time-sink.

These days it seems that we often find ourselves debating what the point
of distributions is anyway.  There are many answers to that, but for the
purposes of this thread, when done right, one of the positive things
distributions add is that we can serve as a kind of fan-out layer, often
sorting out bug reports or integration problems without needing to
consult the upstream maintainer directly.  That means that we can avoid
putting extra demands on the upstream maintainer's perhaps scarce time
by adding more stuff to their inbox.  That's true even in a case like
this where it's a concern about upstream's future plans: before Federico
replied to this thread, Michael had already cited the upstream bug in
which the maintainer had said they were going to maintain a parallel C
version.

Not everyone is a hyper-efficient email-handling machine, and even if
they are maybe it isn't the best possible use of their time; and the
antonym of "hide problems" doesn't have to be "make sure that everyone
who might be relevant is always copied on emails".  Let's be thoughtful
about whether we need to take up upstream maintainers' time.

--
Colin Watson                                       [[hidden email]]