RFC: Support for zstd in .deb packages?

classic Classic list List threaded Threaded
19 messages Options
Reply | Threaded
Open this post in threaded view
|

RFC: Support for zstd in .deb packages?

Guillem Jover
Hi!

In 2016 Paul Wise mentioned the Zstandard compressor on IRC [Z],
and I briefly checked it out as a potential candidate for dpkg
(while also mentioning it to Julian Andres Klode who was considering
adding lz4 support to apt). At the time it looked like it was not
worth it (apt went with lz4), so it got parked.

Recently Julian mentioned it again on IRC, and we each started
implementing support in dpkg and apt respectively, to allow easier
evaluation. I stopped when I realized the code was getting too messy,
but after few weeks Julian and Bálint Réczey ended up implementing
the support for apt and dpkg [D], so that they could merge it in
Ubuntu for their upcoming release, which they did.

Their main driving reason (AFAICT) has been (de)compression speed.

The following is a quick run-down of the items from [F], not all
being important from Debian's perspective, but being for dpkg's:

* License: Permissive (dual BSD + GPL-2), which makes universal
  availability possible.
* Portability: The code seems portable, and it's available on some
  non-Linux systems.
* Availability: I don't think it's as common as gzip/bzip2/xz are now,
  say on non-Linux systems, perhaps not even on Linux distros.
* Implementation size: The shared library and its package are quite
  fatter than any of the other compressors dpkg uses.
* Eternity contract: This would add yet another format that would need
  to be supported pretty much forever, to be able to at least unpack
  .deb's that might be available in the wild. This also increases the
  (Build-)Essential-set.
* Format stability: Although it's supposedly frozen now, it has
  changed quite often in recent times. AFAIR it was also mentioned at
  least in the past that the target was mainly real-time data streaming,
  so long-term data storage might not be a priority? Would need
  clarification from upstream I guess.
* Memory usage: Seemed equivalent or less to current compressors, but
  only as long as equal or less space was desired.
* Space usage: Seemed worse.
* (De)compression speed: Seemed better (compared only to the existing
  supported formats) depending on the compression level used.


I'm still quite skeptical about it being worth it though, given the costs
implied (f.ex. [S]). That it trades space for speed, which could perhaps
improve use-cases like CI or buildds, or rolling distribution users, but
that still varies depending on the network speed, fsys/disk speed, etc,
which might not be an universal improver (to get there we might need to
tie this to delta debs, which would benefit xz too). It makes CD/DVD/BD
images and the archive in general larger. It's not the fastest, and it
doesn't have the highest compression ratio either; if we'd want way
faster (de)compression I think we'd still be better off with something
like lz4 anyway? And that most of the assumed benefits would only be
gained if we switched to it as the new default compressor, which would
require project-wide consensus. As a replacement for gzip, it would
definitely make sense, but otherwise I'm not sure I see it.

An area where there's still room for improvement with xz f.ex. when it
comes to decompression speed, is lack of multi-threaded support, as
liblzma currently only supports it for compression.


In any case, I've CCed both Julian and Bálint, so that they can fill
in with more details and numbers from what they have been trying out.

But in any case, I'm still open to data and opinions given that this
is in the end a matter of trade-offs, so → request for comments. :)

(And BTW I do not consider the current support in Ubuntu a deciding
factor in any way, while it could perhaps fragment the .deb ecosystem,
that's something for them to deal with IMO; should really start adding
the vendor to the generated .deb's. :)

Thanks,
Guillem

[Z] <https://code.facebook.com/posts/1658392934479273/smaller-and-faster-data-compression-with-zstandard/>
    <https://facebook.github.io/zstd/>
[F] <https://wiki.debian.org/Teams/Dpkg/FAQ#Q:_Can_we_add_support_for_new_compressors_for_.deb_packages.3F>
[D] <https://bugs.debian.org/892664>
[S] <https://wiki.debian.org/Teams/Dpkg/DebSupport>

Reply | Threaded
Open this post in threaded view
|

Re: RFC: Support for zstd in .deb packages?

Martin Steigerwald
Hi Guillem.

I have no real opinion on this.

Guillem Jover - 27.04.18, 07:02:
[…]
> In 2016 Paul Wise mentioned the Zstandard compressor on IRC [Z],
> and I briefly checked it out as a potential candidate for dpkg
> (while also mentioning it to Julian Andres Klode who was considering
> adding lz4 support to apt). At the time it looked like it was not
> worth it (apt went with lz4), so it got parked.
[…]
> The following is a quick run-down of the items from [F], not all
> being important from Debian's perspective, but being for dpkg's:
[…]

> * Format stability: Although it's supposedly frozen now, it has
>   changed quite often in recent times. AFAIR it was also mentioned at
>   least in the past that the target was mainly real-time data
> streaming, so long-term data storage might not be a priority? Would
> need clarification from upstream I guess.
> * Memory usage: Seemed equivalent or less to current compressors, but
>   only as long as equal or less space was desired.
> * Space usage: Seemed worse.
> * (De)compression speed: Seemed better (compared only to the existing
>   supported formats) depending on the compression level used.

Regarding technical aspects like these, one more data point: BTRFS
meanwhile offers zstandard compression support. So I bet BTRFS
developers consider it suitable for format stability and long-term data
storage. I am still using lzo on my BTRFS filesystems, so I can not tell
any practical experiences so far.
 
> (And BTW I do not consider the current support in Ubuntu a deciding
> factor in any way, while it could perhaps fragment the .deb ecosystem,
> that's something for them to deal with IMO; should really start
> adding the vendor to the generated .deb's. :)

If zstd compressed deb´s appear in the wild, it may make sense to at
least implement decompression support.

Thanks,
--
Martin


Reply | Threaded
Open this post in threaded view
|

Re: RFC: Support for zstd in .deb packages?

Raphael Hertzog-3
In reply to this post by Guillem Jover
Hello,

On Fri, 27 Apr 2018, Guillem Jover wrote:
> But in any case, I'm still open to data and opinions given that this
> is in the end a matter of trade-offs, so → request for comments. :)

I believe that the "zstd" compression format is providing an interesting
trade-off compared to other compression formats that might be relevant in
multiple use cases. As such, I think it makes sense to support it in dpkg.

It it worth noting that rpm apparently decided to support this format too:
https://lists.ubuntu.com/archives/ubuntu-devel/2018-March/040231.html

Deciding on whether it must be the default is a separate discussion
though.

BTW, there was a similar discussion on the ubuntu-devel list and it
might be interesting for others to look at it:
https://lists.ubuntu.com/archives/ubuntu-devel/2018-March/thread.html#40211

The main usecase promoted in Ubuntu seems to be speeding up initial
installation to have faster availability of a new cloud server. And
possibly also minimizing the time to install updates that have
been downloaded in advance.

I also consider that optimizing the time spent in continuous integration
jobs is a worthy effort. You have the .deb available nearby but you end
up reinstalling them every time in a container and that part is often
as long as the real job in itself. And you really want the results of
those jobs to be made available as soon as possible.

> (And BTW I do not consider the current support in Ubuntu a deciding
> factor in any way, while it could perhaps fragment the .deb ecosystem,
> that's something for them to deal with IMO; should really start adding
> the vendor to the generated .deb's. :)

The above ubuntu-devel discussion clearly indicated that they want to
see it accepted in Debian first to avoid the divergence.

Cheers,
--
Raphaël Hertzog ◈ Debian Developer

Support Debian LTS: https://www.freexian.com/services/debian-lts.html
Learn to master Debian: https://debian-handbook.info/get/

Reply | Threaded
Open this post in threaded view
|

Re: RFC: Support for zstd in .deb packages?

Colin Watson
In reply to this post by Guillem Jover
On Fri, Apr 27, 2018 at 07:02:12AM +0200, Guillem Jover wrote:
> (And BTW I do not consider the current support in Ubuntu a deciding
> factor in any way, while it could perhaps fragment the .deb ecosystem,
> that's something for them to deal with IMO; should really start adding
> the vendor to the generated .deb's. :)

FWIW I'm going to be very reluctant to add support to Launchpad until
and unless it's supported in Debian, for exactly the reasons of
fragmentation, compatibility, etc.  I said as much in
https://lists.ubuntu.com/archives/ubuntu-devel/2018-March/040222.html
too.

(Without support in Launchpad, the format isn't going to be widespread
in Ubuntu.)

--
Colin Watson                                       [[hidden email]]

Reply | Threaded
Open this post in threaded view
|

Re: RFC: Support for zstd in .deb packages?

Adam Borowski-3
In reply to this post by Guillem Jover
(ZSTD)

On Fri, Apr 27, 2018 at 07:02:12AM +0200, Guillem Jover wrote:
> Recently Julian mentioned it again on IRC, and we each started
> implementing support in dpkg and apt respectively, to allow easier
> evaluation. I stopped when I realized the code was getting too messy,
> but after few weeks Julian and Bálint Réczey ended up implementing
> the support for apt and dpkg [D], so that they could merge it in
> Ubuntu for their upcoming release, which they did.
>
> Their main driving reason (AFAICT) has been (de)compression speed.

With default level:

        comp decomp size
xz 8.038s 0.356s 4320292
bz2 2.265s 0.730s 5234516
zst 0.274s 0.102s 5657626
gz 0.880s 0.152s 6515505
Z 0.499s 0.133s 8932459
lzo 0.100s 0.095s 9198874

_Compression_ speed is insane; decompression is merely nice.  Obviously,
tuneable level turns this single point into an envelope, but even xz at its
lowest levels can be multiple times faster than gzip while delivering better
compression.

> * Implementation size: The shared library and its package are quite
>   fatter than any of the other compressors dpkg uses.
> * Eternity contract: This would add yet another format that would need
>   to be supported pretty much forever, to be able to at least unpack
>   .deb's that might be available in the wild. This also increases the
>   (Build-)Essential-set.

> I'm still quite skeptical about it being worth it though, given the costs
> implied (f.ex. [S]). That it trades space for speed, which could perhaps
> improve use-cases like CI or buildds

As the guy who did most of testing of Nick Terrell's work in btrfs,
implemented support in tar (accepted upstream, #894065[1]), mc (ditto),
wages a propaganda campaign for zstd for vmlinuz+initrd, and is an obnoxious
fanboy of zstd in a bunch of other places, I'd say:

Don't.  For .debs, that is.

A .deb that is getting processed goes through dpkg, whose status files
writes and all those fsyncs are so slow that it's pointless to optimize
every last bit of decompression.  If someone already goes as far as
recompressing packages, we got two fine choices: "xz -0" and "none"; the
former being faster than gzip while compressing better, and the latter being
as fast as cat.  You don't get any further than cat on the size-vs-speed
slider...

On the other hand, adding .tar.zst [2] would be a much welcome addition for
files referenced from .dsc.  A machine that installs dpkg-dev has enough
storage that a half-MB library is beneath any notice, and zstd is a fine
replacement for gzip and the likes.

You'd also want to look into getting rid of bzip2 and gzip.  The former is a
low-flying fruit: binNMUing remaining packages would allow retiring bzip2
support in buster+3 or so[3], gzip is probably forever.


Meow!

[1]. Bdale, you know you want to take it, pretty please with a cherry on
top. :)

[2]. Somehow, someone stolen the 'd'.  I'd look at "umount", the culprit
might be the same.

[3]. Or possibly fork+exec("bzip2") for historic .debs, failing with an
error message if not installed.
--
⢀⣴⠾⠻⢶⣦⠀
⣾⠁⢰⠒⠀⣿⡁
⢿⡄⠘⠷⠚⠋⠀ Certified airhead; got the CT scan to prove that!
⠈⠳⣄⠀⠀⠀⠀

Reply | Threaded
Open this post in threaded view
|

Re: RFC: Support for zstd in .deb packages?

Adam Borowski-3
On Fri, Apr 27, 2018 at 01:45:07PM +0200, Adam Borowski wrote:
> Don't.  For .debs, that is.

Scratch that.

apt Depends: libapt-pkg5.0 Depends: libzstd1

While apt is "merely" priority:required rather than fully essential, a
Debian system without apt is so deeply embedded it already requires special
steps, thus there's probably no reason to bother.

If apt has already taken the plunge, it's reasonable for dpkg to follow.
The "reduced essential set" guys will be unhappy, but as we're already
there, it's good to switch other users which need a general-purpose fast
good compressor (the "slow but strong" slot is providen by xz, "weak but
extremely fast" by lz4 -- libzstd happens to include a lz4 implementation).


Meow!
--
⢀⣴⠾⠻⢶⣦⠀
⣾⠁⢰⠒⠀⣿⡁
⢿⡄⠘⠷⠚⠋⠀ Certified airhead; got the CT scan to prove that!
⠈⠳⣄⠀⠀⠀⠀

Reply | Threaded
Open this post in threaded view
|

Re: RFC: Support for zstd in .deb packages?

Ian Jackson-2
In reply to this post by Guillem Jover
Guillem Jover writes ("RFC: Support for zstd in .deb packages?"):
> * Eternity contract: This would add yet another format that would need
>   to be supported pretty much forever, to be able to at least unpack
>   .deb's that might be available in the wild. This also increases the
>   (Build-)Essential-set.

This means that we should be much slower to adopt new compression
schemes than projects where data compression is used for transport or
short-term storage.

I would say that a new compression scheme should be widely used for
several years before we would consider it for source package and at
least half a decade before we would adopt it for .debs.

I am also concerned that we are the target of an advocacy campaign.

Ian.

Reply | Threaded
Open this post in threaded view
|

Re: RFC: Support for zstd in .deb packages?

Julian Andres Klode-4
In reply to this post by Adam Borowski-3
On Fri, Apr 27, 2018 at 02:01:44PM +0200, Adam Borowski wrote:

> On Fri, Apr 27, 2018 at 01:45:07PM +0200, Adam Borowski wrote:
> > Don't.  For .debs, that is.
>
> Scratch that.
>
> apt Depends: libapt-pkg5.0 Depends: libzstd1
>
> While apt is "merely" priority:required rather than fully essential, a
> Debian system without apt is so deeply embedded it already requires special
> steps, thus there's probably no reason to bother.
>
> If apt has already taken the plunge, it's reasonable for dpkg to follow.
> The "reduced essential set" guys will be unhappy, but as we're already
> there, it's good to switch other users which need a general-purpose fast
> good compressor (the "slow but strong" slot is providen by xz, "weak but
> extremely fast" by lz4 -- libzstd happens to include a lz4 implementation).

I specifically called it experimental, and it might not be part of buster
if it turns out to be not useful.

--
debian developer - deb.li/jak | jak-linux.org - free software dev
ubuntu core developer                              i speak de, en

Reply | Threaded
Open this post in threaded view
|

Re: RFC: Support for zstd in .deb packages?

Julian Andres Klode-4
In reply to this post by Adam Borowski-3
On Fri, Apr 27, 2018 at 01:45:07PM +0200, Adam Borowski wrote:

> (ZSTD)
>
> On Fri, Apr 27, 2018 at 07:02:12AM +0200, Guillem Jover wrote:
> > Recently Julian mentioned it again on IRC, and we each started
> > implementing support in dpkg and apt respectively, to allow easier
> > evaluation. I stopped when I realized the code was getting too messy,
> > but after few weeks Julian and Bálint Réczey ended up implementing
> > the support for apt and dpkg [D], so that they could merge it in
> > Ubuntu for their upcoming release, which they did.
> >
> > Their main driving reason (AFAICT) has been (de)compression speed.
>
> With default level:
>
> comp decomp size
> xz 8.038s 0.356s 4320292
> bz2 2.265s 0.730s 5234516
> zst 0.274s 0.102s 5657626
> gz 0.880s 0.152s 6515505
> Z 0.499s 0.133s 8932459
> lzo 0.100s 0.095s 9198874
>
> _Compression_ speed is insane; decompression is merely nice.  Obviously,
> tuneable level turns this single point into an envelope, but even xz at its
> lowest levels can be multiple times faster than gzip while delivering better
> compression.

We only care about zstd at level 19 for dpkg and apt.

> Don't.  For .debs, that is.
>
> A .deb that is getting processed goes through dpkg, whose status files
> writes and all those fsyncs are so slow that it's pointless to optimize
> every last bit of decompression.  If someone already goes as far as
> recompressing packages, we got two fine choices: "xz -0" and "none"; the
> former being faster than gzip while compressing better, and the latter being
> as fast as cat.  You don't get any further than cat on the size-vs-speed
> slider...

Our major use case is cloud initial setup, image building, CI, buildds, all
of which do not require any syncs, and can safely use eatmydata, for example;
hence the enormous speed up.

There's still a speed up with syncs, but it's less than 10% IIRC. This will
get faster over time as SSDs increase their IOPS.

>
> On the other hand, adding .tar.zst [2] would be a much welcome addition for
> files referenced from .dsc.  A machine that installs dpkg-dev has enough
> storage that a half-MB library is beneath any notice, and zstd is a fine
> replacement for gzip and the likes.

this was backed out at guillem's request.

--
debian developer - deb.li/jak | jak-linux.org - free software dev
ubuntu core developer                              i speak de, en

Reply | Threaded
Open this post in threaded view
|

Re: RFC: Support for zstd in .deb packages?

Adrian Bunk-3
In reply to this post by Guillem Jover
On Fri, Apr 27, 2018 at 07:02:12AM +0200, Guillem Jover wrote:

>...
> * Eternity contract: This would add yet another format that would need
>   to be supported pretty much forever, to be able to at least unpack
>   .deb's that might be available in the wild. This also increases the
>   (Build-)Essential-set.
> * Format stability: Although it's supposedly frozen now, it has
>   changed quite often in recent times. AFAIR it was also mentioned at
>   least in the past that the target was mainly real-time data streaming,
>   so long-term data storage might not be a priority? Would need
>   clarification from upstream I guess.

This does not sound like something that should be done soon.

>...
> As a replacement for gzip, it would
> definitely make sense, but otherwise I'm not sure I see it.

The number of packages that use gzip as compressor if rebuilt should be
pretty close to 0. We need gzip for compatibility with older packages,
but no replacement for it.

> An area where there's still room for improvement with xz f.ex. when it
> comes to decompression speed, is lack of multi-threaded support, as
> liblzma currently only supports it for compression.
>...

This sounds like a much less invasive change to me.
And it could already deliver benefits for buster if
someone would be willing to implement that.

And there's another potential low hanging fruit for buster:

xz decompression time is linear with the compressed size,
higher compression would bring both smaller packages and
faster decompression.

There are two downsides to switching xz compression from -6 to -9:[1]

1. higher compression time
This will increase build times.
If this turns out to be a problem, most of our biggest packages are the
-dbgsym packages that could continue to use a lower compression if needed.

2. more memory usage for uncompression
64 MB memory usage can be a problem for some lower end machines.

One way to mitigate both downsides might be different default compression
levels on different ports, the ports where something like faster setup
of cloud servers would matter are not the ports with slow buildds or
where 64 MB memory usage would matter much on any supported hardware.

> Thanks,
> Guillem

cu
Adrian

[1] we are talking about perhaps 5-10% smaller packages and faster
    decompression speed

--

       "Is there not promise of rain?" Ling Tan asked suddenly out
        of the darkness. There had been need of rain for many days.
       "Only a promise," Lao Er said.
                                       Pearl S. Buck - Dragon Seed

Reply | Threaded
Open this post in threaded view
|

Re: RFC: Support for zstd in .deb packages?

Guillem Jover
On Sat, 2018-04-28 at 19:44:08 +0300, Adrian Bunk wrote:
> On Fri, Apr 27, 2018 at 07:02:12AM +0200, Guillem Jover wrote:
> >...
> > As a replacement for gzip, it would
> > definitely make sense, but otherwise I'm not sure I see it.
>
> The number of packages that use gzip as compressor if rebuilt should be
> pretty close to 0. We need gzip for compatibility with older packages,
> but no replacement for it.

I know, I meant this in the sense that if we wanted and could replace
gzip with something better, going with zstd might be the obvious
option. But given that we are stuck with having to support gzip
anyway…

> > An area where there's still room for improvement with xz f.ex. when it
> > comes to decompression speed, is lack of multi-threaded support, as
> > liblzma currently only supports it for compression.
>
> This sounds like a much less invasive change to me.
> And it could already deliver benefits for buster if
> someone would be willing to implement that.

Something else I just noticed yesterday is speed-up improvements in zlib:

  https://bugs.debian.or/763982
  https://github.com/jtkukunas/zlib
  https://github.com/Dead2/zlib-ng/
  https://github.com/gildor2/fast_zlib
  https://github.com/madler/zlib/issues/216
  https://github.com/madler/zlib/issues/231
  https://github.com/madler/zlib/issues/346
  https://github.com/madler/zlib/pull/251
  https://github.com/madler/zlib/pull/335
  https://github.com/madler/zlib/pull/345

Thanks,
Guillem

Reply | Threaded
Open this post in threaded view
|

Re: RFC: Support for zstd in .deb packages?

Nick Terrell
In reply to this post by Guillem Jover
Hi!

On Fri, 27 Apr 2018, Guillem Jover wrote:
[...]
> * Format stability: Although it's supposedly frozen now, it has
> changed quite often in recent times. AFAIR it was also mentioned at
> least in the past that the target was mainly real-time data streaming,
> so long-term data storage might not be a priority? Would need
> clarification from upstream I guess.

I'm one of the upstream maintainers of zstd. If you have any questions,
please feel free to either open an issue on the issue board [1] or email
me or Yann Collet (CCed).

The format has been stable since version 0.8.1, which was released on
August 18th 2016. We have an RFC draft describing the compression
format in progress [2], so we are completely committed to keeping the
format stable.

Long-term data storage is as much a priority for us as real-time data
streaming. IMO one of the strengths of Zstandard is the ability to maintain
the same fast decompression speed at every compression level. That
allows both real-time streaming and long-term storage to use the same
format.

[1] https://github.com/facebook/zstd/issues
[2] https://tools.ietf.org/html/draft-kucherawy-dispatch-zstd-01

Thanks,
Nick Terrell

Reply | Threaded
Open this post in threaded view
|

Re: RFC: Support for zstd in .deb packages?

Scott Kitterman-5
On Monday, April 30, 2018 08:23:15 PM Nick Terrell wrote:

> Hi!
>
> On Fri, 27 Apr 2018, Guillem Jover wrote:
> [...]
>
> > * Format stability: Although it's supposedly frozen now, it has
> > changed quite often in recent times. AFAIR it was also mentioned at
> > least in the past that the target was mainly real-time data streaming,
> > so long-term data storage might not be a priority? Would need
> > clarification from upstream I guess.
>
> I'm one of the upstream maintainers of zstd. If you have any questions,
> please feel free to either open an issue on the issue board [1] or email
> me or Yann Collet (CCed).
>
> The format has been stable since version 0.8.1, which was released on
> August 18th 2016. We have an RFC draft describing the compression
> format in progress [2], so we are completely committed to keeping the
> format stable.
>
> Long-term data storage is as much a priority for us as real-time data
> streaming. IMO one of the strengths of Zstandard is the ability to maintain
> the same fast decompression speed at every compression level. That
> allows both real-time streaming and long-term storage to use the same
> format.
>
> [1] https://github.com/facebook/zstd/issues
> [2] https://tools.ietf.org/html/draft-kucherawy-dispatch-zstd-01

For those that don't follow the minutia of the IETF, the draft is in IETF last
call (which is very far along in the process), so it should be reasonably safe
to assume stability at this point.

Scott K

Reply | Threaded
Open this post in threaded view
|

Re: RFC: Support for zstd in .deb packages?

Marco d'Itri
In reply to this post by Julian Andres Klode-4
On Apr 27, Julian Andres Klode <[hidden email]> wrote:

> Our major use case is cloud initial setup, image building, CI, buildds, all
> of which do not require any syncs, and can safely use eatmydata, for example;
> hence the enormous speed up.
I do not believe that it would be wise to optimize our packaging system
for the niche target of package development.

In my experience as a cloud infrastructure provider, new systems are
cloned/instantiated from golden images and not from debootstrap or d-i.

--
ciao,
Marco

signature.asc (673 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: RFC: Support for zstd in .deb packages?

Bastian Blank
On Tue, May 01, 2018 at 10:36:34AM +0200, Marco d'Itri wrote:

> On Apr 27, Julian Andres Klode <[hidden email]> wrote:
>
> > Our major use case is cloud initial setup, image building, CI, buildds, all
> > of which do not require any syncs, and can safely use eatmydata, for example;
> > hence the enormous speed up.
> I do not believe that it would be wise to optimize our packaging system
> for the niche target of package development.
>
> In my experience as a cloud infrastructure provider, new systems are
> cloned/instantiated from golden images and not from debootstrap or d-i.

And even creating these golden images does not take long.  In my
experience a large fraction of the time actually goes into uploading
such images so they can be used.

Bastian

--
Worlds are conquered, galaxies destroyed -- but a woman is always a woman.
                -- Kirk, "The Conscience of the King", stardate 2818.9

Reply | Threaded
Open this post in threaded view
|

Re: RFC: Support for zstd in .deb packages?

Guillem Jover
In reply to this post by Nick Terrell
Hi!

On Mon, 2018-04-30 at 20:23:15 +0000, Nick Terrell wrote:

> On Fri, 27 Apr 2018, Guillem Jover wrote:
> [...]
> > * Format stability: Although it's supposedly frozen now, it has
> > changed quite often in recent times. AFAIR it was also mentioned at
> > least in the past that the target was mainly real-time data streaming,
> > so long-term data storage might not be a priority? Would need
> > clarification from upstream I guess.
>
> I'm one of the upstream maintainers of zstd. If you have any questions,
> please feel free to either open an issue on the issue board [1] or email
> me or Yann Collet (CCed).

Perfect, thanks for reaching out!

> The format has been stable since version 0.8.1, which was released on
> August 18th 2016. We have an RFC draft describing the compression
> format in progress [2], so we are completely committed to keeping the
> format stable.
>
> Long-term data storage is as much a priority for us as real-time data
> streaming. IMO one of the strengths of Zstandard is the ability to maintain
> the same fast decompression speed at every compression level. That
> allows both real-time streaming and long-term storage to use the same
> format.

Thanks for the clarification!

Regards,
Guillem

Reply | Threaded
Open this post in threaded view
|

Re: RFC: Support for zstd in .deb packages?

Guillem Jover
In reply to this post by Guillem Jover
Hi!

On Fri, 2018-04-27 at 07:02:12 +0200, Guillem Jover wrote:
> The following is a quick run-down of the items from [F], not all
> being important from Debian's perspective, but being for dpkg's:

> * License: Permissive (dual BSD + GPL-2), which makes universal
>   availability possible.

Unfortunately, I've just noticed that the project requires a CLA,
which means universal contributions are *not* possible. :(

  <https://github.com/facebook/zstd/blob/dev/CONTRIBUTING.md>
  <https://code.facebook.com/cla>

Nick & Yann, I'm assuming this is some corporate requirement from
Facebook, and it's probably not going to go away?

Thanks,
Guillem

Reply | Threaded
Open this post in threaded view
|

Re: RFC: Support for zstd in .deb packages?

Nick Terrell

> On May 4, 2018, at 6:22 AM, Guillem Jover <[hidden email]> wrote:
>
> Hi!
>
> On Fri, 2018-04-27 at 07:02:12 +0200, Guillem Jover wrote:
>> The following is a quick run-down of the items from [F], not all
>> being important from Debian's perspective, but being for dpkg's:
>
>> * License: Permissive (dual BSD + GPL-2), which makes universal
>>  availability possible.
>
> Unfortunately, I've just noticed that the project requires a CLA,
> which means universal contributions are *not* possible. :(
>
>  <https://github.com/facebook/zstd/blob/dev/CONTRIBUTING.md>
>  <https://code.facebook.com/cla>
>
> Nick & Yann, I'm assuming this is some corporate requirement from
> Facebook, and it's probably not going to go away?

Yup, it is here to stay.

-Nick

Reply | Threaded
Open this post in threaded view
|

Re: RFC: Support for zstd in .deb packages?

Julian Andres Klode-4
In reply to this post by Marco d'Itri
On Tue, May 01, 2018 at 10:36:34AM +0200, Marco d'Itri wrote:

> On Apr 27, Julian Andres Klode <[hidden email]> wrote:
>
> > Our major use case is cloud initial setup, image building, CI, buildds, all
> > of which do not require any syncs, and can safely use eatmydata, for example;
> > hence the enormous speed up.
> I do not believe that it would be wise to optimize our packaging system
> for the niche target of package development.
>
> In my experience as a cloud infrastructure provider, new systems are
> cloned/instantiated from golden images and not from debootstrap or d-i.

Well, yes, you usually have _some_ cloud-provided image. But usually, people
will use cloud-init to initialize their instances and install packages from
in there, often using eatmydata.

--
debian developer - deb.li/jak | jak-linux.org - free software dev
ubuntu core developer                              i speak de, en