Debian Buster will only be 54% reproducible (while we could be at >90%)

classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|

Debian Buster will only be 54% reproducible (while we could be at >90%)

Holger Levsen-2
hi,

disclaimer: this has not yet been verified by anyone other than myself,
so I could very well be wrong. Reproducible builds are about enabling
anyone to independently verify that... ;p


== Reproducibility in theory ==

According to https://tests.reproducible-builds.org/debian/buster/index_suite_amd64_stats.html
we have 26476 source packages (92.8%) which can be built reproducibly in
buster/amd64, out of 28523 source packages in total.
(These 28523 source packages build 57448 binary packages.)

But these tests are done without looking at the actual .deb files distributed
from ftp.debian.org (and we always knew that and pointed it out:
"93% reproducible _in our current test framework_".)


== Looking at binary packages Debian actually distributes ==

So, Vagrant came up with an idea [1] to check buildinfo.debian.net for
.deb files for which 2 or more .buildinfo exist (where "exist" means
that the .deb files sha1sum is listed in the .buildinfo file) and I
turned that into a jenkins job doing this check for all 57448 binary
packages in amd64/buster/main (incl downloading all those .deb files from
ftp.d.o).

The current main results (from this job [2]) are:

reproducible packages in buster/amd64: 30885: (53.7600%)
unreproducible packages in buster/amd64: 26543: (46.2000%)

and

reproducible binNMUs in buster/amd64: 0: (0%)
unreproducible binNMU in buster/amd64: 7423: (12.9200%)


== why are binNMUs unreproducible? ==

Because of their design, binNMUs are unreproducible, see #894441 [3] for
the details (in short: binNMUs are not what they are ment to be: the source
is changed and thrown away) and our proposed solution: 'binNMUs should
be replaced by easy "no-change-except-debian/changelog-uploads'.

So that accounts for 12%, but 12% are not enough to explain the
difference between 54% and 93%...


== packages which have not been rebuilt since December 2016 ==

And today I remember a thread I started last year in May, titled
"packages which have not been rebuilt since December 2016" [4] (because
these packages were build with an old dpkg not producing .buildinfo
files, which Chris turned into #900837 [5] "release.debian.org:
Mass-rebuild of packages  for reproducible builds" and so today I ran
Chris' script [6] again on coccia.d.o, and today it showed that 'only'
6804 source packages need a rebuild (compared to 9192 eight months ago).

6804 of of 28523 is 23.9%. And 54%+12%+24% equals 90%. Bingo. Bummer.

(While #900837 was only filed in 2018 we knew about this issue since
2015 or so... probably earlier. Sigh.)


== After the release is before the release. ==

So, as we first need to fix #894441 before we can sensibly fix #900837 and
because Buster is practically frozen, I think we can just conclude that
Buster is quite reproducible in theory (similar but better than
Stretch...)  and that we need to make sure to address #894441 ASAP, which
means for Bullseye, the release after Buster.

Fur future reference, a summary of the current status of Debian's
reproducibiliy is available at
https://wiki.debian.org/ReproducibleBuilds#Big_outstanding_issues [7]

Happy hacking and many many thanks to everyone who has contributed so far!


[1] https://lists.reproducible-builds.org/pipermail/rb-general/2018-October/001239.html
[2] https://jenkins.debian.net/job/reproducible_compare_Debian_sha1sums/103/console
[3] https://bugs.debian.org/894441
[4] https://lists.debian.org/debian-devel/2018/05/msg00499.html
[5] https://bugs.debian.org/900837
[6] https://lists.debian.org/debian-devel/2018/06/msg00007.html
[7] https://wiki.debian.org/ReproducibleBuilds#Big_outstanding_issues


--
tschau,
        Holger

-------------------------------------------------------------------------------
               holger@(debian|reproducible-builds|layer-acht).org
       PGP fingerprint: B8BF 5413 7B09 D35C F026 FE9D 091A B856 069A AA1C

signature.asc (849 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

re: Debian Buster will only be 54% reproducible (while we could be at >90%)

peter green-2
> Because of their design, binNMUs are unreproducible, see #894441 [3] for
> the details (in short: binNMUs are not what they are ment to be: the source
> is changed and thrown away)
To be specific, the source tree is extracted, then an entry is added to debian/changelog and then the package is built. This modified source tree is not retained.

It seems to me that binnmus could be made reproducible by storing the debian/changelog modifications in the buildinfo, then re-applying it at reproduction time.

Reply | Threaded
Open this post in threaded view
|

Re: Debian Buster will only be 54% reproducible (while we could be at >90%)

Matthias Klumpp
Am Mi., 6. März 2019 um 10:40 Uhr schrieb peter green <[hidden email]>:
>
> > Because of their design, binNMUs are unreproducible, see #894441 [3] for
> > the details (in short: binNMUs are not what they are ment to be: the source
> > is changed and thrown away)
> To be specific, the source tree is extracted, then an entry is added to debian/changelog and then the package is built. This modified source tree is not retained.
> [...]

(Experience report incoming)

I have once tried that in the Tanglu derivative, and found out that
this wasn't as easy as I initially thought because a lot of packages
run special tools prior to building their sources, e.g. to edit
d/control or to read d/changelog and inject data in several places.
So, the option there was either to create a chroot dedicated to the
source package rebuild (installing all Build-Deps and Pre-Deps prior
to the actual source rebuild), or to not actually rebuild the source
package but just edit the d/changelog file and recreate the tarball.
For Tanglu we went for the "just edit d/changelog and re-tar, re-sign
& upload" which worked fine and without any noticeable issues - this
was mainly due to the limited build power we had at the time. For
Debian, which has a lot more resources, just full rebuilding the
source with all dependencies is likely much cleaner, but this approach
might be a bit slow for huge transitions.

At Ubuntu, some people seem to do this process manually, that is run
some scripts locally, rebuild sources locally & upload (unless that
has changed recently).
In general, having source d/changelog aligned with the actual binaries
produced is a really great goal!

Cheers,
    Matthias

--
I welcome VSRE emails. See http://vsre.info/

Reply | Threaded
Open this post in threaded view
|

Re: Debian Buster will only be 54% reproducible (while we could be at >90%)

Dimitri John Ledkov-3
In reply to this post by Holger Levsen-2
On Tue, 5 Mar 2019 at 13:34, Holger Levsen <[hidden email]> wrote:

>
> hi,
>
> disclaimer: this has not yet been verified by anyone other than myself,
> so I could very well be wrong. Reproducible builds are about enabling
> anyone to independently verify that... ;p
>
>
> == Reproducibility in theory ==
>
> According to https://tests.reproducible-builds.org/debian/buster/index_suite_amd64_stats.html
> we have 26476 source packages (92.8%) which can be built reproducibly in
> buster/amd64, out of 28523 source packages in total.
> (These 28523 source packages build 57448 binary packages.)
>
> But these tests are done without looking at the actual .deb files distributed
> from ftp.debian.org (and we always knew that and pointed it out:
> "93% reproducible _in our current test framework_".)
>

So I guess after we release buster, we should do a mass-source-nmu
no-change uploads to make the .deb reproducible as shipped in the
archive.

Is this thus then an effectively a maas bug-file request to
sourcefully rebuild packages?

--
Regards,

Dimitri.

Reply | Threaded
Open this post in threaded view
|

Re: Debian Buster will only be 54% reproducible (while we could be at >90%)

Holger Levsen-2
In reply to this post by peter green-2
On Wed, Mar 06, 2019 at 09:40:23AM +0000, peter green wrote:
> > Because of their design, binNMUs are unreproducible, see #894441 [3] for
> > the details (in short: binNMUs are not what they are ment to be: the source
> > is changed and thrown away)
> To be specific, the source tree is extracted, then an entry is added to
> debian/changelog and then the package is built. This modified source
> tree is not retained.

indeed.

> It seems to me that binnmus could be made reproducible by storing the
> debian/changelog modifications in the buildinfo, then re-applying it
> at reproduction time.

that's actually the case nowadays, not sure since when. eg
https://buildinfos.debian.net/ftp-master.debian.org/buildinfo/2019/04/23/dmtx-utils_0.7.6-1.1+b1_kfreebsd-amd64.buildinfo
starts like this:


Format: 1.0
Source: dmtx-utils (0.7.6-1.1)
Binary: dmtx-utils
Architecture: kfreebsd-amd64
Version: 0.7.6-1.1+b1
Binary-Only-Changes:
 dmtx-utils (0.7.6-1.1+b1) sid; urgency=low, binary-only=yes
 .
   * Binary-only non-maintainer upload for kfreebsd-amd64; no source changes.
   * rebuild for libdmtx0b
 .
  -- kfreebsd-amd64 / kfreebsd-i386 Build Daemon (kamp) <[hidden email]>  Sun, 14 Apr 2019 01:03:43 +0000
[...]

I yet have to actually test if one can bit by bit reproduce binNMUs
with this information, but I'm quite very hopeful.


--
tschau,
        Holger

-------------------------------------------------------------------------------
               holger@(debian|reproducible-builds|layer-acht).org
       PGP fingerprint: B8BF 5413 7B09 D35C F026 FE9D 091A B856 069A AA1C

signature.asc (849 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Debian Buster will only be 54% reproducible (while we could be at >90%)

Holger Levsen-2
In reply to this post by Dimitri John Ledkov-3
On Sat, Mar 09, 2019 at 09:15:34PM +0000, Dimitri John Ledkov wrote:
[packages uploaded before December 2016, which didnt see binNMU since
then, dont have a .buildinfo file (and were built with a dpkg which
produces unreproducible packages]
> So I guess after we release buster, we should do a mass-source-nmu
> no-change uploads to make the .deb reproducible as shipped in the
> archive.

maybe.

I think I first want to have the numbers right (and then current) to see
how many packages are affected.

Also: a not so low percentages of packages is still uploaded as
non-source-only upload. I *believe* this should be fine (as nowadays the
upload contains the .buildinfo file of that developer build, so if a
package is reproducible this .buildinfo file can be taken).

> Is this thus then an effectively a maas bug-file request to
> sourcefully rebuild packages?

no, at least not yet.


--
tschau,
        Holger

-------------------------------------------------------------------------------
               holger@(debian|reproducible-builds|layer-acht).org
       PGP fingerprint: B8BF 5413 7B09 D35C F026 FE9D 091A B856 069A AA1C

signature.asc (849 bytes) Download Attachment