Re: #914897: debootstrap, buster: Please disabled merged /usr by default

classic Classic list List threaded Threaded
13 messages Options
Reply | Threaded
Open this post in threaded view
|

Re: #914897: debootstrap, buster: Please disabled merged /usr by default

Didier 'OdyX' Raboud-5
Dear Hideki, dear src:debootstrap maintainers,

tl;dr: debootstrap maintainers; can you agree to disable "merged /usr" by
default now, or are you OK letting the TC decide on this subject?

Longer version:

As you might be aware, #914897 (initially filed on src:debootstrap) has now
been reassigned to the Technical Committee.  As, formally, the Maintainer of
src:debootstrap is "[hidden email] and the current Uploaders", I would like
to make sure that the TC is not going to overrule unnecessarily.

Hideki, if I read the debootstrap history correctly, you enabled "merged /usr"
by default in debootstrap 1.0.102.  Given the recent discussion in debian-
devel@ (starting at [0]) and on #914897, could you (or anyone speaking as with
a "debootstrap maintainer" hat on) state if, either of:

* you would be willing to toggle the "merged /usr" default in debootstrap in a
  subsequent upload;
* you maintain that the "merged /usr" default (to yes) is here to stay.

Many thanks in advance for your consideration,

    OdyX

[0] https://lists.debian.org/debian-devel/2018/11/msg00354.html

P.S. I'm aware that this might sound formal, or dismissive of Julien's
     statements.  I just _really_ don't want the TC to eventually overrule
     "the debootstrap maintainers" without need.


Reply | Threaded
Open this post in threaded view
|

Re: Bug#914897: #914897: debootstrap, buster: Please disabled merged /usr by default

Ansgar Burchardt-8
On Fri, 2018-11-30 at 19:40 +0100, Didier 'OdyX' Raboud wrote:
> Dear Hideki, dear src:debootstrap maintainers,
>
> tl;dr: debootstrap maintainers; can you agree to disable "merged /usr" by
> default now, or are you OK letting the TC decide on this subject?

There were discussions about enabling this by default years ago, I
don't think minor issues should be a reason to delay this change.

Note that it has been delayed for after the stretch release as there
were major issues back then (it was enabled by default for a short time
in stretch).  It took >5 months for someone to find a problem this time
which is pretty good for any change.

Ansgar

Reply | Threaded
Open this post in threaded view
|

Re: Bug#914897: #914897: debootstrap, buster: Please disabled merged /usr by default

Ansgar Burchardt-8
Ansgar Burchardt writes:
> There were discussions about enabling this by default years ago, I
> don't think minor issues should be a reason to delay this change.
>
> Note that it has been delayed for after the stretch release as there
> were major issues back then (it was enabled by default for a short time
> in stretch).  It took >5 months for someone to find a problem this time
> which is pretty good for any change.

So, I went through all reproducible build failures in unstable without
notes and added notes for differences caused by building in merged-/usr
vs non-merged-/usr packages.  Together with what other people tagged,
about 60 problems were found.  (The oldest rebuild result for unstable
is 17 days old, the merged-/usr variation was deployed before that[1].)

  [1] https://bugs.debian.org/901473#43

There might be some more that as diffoscope had problems or the diff was
too large for some packages (but I'll assume that in this case
merged-/usr is not the only problem the package had).

This should cover all packages that had no other problems and were
reproducible before, that is about 80% of the archive.

Assuming the rate is similar for packages which have other reproducible
build problems, I would expect 60 / 80*20 = 15 more problems.

I haven't looked at build failures, but I would expect these to be
usually not caused by merged-/usr.

So an estimated ~80 packages which might need adjustments for building
correctly in both merged-/usr and non-merged-/usr.  I guess less than
for the average new release of GCC ;-)

The problems are usually easy to fix by passing an explicit path the the
package's configure script at build time; of course there might be some
package where it is more complicated.

Ansgar

Reply | Threaded
Open this post in threaded view
|

Re: Bug#914897: #914897: debootstrap, buster: Please disabled merged /usr by default

Holger Levsen-2
Hi,

Ansgar, thanks a lot for doing this!

On Sat, Dec 01, 2018 at 06:06:28PM +0100, Ansgar Burchardt wrote:
> So, I went through all reproducible build failures in unstable without
> notes and added notes for differences caused by building in merged-/usr
> vs non-merged-/usr packages.  Together with what other people tagged,
> about 60 problems were found.  (The oldest rebuild result for unstable
> is 17 days old, the merged-/usr variation was deployed before that[1].)

https://tests.reproducible-builds.org/debian/issues/unstable/paths_vary_due_to_usrmerge_issue.html
lists these packages.

what surprises me currently, are those 3 packages which are reproducible
in buster (even though we also vary usrmerge when testing buster).


--
cheers,
        Holger

-------------------------------------------------------------------------------
               holger@(debian|reproducible-builds|layer-acht).org
       PGP fingerprint: B8BF 5413 7B09 D35C F026 FE9D 091A B856 069A AA1C

signature.asc (849 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Bug#914897: #914897: debootstrap, buster: Please disabled merged /usr by default

Simon McVittie-7
On Sat, 01 Dec 2018 at 17:18:35 +0000, Holger Levsen wrote:
> https://tests.reproducible-builds.org/debian/issues/unstable/paths_vary_due_to_usrmerge_issue.html
> lists these packages.
>
> what surprises me currently, are those 3 packages which are reproducible
> in buster (even though we also vary usrmerge when testing buster).

gzip, icecc and mailagent were most recently built for buster on
2018-11-08, which might be long enough ago that the buster chroot was
not merged-/usr?

    smcv

Reply | Threaded
Open this post in threaded view
|

Re: #914897: debootstrap, buster: Please disabled merged /usr by default

Hideki Yamane-4
In reply to this post by Didier 'OdyX' Raboud-5
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512

Hi,

On Fri, 30 Nov 2018 19:40:45 +0100
"Didier 'OdyX' Raboud" <[hidden email]> wrote:
> tl;dr: debootstrap maintainers; can you agree to disable "merged /usr" by
> default now, or are you OK letting the TC decide on this subject?

 Hmm, I'm still considering what's the good way...

> Hideki, if I read the debootstrap history correctly, you enabled "merged /usr"
> by default in debootstrap 1.0.102.

 Yes, that's right. #839046 was filed in Sep 2016, and uploaded in Jun 2018.

> Given the recent discussion in debian-
> devel@ (starting at [0]) and on #914897, could you (or anyone speaking as with
> a "debootstrap maintainer" hat on) state if, either of:
>
> * you would be willing to toggle the "merged /usr" default in debootstrap in a
>   subsequent upload;
> * you maintain that the "merged /usr" default (to yes) is here to stay.

 Well, with a quick look to the thread (I cannot follow all of the email in it,
 tons of emails...), I cannot find the discussion about

  - What is the problem? (broken build for which packages? Just R?)
  - How many packages are affected?
  - Why it was caused? (just symlink to /bin or /sbin isn't enough
    to deal with it?)
  - Does it cause any failure on users' machine?

 So, I want to listen above things (not thought or idea), then reply to
 your question. Please quote if someone can do it.


- --
Regards,

 Hideki Yamane     henrich @ debian.org/iijmio-mail.jp
 http://wiki.debian.org/HidekiYamane
-----BEGIN PGP SIGNATURE-----

iQIzBAEBCgAdFiEEQZYJUbYxgXxV33EdBBJ4KqpAHFMFAlwDzlUACgkQBBJ4KqpA
HFOnfxAAv8s9EQwntX9SBHALIY+5X1Ma98aMhrzZ2SWDt1txznXRt18z/85oOWqs
FGLrm2QY159qWEG2lpsWhAIr7wQJBPcFH5MRQcn6pDM6pXB1ioaTsW9uhd/AMl+s
mCyvWW0xtJ1ww2EXV2hN5X0K4AAre2rajb0P4p6efeY5V9sbMQ/gZa+L2sJuL1P/
/6fK4Kxe893lVuZ3oxtOhKRkdgi1V1X63kUURofuTSZiVzeGYWAuPdnHBxADs9vK
kk6mpUFkYSeOfg45h2KQzUqeTsX5GTogWIFqEOAJ0KJGDusOiFEPWL/pus+De1E7
cyEX2i6yq3wOOQBov5/eNH2gMs9pDaOqM8hR0tjvya4aAJOa7VyFY2GzMdsEHdQe
Ay7EtzG3RLwuiQ0XrSmIyaDdlJpofCGernNgVu+dnBJb/1U4RHgneVbIELULGUYm
DGFov6FpeUQB6wc/fsaoDWQBiwwNCS2qkJnZJg5nu4ne12NqnERqoq2lIR3ivSe2
1Oi9v/ClKqNSKGLAIoRVvllZhs9W1ppwkZIqtC0mZlN05nw7Wyrj4YoRbJ4r70Rd
rdQzTntchOXbYOmdt2H6yUdpnJJoA46+OxlwykvjrUnDnzgheNMJ0wRh36LcOz50
pjQBQGGVVl/9+Tjw/vSCu+alwLwPY34YFOM8I4fh/V0OHbO4fNE=
=yD0D
-----END PGP SIGNATURE-----

Reply | Threaded
Open this post in threaded view
|

Re: Bug#914897: #914897: debootstrap, buster: Please disabled merged /usr by default

Simon McVittie-7
On Sun, 02 Dec 2018 at 21:21:40 +0900, Hideki Yamane wrote:
>   - What is the problem? (broken build for which packages? Just R?)

The problem we're aware of is:

Some packages auto-detect the absolute path to an executable (for example
bash or perl) and hard-code it into their output (for example the #! line
of the bash scripts in quilt). When built on a system with merged /usr,
they can detect and hard-code a path that exists on merged /usr systems
but does not exist on systems with unmerged /usr, such as #!/usr/bin/bash
or #!/bin/perl.  This results in the package working fine on merged-/usr
systems but not on unmerged-/usr systems.

The quilt bug https://bugs.debian.org/913226 is the one that prompted
me to look at this (it caused build failures in a Debian derivative
that I work on, which is in the process of rebasing from stretch to
buster), and is a fairly typical example. The problem case usually
involves an executable that is canonically in /bin, like bash or sed,
being detected in /usr/bin. I've also seen one occurrence of the reverse,
where an executable that is canonically in /usr/bin is found in /bin on
a merged /usr system, although I can't remember which package that was
(it would have to have been re-ordering PATH to look in /bin first).

The same packages would be similarly broken if they were built with
/usr/local/bin in the PATH, on a system where the executable they are
looking for is present in /usr/local/bin. For example, versions of quilt
where #913226 is unfixed could exhibit a similar problem if built on a
system with a /usr/local/bin/bash.

The same things can happen with /usr/sbin and /sbin.

>   - How many packages are affected?

In <https://lists.debian.org/debian-ctte/2018/12/msg00000.html>,
Ansgar points to 60 packages found to have this failure mode in a rebuild
covering around 80% of the archive, and estimates that there should be
about 80 packages in total affected by this class of bug.

>   - Why it was caused? (just symlink to /bin or /sbin isn't enough
>     to deal with it?)

See the quilt bug I linked above, which is a typical example.

On merged /usr systems there is no problem with the "broken" packages,
because the compatibility symlinks /bin -> /usr/bin and /sbin -> /usr/sbin
ensure that paths like /bin/sh and /usr/bin/sh both work equally well.
The problem only occurs when they are *built* on a system *with* merged
/usr, then *used* on a system *without* merged /usr.

>   - Does it cause any failure on users' machine?

Yes, for example see the quilt bug. The failure mode is:

* A developer, D, has a system with merged /usr
* A user, U, has a system without merged /usr
* D builds a package that has this type of bug
  (without using a non-merged-/usr chroot, for example --variant=buildd from
  debootstrap >= 1.0.111)
* The package works for D
* The package doesn't work for U

    smcv

Reply | Threaded
Open this post in threaded view
|

Re: Bug#914897: #914897: debootstrap, buster: Please disabled merged /usr by default

Holger Levsen-2
In reply to this post by Simon McVittie-7
On Sat, Dec 01, 2018 at 10:21:50PM +0000, Simon McVittie wrote:
> gzip, icecc and mailagent were most recently built for buster on
> 2018-11-08, which might be long enough ago that the buster chroot was
> not merged-/usr?

right. I triggered their builds and now they are all shown as unreproducible.


--
cheers,
        Holger

-------------------------------------------------------------------------------
               holger@(debian|reproducible-builds|layer-acht).org
       PGP fingerprint: B8BF 5413 7B09 D35C F026 FE9D 091A B856 069A AA1C

signature.asc (849 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Bug#914897: #914897: debootstrap, buster: Please disabled merged /usr by default

Hideki Yamane-2
In reply to this post by Simon McVittie-7
Hi,

 Thanks Simon, it's perhaps clear for me now.

On Sun, 2 Dec 2018 15:15:21 +0000
Simon McVittie <[hidden email]> wrote:
> >   - What is the problem? (broken build for which packages? Just R?)
>
> The problem we're aware of is:
>
> Some packages auto-detect the absolute path to an executable (for example
> bash or perl) and hard-code it into their output (for example the #! line
> of the bash scripts in quilt).

 Can we check and track this behavior in our packages?

 Once disable merged-usr is good to prevent confusion but we detect such
 as a bug for manage non-merged-usr and merged-usr Debian system in the end,
 right? (or do you want to stay change in debootstrap 1.0.111 forever?)


--
Hideki Yamane <[hidden email]>

Reply | Threaded
Open this post in threaded view
|

Re: Bug#914897: #914897: debootstrap, buster: Please disabled merged /usr by default

Ansgar Burchardt-8
Hi,

Hideki Yamane writes:

> On Sun, 2 Dec 2018 15:15:21 +0000
> Simon McVittie <[hidden email]> wrote:
>> >   - What is the problem? (broken build for which packages? Just R?)
>>
>> The problem we're aware of is:
>>
>> Some packages auto-detect the absolute path to an executable (for example
>> bash or perl) and hard-code it into their output (for example the #! line
>> of the bash scripts in quilt).
>
>  Can we check and track this behavior in our packages?

The Reproducible Builds project was so kind to help and now runs one
build in a non-merged-/usr and a second build in a merged-/usr
environment.  Packages that hardcode the path to utilities, but would
pick up the wrong one in a merged-/usr environment will result in a
difference between the two builds and can thus be found.

See [1] for an overview of issues found this way; as the entire archive
was already rebuilt in this setup, there shouldn't be many more issues
of this type that we don't know about[2].

Not all of these differences even cause issues as in a few packages the
utility with the hardcoded path is not even used at all.

Bug reports were already submitted for over half the packages, often
including a simple patch (usually something like adding BASH=/bin/bash
to dh_auto_configure).

So we look to be on a good track to address the remaining issues.

I don't think that the debootstrap default has to be reverted
temporarily again to deal with this: there are only very few packages
causing problems and these should have a patch soon.

In addition one has to actually built one of the very few packages in a
merged-/usr environment and then install them in a non-merged-/usr
environment to actually trigger the problem and debootstrap already
defaults to non-merged-usr for buildd chroots for now.

Ansgar

  [1] https://tests.reproducible-builds.org/debian/issues/unstable/paths_vary_due_to_usrmerge_issue.html
  [2] https://bugs.debian.org/914897#81

Reply | Threaded
Open this post in threaded view
|

Re: Bug#914897: tech-ctte: Should debootstrap disable merged /usr by default? /usr by default

Simon McVittie-7
In reply to this post by Hideki Yamane-2
Control: retitle 914897 tech-ctte: Should debootstrap disable merged /usr by default?

I'm retitling the bug to avoid misrepresenting the technical committee's
position on this. We have been asked to overrule the debootstrap
maintainer, but we have not yet come to a conclusion on whether we should.

On Wed, 05 Dec 2018 at 13:25:36 +0900, Hideki Yamane wrote:
>  Can we check and track this behavior in our packages?

Yes, we now do. The reproducible-builds infrastructure now uses unmerged
/usr for the first build and merged /usr for the second, since
<https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=901473> was fixed.

debootstrap since 1.0.111 also mitigates this class of bugs by disabling
merged /usr for --variant=buildd chroots (this was
<https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=914208>).

Julien Cristau thinks #914208 was a sufficient/proportionate change, and
doesn't want to go further and default to --no-merged-usr for non-buildd
chroots (and in particular new debian-installer installations).

Ian Jackson thinks #914208 is not a sufficient answer (Ian, I hope I'm
not misrepresenting you here?), and escalated this bug to the technical
committee, asking us to overrule the debootstrap maintainers.

If the debootstrap/debian-installer maintainers agree with Ian on this,
then there is no need for the technical committee to consider his request
to overrule you, which is why Didier asked for your opinion on this
issue before attempting to come to a decision. If you agree with Julien,
then the technical committee still needs to consider this question.

>  Once disable merged-usr is good to prevent confusion but we detect such
>  as a bug for manage non-merged-usr and merged-usr Debian system in the end,
>  right? (or do you want to stay change in debootstrap 1.0.111 forever?)

The technical committee have not come to a conclusion on this.

My personal opinions (not overruling anyone):

If merged /usr becomes the only supported system layout at some future
time, then the change in debootstrap 1.0.111 can certainly be reverted
at that time. (If merged /usr does not become the only supported system
layout, this does not apply.)

It might also be considered appropriate to revert the change in
debootstrap 1.0.111 if data from reproducible-builds demonstrates that
bugs similar to #913226 have all been fixed or are very rare, but this
should be done cautiously, and certainly not before buster is released.

    smcv

Reply | Threaded
Open this post in threaded view
|

Re: Bug#914897: #914897: debootstrap, buster: Please disabled merged /usr by default

Gunnar Wolf via nm
In reply to this post by Ansgar Burchardt-8
Ansgar Burchardt dijo [Wed, Dec 05, 2018 at 08:17:56AM +0100]:

> The Reproducible Builds project was so kind to help and now runs one
> build in a non-merged-/usr and a second build in a merged-/usr
> environment.  Packages that hardcode the path to utilities, but would
> pick up the wrong one in a merged-/usr environment will result in a
> difference between the two builds and can thus be found.
>
> See [1] for an overview of issues found this way; as the entire archive
> was already rebuilt in this setup, there shouldn't be many more issues
> of this type that we don't know about[2].
> (...)

Thanks for this report, Ansgar. This clearly sums up the issue. I am
amazed the whole archive managed to be rebuilt on such a short time!

I guess the next step is to file the multiple bugs pending to be
filed.

> Bug reports were already submitted for over half the packages, often
> including a simple patch (usually something like adding BASH=/bin/bash
> to dh_auto_configure).
>
> So we look to be on a good track to address the remaining issues.

...And I guess this points towards the Technical Committee not having
to intervene in the issue. Which is, IMO, the best possible outcome.

Thanks to everybody following through with this!

Reply | Threaded
Open this post in threaded view
|

Re: Bug#914897: tech-ctte: Should debootstrap disable merged /usr by default? /usr by default

Hideki Yamane-2
In reply to this post by Simon McVittie-7
Hi,

On Wed, 5 Dec 2018 08:39:27 +0000
Simon McVittie <[hidden email]> wrote:
> It might also be considered appropriate to revert the change in
> debootstrap 1.0.111 if data from reproducible-builds demonstrates that
> bugs similar to #913226 have all been fixed or are very rare, but this
> should be done cautiously, and certainly not before buster is released.

 Okay, my opinion is "Push usr-merge effort forward, fix those issues
 with it as bug that is tracked at reproducible builds(*), and turn it
 on again as default (probably after buster cycle)".

 *) https://tests.reproducible-builds.org/debian/issues/unstable/paths_vary_due_to_usrmerge_issue.html


--
Regards,

 Hideki Yamane     henrich @ debian.org/iijmio-mail.jp