simplifying rebootstrap - breaking hurd bootstrap?

classic Classic list List threaded Threaded
9 messages Options
Reply | Threaded
Open this post in threaded view
|

simplifying rebootstrap - breaking hurd bootstrap?

Helmut Grohne
Hi,

I intend to simplify the toolchain bootstrap for rebootstrap and this
has consequences for hurd.

The current linux/glibc toolchain bootstrap essentially works like this:
 * cross binutils
 * linux-libc-dev
 * gcc stage1 cross compiler (baremetal)
 * glibc stage1 (headers + stub libc.so)
 * gcc stage2 cross compiler
 * glibc stage2 (full libc without extra features such as systemtap)
 * gcc stage3 cross compiler
 * gcc rtlibs

The current linux/musl toolchain bootstrap is already simpler. It skips
"glibc stage1" and "gcc stage2" and directly builds the full musl using
the baremetal compiler. Quite a while ago, glibc was also converted to
support this sequence with fewer stages and this is what upstream's
build_many_glibcs.py does and what is tested. In Debian, we should
follow suit. And for everything except hurd-any, this seems to just
work.

The current hurd/glibc toolchain bootstrap seems to work like this:
 * cross binutils
 * linux-libc-dev
 * gnumach stage1
 * gcc stage1 cross compiler (baremetal)
 * hurd stage1
 * cross mig
 * glibc stage1 (headers + stub libc.so)
 * gcc stage2 cross compiler
 * hurd stage2
 * glibc stage2 (full libc without extra features such as systemtap)
 * gcc stage3 cross compiler (including rtlibs)

When removing glibc stage1 and gcc stage2, we'll also have to somehow
merge the hurd stage1 and stage2. I have no clue how to do that.

To make matters worse, the hurd bootstrap is currently somewhat broken.
When using gcc-10, hurd stage3 fails due to -fno-commons (#955447,
fixed-upstream, thanks Samuel). When using gcc-9, I see an unpack error
for hurd-headers-dev which ships /usr/include/sys/procfs.h without
declaring a conflict with libc6-dev-i386. Last year, I saw a build
failure for hurd stage1. Not sure whether that is still present. In any
case, a flaky bootstrap sequence doesn't make working on this any
simpler.

Given the above, I want to move to the simplified toolchain bootstrap
sequence soon. It is unfortunate that doing so will break hurd, but I
believe that the benefits are more important. The roadmap roughly is:
 * Some refactoring to ease transitioning to the new sequence.
 * Add a flag REDUCED_TOOLCHAIN_BOOTSTRAP=yes/no to select the short or
   long toolchain bootstrap. Everything but hurd-any will default to
   yes.
 * Some time later remove the long toolchain bootstrap sequence.

It would be very good if some hurd porters (e.g. busy Samuel) could
follow up on this mail. Keeping hurd support in rebootstrap requires
your effort. This is not urgent just yet, but I'd like to complete this
within a month.

As a first step, I suggest that someone (likely Samuel) replies to this
mail explaining why we presently need hurd stage1 and stage2 to be
separate. stage1 seems to skip libihash. Can we defer building libihash
until stage3? Can we build it in stage1? Is this the only difference
between stage1 and stage2?

Helmut

Reply | Threaded
Open this post in threaded view
|

Re: simplifying rebootstrap - breaking hurd bootstrap?

Samuel Thibault-8
Hello,

Helmut Grohne, le lun. 06 avril 2020 11:58:53 +0200, a ecrit:
> Quite a while ago, glibc was also converted to support this sequence
> with fewer stages and this is what upstream's build_many_glibcs.py
> does and what is tested.

And the hurd port is tested there as well, so it should be working.

> As a first step, I suggest that someone (likely Samuel) replies to this
> mail explaining why we presently need hurd stage1 and stage2 to be
> separate. stage1 seems to skip libihash. Can we defer building libihash
> until stage3? Can we build it in stage1? Is this the only difference
> between stage1 and stage2?

It seems to me that libihash is indeed the only difference. It also
seems that the reason for it was libpthread's use of libihash.
Apparently libihash use in libpthread was removed in January 2018,
and we forgot that we could them drop stage2.

So it seems we can indeed just drop stage2. How should I proceed to get
a smooth transition between hurd and rebootstrap script? Or should I
perhaps just move on and drop in the hurd package the existing stage2
and rename stage3 into stage2, and do the same in the bootstrap script
and submit a MR doing it, leaving bootstrap completely broken between
those two actions?

Samuel

Reply | Threaded
Open this post in threaded view
|

Re: simplifying rebootstrap - breaking hurd bootstrap?

Samuel Thibault-8
Samuel Thibault, le lun. 06 avril 2020 12:43:05 +0200, a ecrit:
> So it seems we can indeed just drop stage2.

For a start, I have attached what I just committed to glibc, to express
that glibc doesn't need libihash-dev any more.

Samuel
Reply | Threaded
Open this post in threaded view
|

Re: simplifying rebootstrap - breaking hurd bootstrap?

Helmut Grohne
In reply to this post by Samuel Thibault-8
Hi Samuel,

Thank you for the quick reply.

On Mon, Apr 06, 2020 at 12:43:05PM +0200, Samuel Thibault wrote:
> And the hurd port is tested there as well, so it should be working.

Right. We should aim to reproduce what is done there to minimize the
chance of failures and regressions.

> It seems to me that libihash is indeed the only difference. It also
> seems that the reason for it was libpthread's use of libihash.
> Apparently libihash use in libpthread was removed in January 2018,
> and we forgot that we could them drop stage2.

Awesome news.

> So it seems we can indeed just drop stage2. How should I proceed to get
> a smooth transition between hurd and rebootstrap script? Or should I
> perhaps just move on and drop in the hurd package the existing stage2
> and rename stage3 into stage2, and do the same in the bootstrap script
> and submit a MR doing it, leaving bootstrap completely broken between
> those two actions?

I don't think there is any need for a smooth transition. We should go
the most efficient path as both hurd and rebootstrap are understaffed.
To me that seems to be:
 * I attempt to simplify the bootstrap to only using hurd stages 1 and
   3.
 * I give you a notice when done (mail or irc).
 * You delete stage2 and rename stage3 to stage2.
 * You upload hurd (preferably together with the -fno-commons changes).
 * You give me a notice (mail or irc).
 * I handle the rebootstrap side. I don't think there is any point in
   bothering you any more than absolutely necessary.

Also hurd's control file has a comment with pending changes after
#818618 is fixed. That bug was fixed in January 2019. Can you also
attempt to clean that up or update the comment to say what it is blocked
on now? (<- wishlist) Though #913965 remains open. Is that still
relevant after fixing #818618?

Unless you reply, I'll assume your consent to that plan and I will be
following up with my results. That'll take a few days depending on the
mood of jenkins.

Helmut

Reply | Threaded
Open this post in threaded view
|

Re: simplifying rebootstrap - breaking hurd bootstrap?

Samuel Thibault-8
Helmut Grohne, le lun. 06 avril 2020 13:45:04 +0200, a ecrit:
>  * I attempt to simplify the bootstrap to only using hurd stages 1 and
>    3.

That will probably boil down to applying the glibc patch I have and just
skip building hurd stage2

>  * (preferably together with the -fno-commons changes).

Sure :) They are already in git.

Ok for the plan!

> Though #913965 remains open. Is that still relevant after fixing
> #818618?

In principle yes, I would get a rejection from dak. But now that we are
on debian-ports it might be simpler and go through, I can give it a try.

Samuel

Reply | Threaded
Open this post in threaded view
|

Re: simplifying rebootstrap - breaking hurd bootstrap?

Samuel Thibault-8
In reply to this post by Helmut Grohne
Hello,

Helmut Grohne, le lun. 06 avril 2020 11:58:53 +0200, a ecrit:
> I see an unpack error for hurd-headers-dev which ships
> /usr/include/sys/procfs.h without declaring a conflict with
> libc6-dev-i386.

FI, I have committed the attached patch to move sys/ files to multiarch
places. That shoud be enough to avoid most conflicts. Moving other
headers from hurd-dev will need the next version of glibc (2.30-5) in
which I made debian/sysdeps/hurd.mk symlink from multiarch places when
they are there.

Samuel
Reply | Threaded
Open this post in threaded view
|

Re: simplifying rebootstrap - breaking hurd bootstrap?

Samuel Thibault-8
In reply to this post by Samuel Thibault-8
Samuel Thibault, le lun. 06 avril 2020 13:53:49 +0200, a ecrit:
> Helmut Grohne, le lun. 06 avril 2020 13:45:04 +0200, a ecrit:
> >  * I attempt to simplify the bootstrap to only using hurd stages 1 and
> >    3.
>
> That will probably boil down to applying the glibc patch I have and just
> skip building hurd stage2

Yes, and drop

                       test "$2" = stage1 || apt_get_install "libihash-dev:$1"

Samuel

Reply | Threaded
Open this post in threaded view
|

Re: simplifying rebootstrap - breaking hurd bootstrap?

Helmut Grohne
In reply to this post by Helmut Grohne
Hi Samuel,

On Mon, Apr 06, 2020 at 01:45:04PM +0200, Helmut Grohne wrote:
> I don't think there is any need for a smooth transition. We should go
> the most efficient path as both hurd and rebootstrap are understaffed.
> To me that seems to be:
>  * I attempt to simplify the bootstrap to only using hurd stages 1 and
>    3.

I'v succeeded doing it once. It's committed to rebootstrap.git, but
jenkins.d.n hasn't exercised the new code for hurd yet. I'm still
working on the fallout, but I'm confident that it'll work out. As of
today, rebootstrap no longer uses hurd stage2.

>  * I give you a notice when done (mail or irc).

Poke. You can move ahead any time. There is no urgency attached, but
please tell me when you do.

>  * You delete stage2 and rename stage3 to stage2.
>  * You upload hurd (preferably together with the -fno-commons changes).
>  * You give me a notice (mail or irc).
>  * I handle the rebootstrap side. I don't think there is any point in
>    bothering you any more than absolutely necessary.

Helmut

Reply | Threaded
Open this post in threaded view
|

Re: simplifying rebootstrap - breaking hurd bootstrap?

Samuel Thibault-8
Hello,

Helmut Grohne, le mer. 15 avril 2020 23:08:55 +0200, a ecrit:
> >  * I give you a notice when done (mail or irc).
>
> Poke. You can move ahead any time. There is no urgency attached, but
> please tell me when you do.

We had some changes pending anyway, so I uploaded it :)

Samuel