Status and open questions for debian-installer with backports support

Next Topic
 
classic Classic list List threaded Threaded
8 messages Options
Reply | Threaded
Open this post in threaded view
|

Status and open questions for debian-installer with backports support

Cyril Brulebois-4
Hi,

It's been a while since my first proof of concept (demonstrated at the
2014 mini-DebConf in Paris), but I think I've managed to reach a point
where I'm rather content and ready to get bits and pieces merged where
they belong. That doesn't mean I have everything figured out, that's
why I'm reaching out to both the installer & images team!


The primary question we need to discuss is what we are aiming for. I'll
start with a (not so) little flashback to explain what I've worked on,
why, and what is likely (not) going to be our limited target.

This is a long read… I've highlighted “to do”-like items with an arrow
(=>).


debian-installer
================

First things first, we needed a debian-installer build that knew about a
backported linux kernel, so I hacked my way in debian-installer.git; it
wasn't too hard since we can count on apt to resolve dependencies, and
after all, we only needed the kernel-image udeb and its modules.

The latest branch is stretch-backports-v2, which combines both a number
of commits that are aimed at the master branch, and a couple others that
want to only be used in a stretch-backports branch:

  https://salsa.debian.org/installer-team/debian-installer/commits/stretch-backports-v2


net-retriever
-------------

As usual, I've tested the resulting installer with the netboot(-gtk)
image and of course it failed to load additional udebs. These netboot*
images are aimed at booting from the network, and they also need to
fetch additional udebs from a mirror. Since net-retriever only knows
about a single source of udebs, it needed to be told about multiple
sources (stretch and stretch-backports) and also how to merge entries.
I've detailed my choices in this commit in particular:

  https://salsa.debian.org/installer-team/net-retriever/commit/09535b0613a85e02d511b92b60683b20405b6d42

For more details, see the whole stretch-backports-v2 branch for
net-retriever:

  https://salsa.debian.org/installer-team/net-retriever/commits/stretch-backports-v2

Now, it doesn't matter too much, since the stretch-backports repository
will evolve over time, and udebs for the specific kernel used during the
d-i build will disappear. I think this is too much of a volatile (no
pun intended) target to support. It's still a very nice way to test d-i
without having to build an ISO with debian-cd, so I'd like to merge
these changes in net-retriever anyway.


base-installer
--------------

Second, it makes quite some sense to not only run d-i with a newer
kernel, but to also install it, so that users can boot from their brand
new hardware that wasn't supported by d-i with stable components only.

The kernel selection and installation is implemented in base-installer
(see library.sh), and that happens to run before apt-setup. Since I
didn't want to butcher sources.list manually to install the kernel
directly at this stage, I've decided to implement installing the latest
kernel from backports in a finish-install script (also shipped in the
base-installer package, for consistency), running at the very end of the
installation process.

This only happens when backports support was enabled (this is detected
through the presence of /etc/udebs-backports-source).

At first, using the linux-image-$arch metapackage looked easy enough but
of course that works for amd64, but not for i386 (one would need to pick
686 or 686-pae). I've still pushed a branch with this implementation as
it can be useful as is:

  https://salsa.debian.org/installer-team/base-installer/commits/stretch-backports-v0

We could probably modify library.sh to reuse it, but I'd like to have
minimal changes (for reasons explained in the very last section). We
could probably just iterate over all installed linux-image-'*' packages,
pick the one from the src:linux package, and upgrade it from the
backports repository.

=> I'll look into that later on.



apt-setup
---------

Also only when /etc/udebs-backports-source is present, the backports
service gets automatically enabled, which makes the later installation
of linux-image packages possible.

Single commit for this:

  https://salsa.debian.org/installer-team/apt-setup/commit/6133bdcd5a7903105ad967ab087f2f12f9d1c59d



debian-cd
=========

As mentioned above, tricks were needed in net-retriever for the netboot*
images, to make it possible to load linux kernel modules udebs. Those
tricks aren't needed when generating an ISO image. As a reminder, on the
d-i side, a cdrom_isolinux image is generated, and the debian-cd tool
needs to be configured to use that image instead of the official image
available on the configured mirror.

I didn't make this dynamic yet, but I've modified my local configuration
to set BACKPORTS=backports-list, to include a number of packages from
the stretch-backports suite. This file was generating by listing all
udebs produced by the linux source package (as done in the d-i build
system):

    USE_UDEBS_BACKPORTS_FROM=stretch-backports
    ...
    source=linux
    ...
    binaries=$(grep-dctrl -s Package -F Source $source $APTDIR/state/lists/*${USE_UDEBS_BACKPORTS_FROM}*Packages | awk '{print $2}')

=> This needs to become dynamic, probably through a helper tool that
   can be called automatically if backports are enabled in debian-cd.

=> It might be nice to have some kind of consistency check, at least
   to make sure the ABI is the same for the linux kernel produced by
   the d-i build, and for the modules available in the backports
   suite. [Note: this might be true for weekly builds too, see recent
   complaints/bug reports.] Bonus points if the source version is
   checked too, for extra caution.

As a consequence, all those were included in the ISO, in the following
file:

    debian/dists/stretch/main/debian-installer/binary-amd64/Packages.gz

This means no extra tweak is needed there, all udebs are available in a
single place: easy!

Of course we still need the modified apt-setup and base-installer
binaries. I've made them available as local packages (LOCALDEBS) and
they were automatically used.

The finish-install script ran as expected, but I ended up with a 4.16
kernel from backports in the installed system, as the linux-latest
binaries still point to it rather than to 4.17 (presumably because
linux/stretch-backports is still Needs-Build on mipsel at the moment).
It was downloaded from the network mirror.

=> It would be nice if we could include the linux-image-* packages from
   backports (and their dependencies) directly on the installation image,
   so as to be independent of what's happening in the stretch-backports
   suite on online mirrors. Users would have reproducible results over
   time, instead of a jumpy target. I don't know debian-cd enough to
   determine how feasible it would be. I'm also not sure what would happen
   if we had “old” linux-image packages on the ISO, and “newer” ones on
   mirrors.


Open questions
==============

Now that I've explained (in length…) how it works, I think it's high
time we define what our target is.

Due to the volatility of a backports suite, I would tend to not even try
to support netboot. Aiming for a least a netinst image would look good
to me. If we can manage that, we can probably also produce a CD1 for
those who want to have more packages than just what's on a netinst (I've
had at least one such request). I'm not going to debate how many
desktops we should have CD1 for, that's really not my call. I'll just
mention that size constraints might be tricky since we'll have both the
linux-image packages for the base suite and those from the backports
suite. The whole DVD/BR thing is probably entirely overkill.

=> Choice to make: netinst and maybe CD1(s)? Something else?

It's probably a good idea to consider building matching unofficial
image(s) with firmwares embedded. I'm not an expert regarding debian-cd
so help is much welcome. Tweaks might be needed to get firmwares
installed and/or upgraded from the backports repository, be it to embed
them on the image, or to make them available to the installed system.

=> Choice: support unofficial firmware-enabled image(s)?

=> Question: how to ensure firmwares from backports are available in d-i
   and in the installed system?

Finally, what architectures are we supporting? I can't speak for
debian-cd, but at least for the d-i part, we'll likely fork master into
stretch-backports and merge there. So we should keep track of any arch-
specific changes and debian-installer is likely to be buildable/built on
all architectures for no extra cost. If there are other bits that users
need to rely on, like flash-kernel and other components, I think they
should just consider installing testing instead. We can't reasonably be
expected to backport all the things.

=> Question: should we restrict architectures we build images for?

Last question: how often do we produce such an image? I don't think it's
reasonable to do that every time a new major version of linux is made
available in the backports repository. Doing that once around the middle
of the release cycle would look good to me. This would kind of mimic the
“and a half” thing we had in the past. Once we get the process right, we
might think about doing so another time or two, but we already have limited
humanpower to deal with stable point releases and alphas for testing
(at least oldstable is gone now)…

=> Question: what to advertise/communicate on? One-shot only is probably
   a good rule of thumb?


What to do with modified components
===================================

My current approach would be to get net-retriever, apt-setup, and
base-installer (once this last one is ready of course) updated in
stable. This would greatly help building, testing, and running
debian-installer without having to special-case everything in various
places. Of course this is subject to a green light from the release
team, but I might be trusted to get things right (and/or to get the
pieces working again if anything breaks).

We would be able to rely on their having built-in support for backports
(remember, it's only enabled when a specific file is present). We could
then just upload src:debian-installer to stretch-backports (after dak's
been patched to allow that), and point debian-cd to the resulting
images.


Cheers,
--
Cyril Brulebois ([hidden email])            <https://debamax.com/>
D-I release manager -- Release team member -- Freelance Consultant

signature.asc (849 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Status and open questions for debian-installer with backports support

Ben Hutchings-3
On Thu, 2018-08-02 at 10:56 +0200, Cyril Brulebois wrote:
[...]
> debian-installer
> ================
[...]
> net-retriever
> -------------
[...]
>   https://salsa.debian.org/installer-team/net-retriever/commit/09535b0613a85e02d511b92b60683b20405b6d42

This looks rather inefficient.  Maybe not a big deal for the small
number of udebs though.

(Time to put a more powerful scripting language in the installer?  Even
awk would be a step up!)

[...]
> base-installer
> --------------
[...]
> We could probably modify library.sh to reuse it, but I'd like to have
> minimal changes (for reasons explained in the very last section). We
> could probably just iterate over all installed linux-image-'*' packages,
> pick the one from the src:linux package, and upgrade it from the
> backports repository.

You mean the src:linux-latest package, right?

This is kind of unfortunate, but should work.  Even when the set of
flavours for a architecture is changed, src:linux-latest builds
transitional packages to support upgrades.

[...]
> debian-cd
> =========
[...]
> => It might be nice to have some kind of consistency check, at least
>    to make sure the ABI is the same for the linux kernel produced by
>    the d-i build, and for the modules available in the backports
>    suite. [Note: this might be true for weekly builds too, see recent
>    complaints/bug reports.] Bonus points if the source version is
>    checked too, for extra caution.

It is not a nice bonus, but really important that the source version is
equal.  Using module udebs older than the kernel image will usually
(but not always) work, but using newer module udebs will often result
in unresolvable symbols for some modules.

[...]
> The finish-install script ran as expected, but I ended up with a 4.16
> kernel from backports in the installed system, as the linux-latest
> binaries still point to it rather than to 4.17 (presumably because
> linux/stretch-backports is still Needs-Build on mipsel at the moment).

That was just forgotten, I'm afraid.  Bastian has just updated linux-
latest.

> It was downloaded from the network mirror.
>
> => It would be nice if we could include the linux-image-* packages from
>    backports (and their dependencies) directly on the installation image,
>    so as to be indpendent of what's happening in the stretch-backports
>    suite on online mirrors.

I agree, it should be possible to install without network sources
enabled.

>                             Users would have reproducible results over
>    time, instead of a jumpy target. I don't know debian-cd enough to
>    determine how feasible it would be. I'm also not sure what would happen
>    if we had “old” linux-image packages on the ISO, and “newer” ones on
>    mirrors.

If they *do* enable network sources then they should get the latest
version.  This is not so different from installing stable and getting a
security update instead of the version on the installation image.

> Open questions
> ==============
>
> Now that I've explained (in length…) how it works, I think it's high
> time we define what our target is.
>
> Due to the volatility of a backports suite, I would tend to not even try
> to support netboot.

Agreed.

>                     Aiming for a least a netinst image would look good
> to me. If we can manage that, we can probably also produce a CD1 for
> those who want to have more packages than just what's on a netinst (I've
> had at least one such request). I'm not going to debate how many
> desktops we should have CD1 for, that's really not my call. I'll just
> mention that size constraints might be tricky since we'll have both the
> linux-image packages for the base suite and those from the backports
> suite. The whole DVD/BR thing is probably entirely overkill.

As I understand it, CD#1 is already fairly useless for non-networked
(or limited network) installations and DVD#1 is a lot more useful.

> => Choice to make: netinst and maybe CD1(s)? Something else?
>
> It's probably a good idea to consider building matching unofficial
> image(s) with firmwares embedded. I'm not an expert regarding debian-cd
> so help is much welcome. Tweaks might be needed to get firmwares
> installed and/or upgraded from the backports repository, be it to embed
> them on the image, or to make them available to the installed system.

Yes, firmware would need to come from backports.

Since there is no dependency relation between the kernel and firmware
package, I don't prune old files until they are no longer referenced by
the kernel version in any supported suite.  So the CD image build would
only need to use firmware packages from backports, not two different
versions.

[...]
> Last question: how often do we produce such an image? I don't think it's
> reasonable to do that every time a new major version of linux is made
> available in the backports repository. Doing that once around the middle
> of the release cycle would look good to me. This would kind of mimic the
> “and a half” thing we had in the past. Once we get the process right, we
> might think about doing so another time or two, but we already have limited
> humanpower to deal with stable point releases and alphas for testing
> (at least oldstable is gone now)…

Doing this even once per release is a nice improvement, but I think
it's not enough to solve the problem of unsupported hardware (even on
x86).  And I think the more often this is done, the easier it will get.
 Still, I realise that you are likely to end up doing most of the work,
and only you can decide how much time to spend on it.

> => Question: what to advertise/communicate on? One-shot only is probably
>    a good rule of thumb?
[...]

I think what you're proposing is to announce this as a one-off, and not
to immediately make any promises about regular builds.  If so, I agree.

Ben.

--
Ben Hutchings
If you seem to know what you are doing, you'll be given more to do.


signature.asc (849 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Status and open questions for debian-installer with backports support

Cyril Brulebois-4
Hi,

Thanks for your comments. I really should have thought about putting
the kernel team in copy but it's been a long write-up…

Ben Hutchings <[hidden email]> (2018-08-03):
> This looks rather inefficient.  Maybe not a big deal for the small
> number of udebs though.

That's correct on both count: inefficient but also not a pratical issue.

> (Time to put a more powerful scripting language in the installer?
> Even awk would be a step up!)

I'd rather discuss that separately (too many topics already). The
net-retriever part is mainly going to be used by people preparing the
upload anyway.

> > base-installer
> > --------------
> [...]
> > We could probably modify library.sh to reuse it, but I'd like to have
> > minimal changes (for reasons explained in the very last section). We
> > could probably just iterate over all installed linux-image-'*' packages,
> > pick the one from the src:linux package, and upgrade it from the
> > backports repository.
>
> You mean the src:linux-latest package, right?
Exactly, sorry for the typo.

> This is kind of unfortunate, but should work.  Even when the set of
> flavours for a architecture is changed, src:linux-latest builds
> transitional packages to support upgrades.

Right, that's what I had in mind, having witnessed the transition from
586 to 686.

> > debian-cd
> > =========
> [...]
> > => It might be nice to have some kind of consistency check, at least
> >    to make sure the ABI is the same for the linux kernel produced by
> >    the d-i build, and for the modules available in the backports
> >    suite. [Note: this might be true for weekly builds too, see recent
> >    complaints/bug reports.] Bonus points if the source version is
> >    checked too, for extra caution.
>
> It is not a nice bonus, but really important that the source version is
> equal.  Using module udebs older than the kernel image will usually
> (but not always) work, but using newer module udebs will often result
> in unresolvable symbols for some modules.
Right now, we build debian-installer and trigger debian-cd builds under
a udeb freeze, so we cannot be out of sync for alpha releases. That's
why I called it a “bonus” for regular uploads.

But yeah, for weekly builds we don't have such a guarantee, and we won't
have that either for backports (even if we can just coordinate with the
kernel team when we're about to build images with backports enabled).

> > The finish-install script ran as expected, but I ended up with a 4.16
> > kernel from backports in the installed system, as the linux-latest
> > binaries still point to it rather than to 4.17 (presumably because
> > linux/stretch-backports is still Needs-Build on mipsel at the moment).
>
> That was just forgotten, I'm afraid.  Bastian has just updated linux-
> latest.

Yes, I've seen the git notification pop up on IRC earlier today, thanks!

> >                             Users would have reproducible results over
> >    time, instead of a jumpy target. I don't know debian-cd enough to
> >    determine how feasible it would be. I'm also not sure what would happen
> >    if we had “old” linux-image packages on the ISO, and “newer” ones on
> >    mirrors.
>
> If they *do* enable network sources then they should get the latest
> version.  This is not so different from installing stable and getting a
> security update instead of the version on the installation image.

Right, that seems reasonable.

(I think I was still a bit surprised by having had 4.16 installed with
4.17 running in d-i; which might not work too well… The other way
around, i.e. a higher version installed, shouldn't be an issue though.)

> > Open questions
> > ==============
> >
> > Now that I've explained (in length…) how it works, I think it's high
> > time we define what our target is.
> >
> > Due to the volatility of a backports suite, I would tend to not even try
> > to support netboot.
>
> Agreed.
>
> >                     Aiming for a least a netinst image would look good
> > to me. If we can manage that, we can probably also produce a CD1 for
> > those who want to have more packages than just what's on a netinst (I've
> > had at least one such request). I'm not going to debate how many
> > desktops we should have CD1 for, that's really not my call. I'll just
> > mention that size constraints might be tricky since we'll have both the
> > linux-image packages for the base suite and those from the backports
> > suite. The whole DVD/BR thing is probably entirely overkill.
>
> As I understand it, CD#1 is already fairly useless for non-networked
> (or limited network) installations and DVD#1 is a lot more useful.
OK, I'll leave that up to Steve and the images team anyway.

> > => Choice to make: netinst and maybe CD1(s)? Something else?
> >
> > It's probably a good idea to consider building matching unofficial
> > image(s) with firmwares embedded. I'm not an expert regarding debian-cd
> > so help is much welcome. Tweaks might be needed to get firmwares
> > installed and/or upgraded from the backports repository, be it to embed
> > them on the image, or to make them available to the installed system.
>
> Yes, firmware would need to come from backports.
>
> Since there is no dependency relation between the kernel and firmware
> package, I don't prune old files until they are no longer referenced by
> the kernel version in any supported suite.  So the CD image build would
> only need to use firmware packages from backports, not two different
> versions.
OK, thanks for mentioning this.

I'm not sure about the code path(s) responsible for enabling firmwares
on the installed system based on what was used/available on the
installation image. I hope Steve knows more about that, and will help me
figure out whether some more modifications are needed in some udebs to
make sure we have the right firmware packages/versions installed.

> [...]
> > Last question: how often do we produce such an image? I don't think it's
> > reasonable to do that every time a new major version of linux is made
> > available in the backports repository. Doing that once around the middle
> > of the release cycle would look good to me. This would kind of mimic the
> > “and a half” thing we had in the past. Once we get the process right, we
> > might think about doing so another time or two, but we already have limited
> > humanpower to deal with stable point releases and alphas for testing
> > (at least oldstable is gone now)…
>
> Doing this even once per release is a nice improvement, but I think
> it's not enough to solve the problem of unsupported hardware (even on
> x86).  And I think the more often this is done, the easier it will get.
> Still, I realise that you are likely to end up doing most of the work,
> and only you can decide how much time to spend on it.
Thanks for sharing this as well. I don't spend enough time on hardware
related topics so I didn't realize how much of an issue that can be.

Given how few changes were needed when I forward-ported my patch series,
it might indeed be rather easy to get more builds with newer kernels.

If we manage to get the debian-cd bits correctly, this might even be
very low maintenance.

But for now I'll concentrate on getting a single such release out, just
to make it official we actually did it.

> > => Question: what to advertise/communicate on? One-shot only is probably
> >    a good rule of thumb?
> [...]
>
> I think what you're proposing is to announce this as a one-off, and not
> to immediately make any promises about regular builds.  If so, I agree.

Thanks for bearing with my poor choice of words. That's exactly what I
meant.


Cheers,
--
Cyril Brulebois ([hidden email])            <https://debamax.com/>
D-I release manager -- Release team member -- Freelance Consultant

signature.asc (849 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Status and open questions for debian-installer with backports support

Steve McIntyre
In reply to this post by Cyril Brulebois-4
Apologies for taking a while to get back to you here... :-/

Also adding a CC to the -kernel list.

On Thu, Aug 02, 2018 at 10:56:59AM +0200, Cyril Brulebois wrote:
>
>This is a long read… I've highlighted “to do”-like items with an arrow
>(=>).

ACK!

>debian-installer
>================
>
>First things first, we needed a debian-installer build that knew about a
>backported linux kernel, so I hacked my way in debian-installer.git; it
>wasn't too hard since we can count on apt to resolve dependencies, and
>after all, we only needed the kernel-image udeb and its modules.
>
>The latest branch is stretch-backports-v2, which combines both a number
>of commits that are aimed at the master branch, and a couple others that
>want to only be used in a stretch-backports branch:
>
>  https://salsa.debian.org/installer-team/debian-installer/commits/stretch-backports-v2

All looks sane enough...

>net-retriever
>-------------
>
>As usual, I've tested the resulting installer with the netboot(-gtk)
>image and of course it failed to load additional udebs. These netboot*
>images are aimed at booting from the network, and they also need to
>fetch additional udebs from a mirror. Since net-retriever only knows
>about a single source of udebs, it needed to be told about multiple
>sources (stretch and stretch-backports) and also how to merge entries.
>I've detailed my choices in this commit in particular:
>
>  https://salsa.debian.org/installer-team/net-retriever/commit/09535b0613a85e02d511b92b60683b20405b6d42
>
>For more details, see the whole stretch-backports-v2 branch for
>net-retriever:
>
>  https://salsa.debian.org/installer-team/net-retriever/commits/stretch-backports-v2
>
>Now, it doesn't matter too much, since the stretch-backports repository
>will evolve over time, and udebs for the specific kernel used during the
>d-i build will disappear. I think this is too much of a volatile (no
>pun intended) target to support. It's still a very nice way to test d-i
>without having to build an ISO with debian-cd, so I'd like to merge
>these changes in net-retriever anyway.

ACK. Agreed with the "inefficient" comments here, but if it works...

>base-installer
>--------------
>
>Second, it makes quite some sense to not only run d-i with a newer
>kernel, but to also install it, so that users can boot from their brand
>new hardware that wasn't supported by d-i with stable components only.

Of course, yeah.

>The kernel selection and installation is implemented in base-installer
>(see library.sh), and that happens to run before apt-setup. Since I
>didn't want to butcher sources.list manually to install the kernel
>directly at this stage, I've decided to implement installing the latest
>kernel from backports in a finish-install script (also shipped in the
>base-installer package, for consistency), running at the very end of the
>installation process.

Ah, OK. More on this later.

>This only happens when backports support was enabled (this is detected
>through the presence of /etc/udebs-backports-source).
>
>At first, using the linux-image-$arch metapackage looked easy enough but
>of course that works for amd64, but not for i386 (one would need to pick
>686 or 686-pae). I've still pushed a branch with this implementation as
>it can be useful as is:
>
>  https://salsa.debian.org/installer-team/base-installer/commits/stretch-backports-v0
>
>We could probably modify library.sh to reuse it, but I'd like to have
>minimal changes (for reasons explained in the very last section). We
>could probably just iterate over all installed linux-image-'*' packages,
>pick the one from the src:linux package, and upgrade it from the
>backports repository.
>
>=> I'll look into that later on.

OK.

>apt-setup
>---------
>
>Also only when /etc/udebs-backports-source is present, the backports
>service gets automatically enabled, which makes the later installation
>of linux-image packages possible.
>
>Single commit for this:
>
>  https://salsa.debian.org/installer-team/apt-setup/commit/6133bdcd5a7903105ad967ab087f2f12f9d1c59d

Absolutely, yeah.

>debian-cd
>=========
>
>As mentioned above, tricks were needed in net-retriever for the netboot*
>images, to make it possible to load linux kernel modules udebs. Those
>tricks aren't needed when generating an ISO image. As a reminder, on the
>d-i side, a cdrom_isolinux image is generated, and the debian-cd tool
>needs to be configured to use that image instead of the official image
>available on the configured mirror.
>
>I didn't make this dynamic yet, but I've modified my local configuration
>to set BACKPORTS=backports-list, to include a number of packages from
>the stretch-backports suite. This file was generating by listing all
>udebs produced by the linux source package (as done in the d-i build
>system):
>
>    USE_UDEBS_BACKPORTS_FROM=stretch-backports
>    ...
>    source=linux
>    ...
>    binaries=$(grep-dctrl -s Package -F Source $source $APTDIR/state/lists/*${USE_UDEBS_BACKPORTS_FROM}*Packages | awk '{print $2}')

Right.

>=> This needs to become dynamic, probably through a helper tool that
>   can be called automatically if backports are enabled in debian-cd.

Nod, we can work that out.

>=> It might be nice to have some kind of consistency check, at least
>   to make sure the ABI is the same for the linux kernel produced by
>   the d-i build, and for the modules available in the backports
>   suite. [Note: this might be true for weekly builds too, see recent
>   complaints/bug reports.] Bonus points if the source version is
>   checked too, for extra caution.

Also agreeing with Ben, we need this checked to be right.

>As a consequence, all those were included in the ISO, in the following
>file:
>
>    debian/dists/stretch/main/debian-installer/binary-amd64/Packages.gz
>
>This means no extra tweak is needed there, all udebs are available in a
>single place: easy!

Nod. That was what I was aiming for in the debian-cd code, as it was
obvious anna wouldn't cope otherwise.

>Of course we still need the modified apt-setup and base-installer
>binaries. I've made them available as local packages (LOCALDEBS) and
>they were automatically used.

OK.

>The finish-install script ran as expected, but I ended up with a 4.16
>kernel from backports in the installed system, as the linux-latest
>binaries still point to it rather than to 4.17 (presumably because
>linux/stretch-backports is still Needs-Build on mipsel at the moment).
>It was downloaded from the network mirror.
>
>=> It would be nice if we could include the linux-image-* packages from
>   backports (and their dependencies) directly on the installation image,
>   so as to be independent of what's happening in the stretch-backports
>   suite on online mirrors. Users would have reproducible results over
>   time, instead of a jumpy target. I don't know debian-cd enough to
>   determine how feasible it would be. I'm also not sure what would happen
>   if we had “old” linux-image packages on the ISO, and “newer” ones on
>   mirrors.

Listing the linux-image-* package in the BACKPORTS file should
accomplish this already. The code in check_backports_packages is
designed to work this way, anyway - pulling in the specified packages
from backports, and any specific dependencies that are newer than
what's in stable. But it's been nearly 2 years since I wrote this
code. :-)

>Open questions
>==============
>
>Now that I've explained (in length…) how it works, I think it's high
>time we define what our target is.
>
>Due to the volatility of a backports suite, I would tend to not even try
>to support netboot.

Holy crap, no! :-)

>Aiming for a least a netinst image would look good to me. If we can
>manage that, we can probably also produce a CD1 for those who want to
>have more packages than just what's on a netinst (I've had at least
>one such request). I'm not going to debate how many desktops we
>should have CD1 for, that's really not my call. I'll just mention
>that size constraints might be tricky since we'll have both the
>linux-image packages for the base suite and those from the backports
>suite. The whole DVD/BR thing is probably entirely overkill.

Agreed. CD#1 is basically dead, as Ben commented. I also have no
desire to support a full range of media here. I'm thinking a netinst
and a DVD#1, and *that's all*. If you want more packages, use the
normal set in addition if you don't have network. To be honest, I'd
alwo want to put warnings out saying "ONLY use these if you're going
to be updating regularly!". Being stuck on an old -backport kernel or
X that's not getting security updates is not a great service for
users.

>=> Choice to make: netinst and maybe CD1(s)? Something else?

netinst and DVD#1.

>It's probably a good idea to consider building matching unofficial
>image(s) with firmwares embedded. I'm not an expert regarding debian-cd
>so help is much welcome. Tweaks might be needed to get firmwares
>installed and/or upgraded from the backports repository, be it to embed
>them on the image, or to make them available to the installed system.

Ah, good catch. The code to force firmware inclusion will also need an
update to also use backports firmware when a backports kernel is
included. That should be really easy. In terms of config, we're
already building netinst and DVD#1 versions with firmware. We'll keep
that same config.

>=> Choice: support unofficial firmware-enabled image(s)?

Yes, people will need them.

>=> Question: how to ensure firmwares from backports are available in d-i
>   and in the installed system?
>
>Finally, what architectures are we supporting? I can't speak for
>debian-cd, but at least for the d-i part, we'll likely fork master into
>stretch-backports and merge there. So we should keep track of any arch-
>specific changes and debian-installer is likely to be buildable/built on
>all architectures for no extra cost. If there are other bits that users
>need to rely on, like flash-kernel and other components, I think they
>should just consider installing testing instead. We can't reasonably be
>expected to backport all the things.

Sure.

>=> Question: should we restrict architectures we build images for?

Probably. I'm only thinking x86, arm64 and maybe armhf. Wait for
people to ask for others and add on a case-by-case basis.

>Last question: how often do we produce such an image? I don't think it's
>reasonable to do that every time a new major version of linux is made
>available in the backports repository. Doing that once around the middle
>of the release cycle would look good to me. This would kind of mimic the
>“and a half” thing we had in the past. Once we get the process right, we
>might think about doing so another time or two, but we already have limited
>humanpower to deal with stable point releases and alphas for testing
>(at least oldstable is gone now)…
>
>=> Question: what to advertise/communicate on? One-shot only is probably
>   a good rule of thumb?

Let's try to do one first, and then work out what to do if it
succeeds. :-)

In the long run, I'm thinking I'd like to do a smallish number in the
lifetime of a stable release. Maybe (roughly!) following the point
release schedule? Every 2-3 months?

>What to do with modified components
>===================================
>
>My current approach would be to get net-retriever, apt-setup, and
>base-installer (once this last one is ready of course) updated in
>stable. This would greatly help building, testing, and running
>debian-installer without having to special-case everything in various
>places. Of course this is subject to a green light from the release
>team, but I might be trusted to get things right (and/or to get the
>pieces working again if anything breaks).

Nod.

>We would be able to rely on their having built-in support for backports
>(remember, it's only enabled when a specific file is present). We could
>then just upload src:debian-installer to stretch-backports (after dak's
>been patched to allow that), and point debian-cd to the resulting
>images.

Sounds cool. I'll probably need to tweak on my end for that to work,
but it should be straighforward.

In terms of *my* code changes for debian-cd, all the changes so far
are already in stable (not that it matters all that much). So long as
the changes are not too intrusive, I'll also be taking any changes
onto the buildd/stretch branch too.

--
Steve McIntyre, Cambridge, UK.                                [hidden email]
"We're the technical experts.  We were hired so that management could
 ignore our recommendations and tell us how to do our jobs."  -- Mike Andrews

Reply | Threaded
Open this post in threaded view
|

Re: Status and open questions for debian-installer with backports support

John Paul Adrian Glaubitz
In reply to this post by Cyril Brulebois-4
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

On 08/02/2018 10:56 AM, Cyril Brulebois wrote:
> It's been a while since my first proof of concept (demonstrated at the 2014 mini-DebConf in Paris), but I think I've managed to reach a point where I'm
> rather content and ready to get bits and pieces merged where they belong. That doesn't mean I have everything figured out, that's why I'm reaching out to
> both the installer & images team!

Can we also add the patches for choose-mirror and net-retriever to support
Debian Ports at some point? We have lots of users that are confused that
debian-installer does not provide a mirror list from where they can choose
but rather have to type in the mirror information manually.

> https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=879145 https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=879130

There is also:

> https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=879147 https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=879151

Thanks,
Adrian

- --
 .''`.  John Paul Adrian Glaubitz
: :' :  Debian Developer - [hidden email]
`. `'   Freie Universitaet Berlin - [hidden email]
  `-    GPG: 62FF 8A75 84E0 2956 9546  0006 7426 3B37 F5B5 F913
-----BEGIN PGP SIGNATURE-----

iQIzBAEBCAAdFiEEYv+KdYTgKVaVRgAGdCY7N/W1+RMFAltxXXoACgkQdCY7N/W1
+ROgbg//dxJC1qX7Jfa3qzbqnjQ7pUh68/1OwEF6xwQj4NrkFgDchVVTASvzEwul
goKx+tELK3LVfy+ncchRrD2vImT1aY4iBpywCLSi/T9bE+z/15SOEZlb1QZw2bxp
1vw4HW13HzOBrR7TK+lu7QeAcWMKanWMO+T53IT47/NeQXhbtyCeJ1AP3+mPwqjR
GmR4rsnFX/kjYfIXDLEI9ByMbd3NQ/IQNEbibxxIYiHrW3Tat4xBfw0vwMplySdt
PDo11uqrrbbuhBXTVnxEilqC2Td+Yr9uglEZ6hJ4+RPDBLcCInnLgw5GCvfQq80v
6igQIQp+nJkZPohFawffX6grEzmxvuOSUZSfuyjAioKuBWevBggV2W4/LiplALA1
pUTvYW6RvDjtaCfpVrCH1uUcQMCvZorwl35R9UXRxKHwcCq/nWHNi5ym0g7bfdKK
gX8LkdjtnYiT+7pJDK3tw1dCMzYsys9BUmBprkjCtfvHkfQSvyJKVng+xOwCvfX5
KKj9ohCYjTBcpe9cHbu0ZlvR8moW7OlpWcS5HyKK1PR37BNWfPjV+O6KX+ovUnIT
cUjyGZMYoxfeusr/MY9OF02iSLaogtGVFJySlAppJoJ4ERCbeQNDuqZ71zWMD1QM
rCR3qlzbq1umVxjQE7nlTm2eTvOT92R7mR2ijKDz4kVMFreX/rM=
=8M/T
-----END PGP SIGNATURE-----

Reply | Threaded
Open this post in threaded view
|

Re: Status and open questions for debian-installer with backports support

Cyril Brulebois-4
John Paul Adrian Glaubitz <[hidden email]> (2018-08-13):

> Can we also add the patches for choose-mirror and net-retriever to support
> Debian Ports at some point? We have lots of users that are confused that
> debian-installer does not provide a mirror list from where they can choose
> but rather have to type in the mirror information manually.
>
> > https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=879145 https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=879130
>
> There is also:
>
> > https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=879147 https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=879151
I'm aware of these issues, but it seems to make sense to concentrate
on release architectures, which have a huge user base, and which need
improved hardware support.

There are enough moving parts as it is to avoid adding ports-specific
issues to the mix (as already mentioned in my reply to one of those
bug reports). Please let us figure out how to deal with release archs
and backports first.


Cheers,
--
Cyril Brulebois ([hidden email])            <https://debamax.com/>
D-I release manager -- Release team member -- Freelance Consultant

signature.asc (849 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Status and open questions for debian-installer with backports support

Julien Cristau-6
In reply to this post by Steve McIntyre
On 08/13/2018 01:57 AM, Steve McIntyre wrote:
> On Thu, Aug 02, 2018 at 10:56:59AM +0200, Cyril Brulebois wrote:
>> => Question: should we restrict architectures we build images for?
>
> Probably. I'm only thinking x86, arm64 and maybe armhf. Wait for
> people to ask for others and add on a case-by-case basis.
>
I'd even suggest s/x86/amd64/

New 32-bit x86 hardware shouldn't be a huge use case nowadays.

Cheers,
Julien

Reply | Threaded
Open this post in threaded view
|

Re: Status and open questions for debian-installer with backports support

Steve McIntyre
On Mon, Aug 13, 2018 at 01:44:48PM +0200, Julien Cristau wrote:

>On 08/13/2018 01:57 AM, Steve McIntyre wrote:
>> On Thu, Aug 02, 2018 at 10:56:59AM +0200, Cyril Brulebois wrote:
>>> => Question: should we restrict architectures we build images for?
>>
>> Probably. I'm only thinking x86, arm64 and maybe armhf. Wait for
>> people to ask for others and add on a case-by-case basis.
>>
>I'd even suggest s/x86/amd64/
>
>New 32-bit x86 hardware shouldn't be a huge use case nowadays.

That's a fair point, yes.

--
Steve McIntyre, Cambridge, UK.                                [hidden email]
"You can't barbecue lettuce!" -- Ellie Crane