The noudeb build profile and dh-only rules files

classic Classic list List threaded Threaded
20 messages Options
Reply | Threaded
Open this post in threaded view
|

The noudeb build profile and dh-only rules files

Theodore Y. Ts'o
I'm the middle of an effort to simplify the debian/rules file for
e2fsprogs so that someday, maybe, I'll be able to convert it to use
dh.  One of the things which I noticed while trying to rip things out
of debian/rules to make the dh conversion easier (possible?) was the
support for noudeb.

How important is noudeb, and why is defined in the first place?

And will debhelper do the the right thing in terms of filtering out
udeb packages if noudeb is psecified in DEB_BUILD_PROFILES?  If
support of noudeb is free in the brave new dh-only world, that's
great.

If it has to be hacked in manually, how important is it to support
noudeb?

I've tried to do some web searches to answer the above questions, but
I'm afraid my Google-fu has failed me.  So maybe this is something
that can be documented in a few places, including the debhelper man
pages, and perhaps, in the BuildProfileSpec Debian Wiki page?

Many thanks!!

                                                - Ted

Reply | Threaded
Open this post in threaded view
|

Re: The noudeb build profile and dh-only rules files

Samuel Thibault-8
Hello,

Theodore Ts'o, le lun. 08 juil. 2019 13:25:32 -0400, a ecrit:
> How important is noudeb, and why is defined in the first place?

My usage of noudeb is mostly to avoid the two-times-longer build time

Samuel

Reply | Threaded
Open this post in threaded view
|

Re: The noudeb build profile and dh-only rules files

Theodore Y. Ts'o
On Mon, Jul 08, 2019 at 07:36:30PM +0200, Samuel Thibault wrote:
> Hello,
>
> Theodore Ts'o, le lun. 08 juil. 2019 13:25:32 -0400, a ecrit:
> > How important is noudeb, and why is defined in the first place?
>
> My usage of noudeb is mostly to avoid the two-times-longer build time
>

It used to be that I built e2fsprogs twice; once for udebs, and once
for the "normal" build.  I'm planning on ripping that out as being
complexity that seems incredibly painful to convert to using dh, and
the cost seems to be growing the installed size of e2fsprogs-udeb by
145k (or roughly 15%).

Back in the days of boot/root installation floppies, saving every last
byte was clearly important.  My plan is to drop it to save developer
maintenance headache, and it also avoids the double-compilation build
time extension.  (I assume that's what you were referring to when you
mentioned "avoid the two-times-longer build time", right?)

                          - Ted

P.S.  If anyone thinks that increasing the size of the debian
installer by 145k is unacceptable, please let me know now....

Reply | Threaded
Open this post in threaded view
|

Re: The noudeb build profile and dh-only rules files

Colin Watson
In reply to this post by Theodore Y. Ts'o
On Mon, Jul 08, 2019 at 01:25:32PM -0400, Theodore Ts'o wrote:
> I'm the middle of an effort to simplify the debian/rules file for
> e2fsprogs so that someday, maybe, I'll be able to convert it to use
> dh.  One of the things which I noticed while trying to rip things out
> of debian/rules to make the dh conversion easier (possible?) was the
> support for noudeb.
>
> How important is noudeb, and why is defined in the first place?

I'm afraid memory has failed me in terms of why it was defined, other
than perhaps performance for people doing rapid iteration.

> And will debhelper do the the right thing in terms of filtering out
> udeb packages if noudeb is psecified in DEB_BUILD_PROFILES?  If
> support of noudeb is free in the brave new dh-only world, that's
> great.

If the udeb stanzas in debian/control have "Build-Profiles: <!noudeb>",
then debhelper will honour that when deciding which packages to build,
so yes, anything built into debhelper should just work.

Even after simplification, you may well have remaining code in
debian/rules that refers to udebs, or an extra udeb build pass, or
something else that dh wouldn't be able to handle by itself.  In such
cases, you can wrap that code in a dh_listpackages check of the kind
described in
https://wiki.debian.org/BuildProfileSpec#The_Build-Profiles_field.

HTH,

--
Colin Watson                                       [[hidden email]]

Reply | Threaded
Open this post in threaded view
|

Re: The noudeb build profile and dh-only rules files

Colin Watson
In reply to this post by Theodore Y. Ts'o
On Mon, Jul 08, 2019 at 02:10:13PM -0400, Theodore Ts'o wrote:

> On Mon, Jul 08, 2019 at 07:36:30PM +0200, Samuel Thibault wrote:
> > Theodore Ts'o, le lun. 08 juil. 2019 13:25:32 -0400, a ecrit:
> > > How important is noudeb, and why is defined in the first place?
> >
> > My usage of noudeb is mostly to avoid the two-times-longer build time
>
> It used to be that I built e2fsprogs twice; once for udebs, and once
> for the "normal" build.  I'm planning on ripping that out as being
> complexity that seems incredibly painful to convert to using dh, and
> the cost seems to be growing the installed size of e2fsprogs-udeb by
> 145k (or roughly 15%).

Per my other reply, you may find that it isn't that painful after all
once you find the right approach.  For instance, while a separate udeb
build pass does make
https://salsa.debian.org/ssh-team/openssh/blob/master/debian/rules a
little bit more complicated, I wouldn't say it's something I find myself
having to think about very often.

> P.S.  If anyone thinks that increasing the size of the debian
> installer by 145k is unacceptable, please let me know now....

This is something you'd need to run past debian-boot@, as there may well
be particular images for which it still matters (I haven't kept up, but
it's certainly something they'd want to be informed of).

--
Colin Watson                                       [[hidden email]]

Reply | Threaded
Open this post in threaded view
|

Re: The noudeb build profile and dh-only rules files

Colin Watson
On Mon, Jul 08, 2019 at 07:28:50PM +0100, Colin Watson wrote:
> On Mon, Jul 08, 2019 at 02:10:13PM -0400, Theodore Ts'o wrote:
> > P.S.  If anyone thinks that increasing the size of the debian
> > installer by 145k is unacceptable, please let me know now....
>
> This is something you'd need to run past debian-boot@, as there may well
> be particular images for which it still matters (I haven't kept up, but
> it's certainly something they'd want to be informed of).

Oh, and apologies if this is obvious, but the other reason a separate
udeb build pass may be necessary is if certain configure options make
the code actually not work in the context of d-i.  This is the case for
openssh (for example, it builds an sshd that can be used as part of
d-i's network-console feature, but PAM wouldn't work in that context).
I don't know whether it's the case for e2fsprogs.

--
Colin Watson                                       [[hidden email]]

Reply | Threaded
Open this post in threaded view
|

Re: The noudeb build profile and dh-only rules files

Moritz Mühlenhoff-2
In reply to this post by Theodore Y. Ts'o
Theodore Ts'o <[hidden email]> schrieb:
> Back in the days of boot/root installation floppies, saving every last
> byte was clearly important.

It's probably worth discussing/investigating whether udebs in general still
make sense for d-i in 2019?

It was a design choice made 15 years ago, but disk/network constraints
changed a lot since then. Maybe ditching them entirely would actually
reduce a lot of toil in d-i and make d-i development more flexible?
(Honest question, not trying to insinuate anything)

Cheers,
        Moritz

Reply | Threaded
Open this post in threaded view
|

Re: The noudeb build profile and dh-only rules files

Samuel Thibault-8
In reply to this post by Theodore Y. Ts'o
Theodore Ts'o, le lun. 08 juil. 2019 14:10:13 -0400, a ecrit:
> and it also avoids the double-compilation build time extension.  (I
> assume that's what you were referring to when you mentioned "avoid the
> two-times-longer build time", right?)

Yes.

Samuel

Reply | Threaded
Open this post in threaded view
|

Re: The noudeb build profile and dh-only rules files

Simon McVittie-7
In reply to this post by Colin Watson
On Mon, 08 Jul 2019 at 19:23:39 +0100, Colin Watson wrote:
> On Mon, Jul 08, 2019 at 01:25:32PM -0400, Theodore Ts'o wrote:
> > How important is noudeb, and why is defined in the first place?
>
> I'm afraid memory has failed me in terms of why it was defined, other
> than perhaps performance for people doing rapid iteration.

I think I was one of the originators of noudeb, and more rapid iteration
was certainly my main motivation. In packages like dbus and glib2.0,
noudeb can avoid doing a full second build (or even a third build in
the case of dbus) when doing test-builds, and the udebs from a test-build
aren't usually going to be tested or used anyway.

If widely adopted it's also a convenient way to speed up multiple
packages' builds by skipping udebs globally, in distributions whose
maintainers know that they aren't going to use debian-installer at all,
like some of the Debian derivatives I've worked on in my job.

If your package doesn't do an extra build for udebs, or just has short
compile times in general, then noudeb will have little impact on either
of those goals and is unimportant.

dbus definitely has to do a separate build pass for udebs, because the
"full-fat" version has dependencies that don't exist as udebs:
libdbus -> libsystemd, and dbus -> libaudit/libapparmor. (Having said
that, dbus' udebs are only there for the benefit of AT-SPI, but according
to #929132 AT-SPI in the graphical installer hasn't been implemented in
the 5+ years since it was requested, so maybe they can be dropped.)

I thought glib2.0 was in the same situation, but libmount-udeb does
exist, so it might actually be possible to stop doing the secondary build
for glib2.0 in bullseye. glib2.0 is used in the graphical installer,
where GTK, fonts and graphics probably swamp the size saving from doing
a reduced-functionality glib2.0 build.

> If the udeb stanzas in debian/control have "Build-Profiles: <!noudeb>",
> then debhelper will honour that when deciding which packages to build,
> so yes, anything built into debhelper should just work.

Treating udebs as implicitly having Build-Profiles: <!noudeb> might be a
reasonable feature request for debhelper, but I don't think it implements
that right now. Like everything else in udebs, it might be too niche to
be considered a worthwhile investment of effort.

> Even after simplification, you may well have remaining code in
> debian/rules that refers to udebs, or an extra udeb build pass, or
> something else that dh wouldn't be able to handle by itself.  In such
> cases, you can wrap that code in a dh_listpackages check of the kind
> described in
> https://wiki.debian.org/BuildProfileSpec#The_Build-Profiles_field.

You might find that dbus and glib2.0 are useful examples for this.

    smcv

Reply | Threaded
Open this post in threaded view
|

Re: The noudeb build profile and dh-only rules files

Colin Watson
In reply to this post by Moritz Mühlenhoff-2
On Mon, Jul 08, 2019 at 10:04:06PM +0200, Moritz Mühlenhoff wrote:

> Theodore Ts'o <[hidden email]> schrieb:
> > Back in the days of boot/root installation floppies, saving every last
> > byte was clearly important.
>
> It's probably worth discussing/investigating whether udebs in general still
> make sense for d-i in 2019?
>
> It was a design choice made 15 years ago, but disk/network constraints
> changed a lot since then. Maybe ditching them entirely would actually
> reduce a lot of toil in d-i and make d-i development more flexible?
> (Honest question, not trying to insinuate anything)

I think there are a lot of things one could plausibly say about good
directions for the Debian installer.  "Move to debs" isn't the one I'd
pick, although if somebody thinks it's useful they could certainly try
working out the details.  I would say that udebs themselves don't
particularly cause much toil in d-i, although they do cause some for
maintainers of non-installer packages that are reused by d-i (as shown
in this thread).

Note that udebs don't exist solely because of disk constraints, much
less network constraints (which were never really a big consideration):
another reason for the separate package type was so that there was no
plausible way they could accidentally be installed on "real" systems,
and so a good argument could be made that they didn't need to comply
with all of Debian policy.  For the ones that are merely minor
variations of existing runtime library packages, the difference is less
obvious, but it's more striking when you look at the udebs that form
part of the installer itself.  So IMO any attempt to drop udebs and move
d-i to debs needs a lot of thought about the knock-on effects.


(Some more stream-of-consciousness stuff follows.  Credentials: I have a
solid decade or so of contributions to d-i and either wrote or heavily
hacked on quite a few of its core components, although I haven't been
particularly active recently: so any express or implied criticism should
be read as being directed at least as much at myself as anyone else.
But in any case I'm not trying to apportion blame.  I'm taking some
rhetorical liberties when I say "we designed d-i" etc., since I only got
involved in 2004 and wasn't around in the earliest stages.)


In 2003ish, I think setting the goal that d-i should reuse as much as
possible from Debian made a lot of sense.  boot-floppies suffered really
badly from being more or less a monolith that was off on its own
relative to the rest of Debian, and you can see the reaction to that
very clearly in d-i's design: it's broken down into many small
components which can be maintained and uploaded (and migrated to
testing, for that matter) independently, and reuses both code and
concepts from elsewhere in Debian.  Once it got past a tipping point of
basic usability in late 2003 / early 2004, it became very accessible for
Debian developers to come in and hack on.  I have fond memories of time
spent in various in-person d-i meetings that routinely had something
like a dozen active hackers: maybe this just reflects my interests, but
it's by far the largest group I've worked with on any single project in
Debian before or since.

If we'd been doing the same thing 10-15 years later, I think we'd have
taken different decisions:

 * There might have been other interesting axes of reuse, as well as or
   instead of prioritising reusing other parts of Debian: just as we
   might have decided to reuse Anaconda back in the day, nowadays I hear
   good things about Calamares.  There've been benefits to being
   Debian-specific, but there've also been costs.

 * I think/hope that we collectively understand software engineering
   rather better than we did then, and we would certainly have designed
   something that was more amenable to unit testing.  (Some parts of d-i
   do have localised unit tests, but some of the most difficult
   components are really very challenging to test outside an installer
   environment, and mostly that isn't a property intrinsic to what
   they're doing but rather to their implementation choices.)

 * Size constraints and such are indeed not what they were 15 years ago,
   although there's still some value in trying to be lean.  The thing I
   would take from this is not to abandon udebs, though, since despite
   the name I don't think size is their only important property; rather,
   if we were designing d-i today I think we might very well make
   different language choices.  We've pushed shell a long way, but it
   really isn't a particularly suitable language.

 * Things like cloud instances are a big deal now when they really
   weren't 15 years ago.  In many cases these are deployed using
   pre-built images and an installer doesn't really get involved, but
   there are still more situations where installer performance matters
   now than when we were designing d-i.

You could take the lesson from this that we should ditch d-i and move to
another system.  (Indeed, Joey wrote
https://joeyh.name/blog/entry/propellor_is_d-i_2.0/ a while back which
is arguably saying pretty much that, or people are working on Calamares,
or there's the Ubuntu effort to use subiquity, etc.)  On the other hand,
unlike the transition away from boot-floppies, we now have a widely-used
system with a very flexible customisation mechanism (preseeding), and
I'd be concerned that Debian might systematically undervalue its users'
time if it chose to move away from d-i.  We are where we are.  It
doesn't hurt us that other installers exist, but I'd like d-i to stay
healthy and improve.  So here are some very biased ways I think we could
systematically improve d-i in the next decade:

 * Incrementally rethink some implementation choices.

   Our policy for a long time has been that we only use busybox sh and
   C, and there were good reasons for that at the time, but it's also
   2019 now and maybe other choices are possible without doing
   unreasonable things to the sizes of those images that are still
   constrained.

   A memory-safe language with good testing support and a good testing
   culture would be great, though it does also need to work on every
   Debian architecture, which IIRC Rust doesn't quite; we've kicked
   around the idea of maybe a stripped-down Python in the past, which
   would preserve some useful live-hackability properties while being
   much more capable.  Only the parts of the installer up to the
   retriever are really tightly size-constrained, and lots of
   interesting things like the partitioner come after that point.
   Anything like this could be done piecewise, but I think we would want
   to pick basically one extra language and stick with it, so it'd need
   to be worthwhile.

 * Targeted rewrites of components with overwhelming technical debt.

   I proposed an os-prober rewrite a while ago
   (https://lists.debian.org/debian-boot/2017/01/msg00245.html), but
   haven't got very far with it; maybe this is the sort of size that
   could be tackled as a GSoC project?

   partman is also extraordinarily delicate.  I think there are probably
   fewer than half a dozen people on the planet who properly understand
   it, and it could really do with (at least) a robust test suite and
   the ability for recipes to be nested in ways that correspond roughly
   to the various ways in which block devices can be nested, rather than
   the current extremely ad-hoc arrangements for dealing with things
   like RAID and encryption.  Since it runs after the retriever, it can
   probably afford to use a better implementation language, although it
   is itself divided into many components so any rewrite would involve a
   lot of thought.

   Anything like this should have solid unit tests as a mandatory part
   of its development.

 * Look into reducing the burden on non-installer maintainers.

   This sort of thread comes up because Debian developers who don't
   really work on the installer but whose software happens to be part of
   the installer need to carry code to support it.  Maybe there are ways
   to relax that and at the same time avoid some of the problems we
   sometimes run into where d-i ends up blocked on non-installer
   packages, without having to abandon the useful properties of udebs
   entirely.  For example, we could allow d-i to be more flexible about
   pulling in parts of debs at build time, or we could work out how to
   mix some (maybe tagged?) debs and udebs in the installer environment,
   or something like that.  Not sure, and it wouldn't be
   straightforward, but it's worth a look.

 * Performance.

   Migrating hot spots away from big piles of shell would help, but I'm
   sure there are other bits of low-hanging fruit.  I profiled partman
   some years ago and was able to find serious improvements in its
   partition update and display code without needing to do anything more
   complicated than a bit of caching and hoisting code out of inner
   loops.  There's surely more of the same.  Other approaches might
   involve doing more to ensure we're disabling fsync et al where
   appropriate, or installing the relatively fixed base system from an
   image rather than from debs.

 * Just ordinary maintenance.  Really.

   Cyril has been doing sterling work keeping things going, but I hope
   they won't be offended when I say that I have the impression that
   d-i's bus factor seems to be decreasing gradually; maybe it works
   well enough for most DDs that they don't need to be very involved,
   but it's not as sustainable as the days when we could easily get a
   dozen or more people on feature development.  If anything on this
   list sounds interesting to you, go and help out for a while first so
   that you get a feel for how things work!

--
Colin Watson                                       [[hidden email]]

Reply | Threaded
Open this post in threaded view
|

Re: The noudeb build profile and dh-only rules files

Michael Biebl-3
In reply to this post by Moritz Mühlenhoff-2
Am 08.07.19 um 22:04 schrieb Moritz Mühlenhoff:

> Theodore Ts'o <[hidden email]> schrieb:
>> Back in the days of boot/root installation floppies, saving every last
>> byte was clearly important.
>
> It's probably worth discussing/investigating whether udebs in general still
> make sense for d-i in 2019?
>
> It was a design choice made 15 years ago, but disk/network constraints
> changed a lot since then. Maybe ditching them entirely would actually
> reduce a lot of toil in d-i and make d-i development more flexible?
> (Honest question, not trying to insinuate anything)
As a DD being affected by the additional workload that is caused by
having to provide udebs I would very much welcome a move to abandon udebs.


--
Why is it that all of the instruments seeking intelligent life in the
universe are pointed away from Earth?


signature.asc (849 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: The noudeb build profile and dh-only rules files

Paul Wise via nm
In reply to this post by Colin Watson
On Tue, Jul 9, 2019 at 7:42 AM Colin Watson wrote:

>    A memory-safe language with good testing support and a good testing
>    culture would be great, though it does also need to work on every
>    Debian architecture, which IIRC Rust doesn't quite; we've kicked
>    around the idea of maybe a stripped-down Python in the past,

This reminded me of Keith Packard's "Python for embedded systems", Snek:

https://keithp.com/snek/

--
bye,
pabs

https://wiki.debian.org/PaulWise

Reply | Threaded
Open this post in threaded view
|

Re: The noudeb build profile and dh-only rules files

Theodore Y. Ts'o
In reply to this post by Colin Watson
On Mon, Jul 08, 2019 at 07:28:50PM +0100, Colin Watson wrote:
>
> Per my other reply, you may find that it isn't that painful after all
> once you find the right approach.  For instance, while a separate udeb
> build pass does make
> https://salsa.debian.org/ssh-team/openssh/blob/master/debian/rules a
> little bit more complicated, I wouldn't say it's something I find myself
> having to think about very often.

Thanks, that's really helpful.  One of the really frustrating things
I've found about trying to use dh is that there is a real lack of
examples which are more complicated than:

#!/usr/bin/make -f
#
# See?   dh is easy-peasy!

%:
        dh $@

Sure, there are few a examples using one or two override declarations,
but trying to use dh on a non-trivial package is.... non-obvious,
given the current documentation.  Some more advanced tutorials, or
some links to good examplars of how to use dh in "real world",
non-trivial packaging setups, would be really good.  It looks like
openssh is a good example, but I'm sure they must be others.


> > P.S.  If anyone thinks that increasing the size of the debian
> > installer by 145k is unacceptable, please let me know now....
>
> This is something you'd need to run past debian-boot@, as there may well
> be particular images for which it still matters (I haven't kept up, but
> it's certainly something they'd want to be informed of).

Ack, I'll do that.  I did try to do some research, and I think I saw a
netinstall image which was 48 MiB, but I didn't see anything smaller.
Of course, I haven't checked all architectures, so checking with
debian-boot would be good.

> Oh, and apologies if this is obvious, but the other reason a separate
> udeb build pass may be necessary is if certain configure options make
> the code actually not work in the context of d-i.  This is the case for
> openssh (for example, it builds an sshd that can be used as part of
> d-i's network-console feature, but PAM wouldn't work in that context).
> I don't know whether it's the case for e2fsprogs.

Yeah, it wasn't the case of e2fsprogs not working (I've always tried
to make e2fsprogs very self-contained with an absolute minimal number
of dependencies, since it has to work before /usr is mounted.)  It was
doing things like disabling gettext/NLS which was responsible for the
bulk of the 145k decrease in size, IIRC.

As I said, back in the days of installation floppies, it was clearly
worth it to do a second pass build just to save bytes.  But these
days?  Realistically, e2fsprogs-udeb has been growing in size as ext4
has become a more featureful file system; for example, between Debian
Jessie and Debian Buster, e2fsprogs-udeb has grown by 142k.  And back
in the ext2/ext3 days of installation floppies, before e2fsprogs
supported 64-bit block numbers, ext4 features like extents, inline
data, bigalloc, metadata checksums, etc., e2fsprogs-udeb was even
smaller.

                                                - Ted
                                                       

Reply | Threaded
Open this post in threaded view
|

Could we generate d/control instead of working with "assembly level code" directly (was: Re: The noudeb build profile and dh-only rules files)

Niels Thykier
In reply to this post by Simon McVittie-7
Simon McVittie:

> On Mon, 08 Jul 2019 at 19:23:39 +0100, Colin Watson wrote:
> [...]
>> If the udeb stanzas in debian/control have "Build-Profiles: <!noudeb>",
>> then debhelper will honour that when deciding which packages to build,
>> so yes, anything built into debhelper should just work.
>
> Treating udebs as implicitly having Build-Profiles: <!noudeb> might be a
> reasonable feature request for debhelper, but I don't think it implements
> that right now. Like everything else in udebs, it might be too niche to
> be considered a worthwhile investment of effort.
>
> [...]
I have been tempted to do something like this but have decided against
it.  The debhelper tools are not the only tools reading d/control, so
there is a limit to how many packaging papercuts we can work around in
debhelper (e.g. see the "related" issue for Rules-Requires-Root in
#884999).  This one "just might work with a bit of abuse", but the next
one will probably fall flat on its face (or, indeed, the previous
request did).


Instead, I have been toying with the idea of treating d/control as
something we generate. While not entirely novel in itself, once you
start generating d/control, you can do interesting rewrites such as:


Input file (fiction):
"""
Source: foo
...

Package: new-package
<regular fields here that the maintainer need>

Package: old-package
Replaced-By: new-package (= 1.0)

Package: new-udeb
Package-Type: udeb
"""


Generated output:
"""
Source: foo
...

Package: new-package
<regular fields here that the maintainer need>
Breaks: old-package (<< 1.0~)
Replaces: old-package (<< 1.0~)

Package: old-package
Priority: optional
Section: oldlibs
Architecture: all
Depends: new-package (>= 1.0~)
Description: transitional package - replaced by new-package
This package is for transitional purposes and can safely be removed.

Package: new-udeb
Package-Type: udeb
Build-Profiles: <!noudeb>
Section: debian-installer
Priority: optional
<Other fields if I forgot any in my hand-written unplanned demo>
"""


It is not perfect for packages that uses dpkg-control/dh_control with -v
to set a "per-binary" version, but I still think it would be a great
reduction in cognitive burden for the maintainer to work with a
d/control that is on a higher level than "assembly".

Unfortunately, the whole act of generating d/control introduces its own
set of papercuts/learning curve as most tools need a d/control or a .dsc
(which itself needs a d/control).

The alternative would be to have *all* tools working on d/control have a
shared interface on deciding the defaults.  But that becomes a mess as
because we probably have thousands of places to update in/around the
Debian infrastructure alone.


Comparison with debdry (which I assume someone would bring up in
response): debdry had the goal of relying on tools to generate the
d/control along with most other packaging files (e.g. d/rules rewriting
would also be in scope).  Most of its changes to the d/control syntax
revolved around merging manual changes with the automatic ones.  The
rewrites I propose above and debdry's approach can be combined/composed
(including by adding this feature directly to debdry and have it be the
reference implementation for these rewrites)

Thanks,
~Niels

Reply | Threaded
Open this post in threaded view
|

Re: Could we generate d/control instead of working with "assembly level code" directly (was: Re: The noudeb build profile and dh-only rules files)

Simon Richter
Hi,

On Tue, Jul 09, 2019 at 06:12:00AM +0000, Niels Thykier wrote:

> Instead, I have been toying with the idea of treating d/control as
> something we generate. While not entirely novel in itself, once you
> start generating d/control, you can do interesting rewrites such as:

I've started work in that direction with the debian-xcontrol package a few
years ago. The debian/control file is an API that is difficult to extend
because it lives at the boundary of an Internet service, so any change
basically requires writing adapter code that translates API versions, and
the control file is somewhat unversioned (Policy-Version exists, but is not
used as a switch).

Your proposal of generating some of the fields doesn't affect the API
itself, as long as the fields are populated at the right time. We don't
have a mechanism for updating the control file at build time, because any
part of the build process that would be able to do so is after the part
where the control file is consumed for the first time, so it would give an
inconsistent view.

The way it was generally done with xcontrol was to regenerate the control
file and compare whether the generated file was equivalent to the existing
one, and fail the build if not. This is a bit fragile, but it's the best we
can do here without changing Policy.

Your proposal effectively reinvents debian-xcontrol, which wasn't a roaring
success because it was more annoying to use than just writing a few lines
by hand. It's not a bad idea per se, but it will need more benefit to fly.

   Simon

Reply | Threaded
Open this post in threaded view
|

Re: Could we generate d/control instead of working with "assembly level code" directly (was: Re: The noudeb build profile and dh-only rules files)

Raphael Hertzog-3
On Tue, 09 Jul 2019, Simon Richter wrote:
> Your proposal of generating some of the fields doesn't affect the API
> itself, as long as the fields are populated at the right time. We don't
> have a mechanism for updating the control file at build time, because any
> part of the build process that would be able to do so is after the part
> where the control file is consumed for the first time, so it would give an
> inconsistent view.

On a somewhat related topic, I filed this wishlist bug a long time ago:
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=597340

The idea is to have implicit substvars at the end of every field so that
you can extend the content of the field, and possibly introduce fields
that are not present in the original source tree.

Cheers,
--
Raphaël Hertzog ◈ Debian Developer

Support Debian LTS: https://www.freexian.com/services/debian-lts.html
Learn to master Debian: https://debian-handbook.info/get/

Reply | Threaded
Open this post in threaded view
|

Re: Could we generate d/control instead of working with "assembly level code" directly (was: Re: The noudeb build profile and dh-only rules files)

Theodore Y. Ts'o
In reply to this post by Simon Richter
On Tue, Jul 09, 2019 at 12:24:40PM +0200, Simon Richter wrote:
> Your proposal of generating some of the fields doesn't affect the API
> itself, as long as the fields are populated at the right time. We don't
> have a mechanism for updating the control file at build time, because any
> part of the build process that would be able to do so is after the part
> where the control file is consumed for the first time, so it would give an
> inconsistent view.

I used to handle this back when I had the goal of making sure that
e2fsprogs from the git repository could successfully build as far back
as oldoldstable.  The idea was that sometimes people would want to be
able to get an updated version of e2fsprogs with all of the bug fixes;
and while I'm not willing to manually extract large number of bug
fixes and backport them to ancient distro versions of Debian and
Ubuntu --- our backport process to Debian Obsolete^H^H^H^H^H^H^H^H
Stable is *not* fun for me, as far as I'm concerned), I could at least
make sure that modern versions of e2fsprogs could be trivially
repackaged for ancient versions of Debian/Ubuntu.

The way I did this was to make a default target in debian/rules called
debian-files, which would create (or re-create) debian/control from a
debian/control.in file.  Then to build e2fsprogs on debian, one would
first unpack the e2fsprogs' upstream tarfile distribution, or check it
out from git, and then run:

./debian/rules
dpkg-buildpackage

The Debian source package would have the automagically generated
debian/control file, so it was fully compatible with all of Debian's
package tooling, but it would also have the debian/control.in file,
which as far as *I* was concerned was the preferred form for
modification.

Cheers,

                                                - Ted

Reply | Threaded
Open this post in threaded view
|

Re: The noudeb build profile and dh-only rules files

Ian Jackson-2
In reply to this post by Theodore Y. Ts'o
Theodore Ts'o writes ("Re: The noudeb build profile and dh-only rules files"):
> Thanks, that's really helpful.  One of the really frustrating things
> I've found about trying to use dh is that there is a real lack of
> examples which are more complicated than:
...
> # See?   dh is easy-peasy!
...
> Sure, there are few a examples using one or two override declarations,
> but trying to use dh on a non-trivial package is.... non-obvious,
> given the current documentation.  Some more advanced tutorials, or
> some links to good examplars of how to use dh in "real world",
> non-trivial packaging setups, would be really good.  It looks like
> openssh is a good example, but I'm sure they must be others.

Here's an example from a package I converted to dh in the last year or
so:
  https://browse.dgit.debian.org/xen.git/tree/debian/rules

I'm not sure where I should send this so that the next person with
your question finds it (other than just posting it to -devel when it
seems relevant).

I highly recommend the approach of writing a comment next to each
override etc., explaining what it's there for.

Ian.

--
Ian Jackson <[hidden email]>   These opinions are my own.

If I emailed you from an address @fyvzl.net or @evade.org.uk, that is
a private address which bypasses my fierce spamfilter.

Reply | Threaded
Open this post in threaded view
|

Re: The noudeb build profile and dh-only rules files

Andreas Metzler-2
In reply to this post by Theodore Y. Ts'o
Theodore Ts'o <[hidden email]> wrote:
[...]
> Thanks, that's really helpful.  One of the really frustrating things
> I've found about trying to use dh is that there is a real lack of
> examples which are more complicated than:

> #!/usr/bin/make -f
> #
> # See?   dh is easy-peasy!

> %:
>         dh $@

> Sure, there are few a examples using one or two override declarations,
> but trying to use dh on a non-trivial package is.... non-obvious,
> given the current documentation.
[...]

Hello,

FWIW my personal experience with converting was that I had to stop
treating debian/rules as a makefile with a dependency tree of
targets. It is just list of dh_auto_*. If you want to change the
buildprocess, override dh_auto_build, etc.

cu Andreas
--
`What a good friend you are to him, Dr. Maturin. His other friends are
so grateful to you.'
`I sew his ears on from time to time, sure'

Reply | Threaded
Open this post in threaded view
|

Re: The noudeb build profile and dh-only rules files

Steve McIntyre
In reply to this post by Paul Wise via nm
Paul Wise wrote:

>On Tue, Jul 9, 2019 at 7:42 AM Colin Watson wrote:
>
>>    A memory-safe language with good testing support and a good testing
>>    culture would be great, though it does also need to work on every
>>    Debian architecture, which IIRC Rust doesn't quite; we've kicked
>>    around the idea of maybe a stripped-down Python in the past,
>
>This reminded me of Keith Packard's "Python for embedded systems", Snek:
>
>https://keithp.com/snek/

While that's a lovely thing for *really* small systems, if we're
contemplating a switch then I think we'd be much better just using a
*normal* Python from our own archive which we already know, maybe with
just a reduced set of modules available in the installer, Let's not
just jump from one limited special-case environment to another!

--
Steve McIntyre, Cambridge, UK.                                [hidden email]
  Armed with "Valor": "Centurion" represents quality of Discipline,
  Honor, Integrity and Loyalty. Now you don't have to be a Caesar to
  concord the digital world while feeling safe and proud.