Is running dpkg-buildpackage manually from the command line forbidden?

classic Classic list List threaded Threaded
20 messages Options
Reply | Threaded
Open this post in threaded view
|

Is running dpkg-buildpackage manually from the command line forbidden?

Daniel Schepler-6
I've been running a manual test bootstrap of Debian (starting with
cross-compiled packages amd64 -> i386 up to the point I was able to
install debhelper), and posting a few bugs I've found along the way.
These are where I found that having extra packages installed during
the dpkg-buildpackage run either failed or resulted in broken
packages.  (Some examples of the type of thing I mean: #948522,
#887902.)

However, I've been getting push back on some of these, with
maintainers of the opinion that it isn't actually a bug.  So, I
thought I'd consult here to get more opinions on whether these are
true bugs, or whether I'm at fault for trying to run dpkg-buildpackage
manually instead of using it through pbuilder or sbuild.

My arguments in favor of such things being bugs:

1. I've been using Debian since before pbuilder or sbuild even
existed, and I don't remember ever seeing any announcements along the
lines of "using pbuilder or sbuild is now mandatory, running
dpkg-buildpackage manually is forbidden".  (Just announcements that of
course, testing package builds using one of those before uploading is
strongly encouraged.)
2. The mere existence of the Build-Conflicts field.
3. The general principle that the Build-Depends are meant not to
describe every possible way the package *could* be built, but to pin
down the exact environment in which the package *should* be built, in
order to avoid unnecessary differences in the resulting packages
between architectures.
4. The build-essential package set could evolve over time, and in a
few cases that could come back to bite maintainers.  For a
hypothetical example: what if Debian eventually decided to add cmake,
ninja, and meson to the build-essential set?  Or, what if there were a
source package that made a build time check of whether a libpopt.so.2
file exists to be dlopen()ed, but if it's found enabled broken code;
and then, eventually, one of the build-essential packages added a
dependency on libpopt?

Possible arguments I can anticipate against such things being considered bugs:

1. It would be very difficult to impossible to test for every possible
combination of packages that satisfy the Build-Depends.  (Though I
would think a vast majority of such bugs would be detected by a
reproducible-builds type setup with one build being the standard
minimal chroot, and the other build using a chroot with as many
packages as possible installed.)
2. It would be pointless to worry about such things, especially now
that all uploads to the archives must be source only.  (To which my
answer would be: requiring use of pbuilder or sbuild would place a
burden on users who previously would have made local patches by a
sequence of "apt-get source package; cd package-*/; edit;
dpkg-buildpackage -b -uc; sudo apt-get install ../*.deb )

(Somewhat related to this: I've also found a few packages that hang
when building from the command line, waiting for input from stdin;
whereas in pbuilder or sbuild, with input redirected to /dev/null or
similar, the builds succeed.  Would these be considered bugs as well?
Of course, in some situations it looks like it detects an
incontrovertible bug, such as when an "rm" command hangs on the prompt
for confirmation on a read-only file, and the /dev/null stdin case
would just result in those files being left in place.  I've especially
been seeing the latter sort of thing related to Perl packages now that
recent Perls install lots of files as read only.)
--
Daniel Schepler

Reply | Threaded
Open this post in threaded view
|

Re: Is running dpkg-buildpackage manually from the command line forbidden?

Ansgar Burchardt-5
On Thu, 2020-01-16 at 08:50 -0800, Daniel Schepler wrote:
> These are where I found that having extra packages installed during
> the dpkg-buildpackage run either failed or resulted in broken
> packages.  (Some examples of the type of thing I mean: #948522,
> #887902.)

If you build outside a controlled environment the set of additional
packages are just a small part of your worries.  There are others such
as

- more creative solutions to satisfy Build-Depends,
- anything in /usr/local,
- anything in /opt (if PATH or other configuration includes /opt),
- which alternative is active,
- diversions,
- local configuration, including environment variables (PATH,
  variables changing behavior of the Python interpreter, ...),
- modified files,
- local packages that do not exist in Debian's archive, or
- modified packages.

In many cases this will not be a problem, but trying to make any
guarantees that the package building process will still work reliably
seems to be a lost cause to me.

It is fine to ask a maintainer to include some `--disable-something`
flag if this is easy to do and reasonably likely to be a problem, but I
would not expect people to actively spend effort on this.

Ansgar

Reply | Threaded
Open this post in threaded view
|

Re: Is running dpkg-buildpackage manually from the command line forbidden?

Sam Hartman-3
In reply to this post by Daniel Schepler-6
>>>>> "Daniel" == Daniel Schepler <[hidden email]> writes:

    Daniel> However, I've been getting push back on some of these, with
    Daniel> maintainers of the opinion that it isn't actually a bug.  So, I
    Daniel> thought I'd consult here to get more opinions on whether these are
    Daniel> true bugs, or whether I'm at fault for trying to run dpkg-buildpackage
    Daniel> manually instead of using it through pbuilder or sbuild.

I think it is often a bug, but rarely if ever a serious bug.

I'd say that a good test is whether you can produce a clean, simple
patch.
(adding build-conflicts seems like a fine answer in a lot of places).
If something like build-conflicts is sufficient, then sure, file the bug
with a patch if it  actually annoys you.

If the patch is going to be ugly, then perhaps it's not worth it.

Reply | Threaded
Open this post in threaded view
|

Re: Is running dpkg-buildpackage manually from the command line forbidden?

Simon McVittie-7
In reply to this post by Daniel Schepler-6
On Thu, 16 Jan 2020 at 08:50:25 -0800, Daniel Schepler wrote:

> I've been running a manual test bootstrap of Debian (starting with
> cross-compiled packages amd64 -> i386 up to the point I was able to
> install debhelper), and posting a few bugs I've found along the way.
> These are where I found that having extra packages installed during
> the dpkg-buildpackage run either failed or resulted in broken
> packages.  (Some examples of the type of thing I mean: #948522,
> #887902.)
>
> However, I've been getting push back on some of these, with
> maintainers of the opinion that it isn't actually a bug.  So, I
> thought I'd consult here to get more opinions on whether these are
> true bugs, or whether I'm at fault for trying to run dpkg-buildpackage
> manually instead of using it through pbuilder or sbuild.

I would say it's often a bug, but usually (always?) a non-RC bug, because
there's a straightforward workaround (build in a minimal environment).

Opening bugs as RC tends to put maintainers on the defensive (if nothing
else because RC bugs have automatic consequences, like autoremovals), so
it's often better to err on the side of opening bugs with a lower severity
(and escalating if there's later consensus that this should happen)
rather than starting with a severity that could be seen as unjustified.

FTBFS and broken packages when built in a non-minimal environment are
special cases of the more general problem of getting a different package
when built in a non-minimal environment, which is sometimes trivial
to solve, but somewhat intractable to detect and solve in general,
because there are so many different non-minimal environments - as Ansgar
mentioned, not just packages that we can capture in Build-Depends and
Build-Conflicts, but also the contents of directories like /usr/local,
/opt and ~/bin, and various properties of your system such as whether it
has undergone the /usr merge, whether alternatives have been reconfigured,
the filesystem you're running on, whether an LSM is in use, ...

I think it's best-practice for package maintainers to explicitly disable
optional features that are autodetected but aren't wanted in the package
(./configure --disable-weasels --without-libstoat, or equivalent), but
this isn't always possible without relatively extensive patches to the
upstream build system, which themselves carry a cost.

> The mere existence of the Build-Conflicts field [implies that we are
> meant to use it]

Yes to a point, but here's a somewhat devil's-advocate response to that:

Suppose we lived in a parallel timeline where there is a serious QA effort
put into making sure packages produce the same results, regardless of
how they are built, as long as their Build-Depends and Build-Conflicts
(and their -Arch, -Indep flavours) are satisfied.

In that world, one rational thing for maintainers of packages with
optional features to do would be to add an ever-increasing number of
Build-Conflicts (every time someone notices a package that can alter or
breaks the build). I could easily imagine it becoming no longer feasible
to build large sets of packages unless you use sbuild or equivalent,
because at least a few of those packages will have a Build-Conflicts on
something you needed for another of the packages - which might not even
be necessary any more, but as an individual package maintainer, there's
little incentive to *remove* Build-Conflicts on things that you yourself
don't use, particularly if insufficient Build-Conflicts are considered RC.

So, pragmatically, it might be a good idea to optimize away the steps
where we put lots of effort into QA, report lots of bugs, get them
fixed by addition of Build-Conflicts, and find ourselves having to use
a minimal build environment to get your desired set of packages built -
by using a minimal build environment to start with. And we're back to
the advice to use pbuilder or sbuild.

I don't think it's coincidence that there has been so much parallel
evolution between Debian's pbuilder and sbuild (among others), Fedora's
mock, openSUSE's obs-build, Flatpak apps' flatpak-builder, and lots of
projects' Docker-based build systems - getting a reliable and reproducible
build of a complex package/module in minimal environments is already quite
difficult, which makes the harder task of making it reliably reproducible
in arbitrary non-minimal environments look rather unappealing.

> It would be very difficult to impossible to test for every possible
> combination of packages that satisfy the Build-Depends.  (Though I
> would think a vast majority of such bugs would be detected by a
> reproducible-builds type setup with one build being the standard
> minimal chroot, and the other build using a chroot with as many
> packages as possible installed.)

I think I dimly remember someone setting up "the buildd from hell"
which deliberately did this as a QA mechanism, but it doesn't seem to
have continued in any systematic way.

> requiring use of pbuilder or sbuild would place a
> burden on users who previously would have made local patches by a
> sequence of "apt-get source package; cd package-*/; edit;
> dpkg-buildpackage -b -uc; sudo apt-get install ../*.deb

I think in an ideal world, we'd have better tools for those users to build
modified packages in a way that more closely resembles what happens on
the production Debian infrastructure - which might for instance mean CI
services, or my vectis[1] tool, or sbuild-debian-developer-setup, or even
autopkgtest (which is really for testing packages, but as a side-effect,
it knows how to build packages in an environment that in practice is going
to be close-to-minimal).

pbuilder and the usual sbuild+schroot setup have the disadvantage
of requiring root privileges and crossing privilege boundaries,
but vectis uses virtual machines (in practice this means kvm group
membership or udev/logind uaccess, to get write access to /dev/kvm)
and as namespace/container stuff gets more powerful and more trusted, I'd
hope that we'll get a better ability to install build-dependencies and do
builds in unprivileged containers.

A justification for non-minimal build environments that I'm personally
more interested in is debugging, whether by users developing local patches
or by a package's maintainer. For successful, one-off builds, throwing
a .dsc or source directory at a build service and waiting for a .deb to
fall out is sufficient - but when the package isn't building correctly,
it's often useful to be able to try it interactively and iteratively
(without doing a full rebuild every time), in a non-minimal environment
that contains debugging tools. In this situation, you'll often end up
with a package that you are not entirely confident is reproducible,
and the next thing to do after your patch seems ready to propose is to
rebuild the package "properly" to confirm that it still works.

Re-bootstrapping seems a lot more like the "one-off build" use-case to me,
so I would recommend something sbuild-like.

    smcv

[1] https://salsa.debian.org/smcv/vectis

Reply | Threaded
Open this post in threaded view
|

Re: Is running dpkg-buildpackage manually from the command line forbidden?

Ondřej Surý-4
In reply to this post by Daniel Schepler-6
Daniel,

while your effort is valiant, I see a little value in it as there’s no real world use case. While your arguments are valid, you are imposing additional work on generally already overloaded maintainers with unclear goal and purpose.

Perhaps your energy and enthusiasm (which I appreciate) could be spent on helping fixing reproducible builds in packages or cross-building. Those are practical and you won’t find any resistance in accepting patches for these two use cases.

Ondrej
--
Ondřej Surý <[hidden email]>

> On 16 Jan 2020, at 18:21, Daniel Schepler <[hidden email]> wrote:
>
> I've been running a manual test bootstrap of Debian (starting with
> cross-compiled packages amd64 -> i386 up to the point I was able to
> install debhelper), and posting a few bugs I've found along the way.
> These are where I found that having extra packages installed during
> the dpkg-buildpackage run either failed or resulted in broken
> packages.  (Some examples of the type of thing I mean: #948522,
> #887902.)
>
> However, I've been getting push back on some of these, with
> maintainers of the opinion that it isn't actually a bug.  So, I
> thought I'd consult here to get more opinions on whether these are
> true bugs, or whether I'm at fault for trying to run dpkg-buildpackage
> manually instead of using it through pbuilder or sbuild.
>
> My arguments in favor of such things being bugs:
>
> 1. I've been using Debian since before pbuilder or sbuild even
> existed, and I don't remember ever seeing any announcements along the
> lines of "using pbuilder or sbuild is now mandatory, running
> dpkg-buildpackage manually is forbidden".  (Just announcements that of
> course, testing package builds using one of those before uploading is
> strongly encouraged.)
> 2. The mere existence of the Build-Conflicts field.
> 3. The general principle that the Build-Depends are meant not to
> describe every possible way the package *could* be built, but to pin
> down the exact environment in which the package *should* be built, in
> order to avoid unnecessary differences in the resulting packages
> between architectures.
> 4. The build-essential package set could evolve over time, and in a
> few cases that could come back to bite maintainers.  For a
> hypothetical example: what if Debian eventually decided to add cmake,
> ninja, and meson to the build-essential set?  Or, what if there were a
> source package that made a build time check of whether a libpopt.so.2
> file exists to be dlopen()ed, but if it's found enabled broken code;
> and then, eventually, one of the build-essential packages added a
> dependency on libpopt?
>
> Possible arguments I can anticipate against such things being considered bugs:
>
> 1. It would be very difficult to impossible to test for every possible
> combination of packages that satisfy the Build-Depends.  (Though I
> would think a vast majority of such bugs would be detected by a
> reproducible-builds type setup with one build being the standard
> minimal chroot, and the other build using a chroot with as many
> packages as possible installed.)
> 2. It would be pointless to worry about such things, especially now
> that all uploads to the archives must be source only.  (To which my
> answer would be: requiring use of pbuilder or sbuild would place a
> burden on users who previously would have made local patches by a
> sequence of "apt-get source package; cd package-*/; edit;
> dpkg-buildpackage -b -uc; sudo apt-get install ../*.deb )
>
> (Somewhat related to this: I've also found a few packages that hang
> when building from the command line, waiting for input from stdin;
> whereas in pbuilder or sbuild, with input redirected to /dev/null or
> similar, the builds succeed.  Would these be considered bugs as well?
> Of course, in some situations it looks like it detects an
> incontrovertible bug, such as when an "rm" command hangs on the prompt
> for confirmation on a read-only file, and the /dev/null stdin case
> would just result in those files being left in place.  I've especially
> been seeing the latter sort of thing related to Perl packages now that
> recent Perls install lots of files as read only.)
> --
> Daniel Schepler
>

Reply | Threaded
Open this post in threaded view
|

Re: Is running dpkg-buildpackage manually from the command line forbidden?

Daniel Schepler-6
On Thu, Jan 16, 2020 at 11:18 AM Ondřej Surý <[hidden email]> wrote:
> while your effort is valiant, I see a little value in it as there’s no real world use case. While your arguments are valid, you are imposing additional work on generally already overloaded maintainers with unclear goal and purpose.
>
> Perhaps your energy and enthusiasm (which I appreciate) could be spent on helping fixing reproducible builds in packages or cross-building. Those are practical and you won’t find any resistance in accepting patches for these two use cases.

OK, makes sense.  This was, in fact, an offshoot of the beginning
stages of a project to create some way to automate bootstrapping an
architecture starting with automated cross builds of the core
packages.

(Incidentally, another offshoot was creating local patches to sbuild
which add an operation mode using systemd-nspawn --ephemeral to start
a container (along with the base being a BTRFS subvolume to speed up
the cloning), systemd-run -M debian-sid-amd64-xxx
[--property=PrivateNetwork=yes] cmd..., etc.  When I sent a message to
[hidden email] there didn't seem to be any interest in
having me clean up the patches and send them on.  Still, do you think
if I posted bug reports for issues I found due to the builds running
under seccomp filters, as wishlist bugs and with either suggested
patches or a request for advice on further debugging it myself where I
got stuck, that maintainers might be willing to consider them?

Also, by the way, the amd64 -> i386 cross built core packages largely
worked OK, with the exception of gcc-9, which ended up with incorrect
include-fixed/limits.h, and with a compiler that required -lssp when
building with -fstack-protector-strong or -fstack-protector as almost
all Debian packages do.  To anybody on the list: is there something I
did wrong with the cross build there, which was to run
"dpkg-buildpackage -a i386 -B -Pcross"?)
--
Daniel Schepler

Reply | Threaded
Open this post in threaded view
|

Re: Is running dpkg-buildpackage manually from the command line forbidden?

Sam Hartman-3
>>>>> "Daniel" == Daniel Schepler <[hidden email]> writes:

    Daniel> (Incidentally, another offshoot was creating local patches to sbuild
    Daniel> which add an operation mode using systemd-nspawn --ephemeral to start
    Daniel> a container (along with the base being a BTRFS subvolume to speed up
    Daniel> the cloning), systemd-run -M debian-sid-amd64-xxx
    Daniel> [--property=PrivateNetwork=yes] cmd..., etc.  When I sent a message to
    Daniel> [hidden email] there didn't seem to be any interest in
    Daniel> having me clean up the patches and send them on.

This sounds like a communication problem or something.  Having
systemd-nspawn container support in sbuild would be really helpful.
Would you be willing to try reaching out to the sbuild maintainers
again?  If you don't get an answer, in a month or so, please reach out
to me as DPL.  My job would not to be to convince the maintainer to
accept your patches, but rather to facilitate communication so you can
get a clear answer.  The DPL does that from time to time when things
seem to have broken down.

--Sam

Reply | Threaded
Open this post in threaded view
|

Re: Is running dpkg-buildpackage manually from the command line forbidden?

Paul Wise via nm
In reply to this post by Ondřej Surý-4
On Thu, Jan 16, 2020 at 7:18 PM Ondřej Surý wrote:

> while your effort is valiant, I see a little value in it as there’s no real world use case. While your arguments are valid, you are imposing additional work on generally already overloaded maintainers with unclear goal and purpose.

Personally, I think there is value in Daniel's work on bootstrapping
Debian from other operating systems and Helmut's work on bootstrapping
Debian from existing Debian architectures. Both are important projects
and we need Debian and FLOSS in general to be more bootstrappable than
it is now, just like the Reproducible Builds folks made everything
more reproducible. At some point I'd even like to see Debian
bootstrapped from whatever comes out of the the Bootstrappable Builds
project:

https://bootstrappable.org/

--
bye,
pabs

https://wiki.debian.org/PaulWise

Reply | Threaded
Open this post in threaded view
|

Re: Is running dpkg-buildpackage manually from the command line forbidden?

Johannes Schauer-3
In reply to this post by Simon McVittie-7
Hi,

Quoting Simon McVittie (2020-01-16 19:47:02)
> I think I dimly remember someone setting up "the buildd from hell" which
> deliberately did this as a QA mechanism, but it doesn't seem to have
> continued in any systematic way.

is there more information about it somewhere? My inbox has only two emails from
xnox and pabs (in CC) about it. How did it work?

> I think in an ideal world, we'd have better tools for those users to build
> modified packages in a way that more closely resembles what happens on the
> production Debian infrastructure - which might for instance mean CI services,
> or my vectis[1] tool, or sbuild-debian-developer-setup, or even autopkgtest
> (which is really for testing packages, but as a side-effect, it knows how to
> build packages in an environment that in practice is going to be
> close-to-minimal).

My advice would also be to switch from debootstrap to mmdebstrap because the
latter is able to create a chroot with only Essential:yes packages in it while
debootstrap includes Priority:required packages as well. Alternatively,
debootstrap could gain a variant only installing Essential:yes packages and
their dependencies (why doesn't it have that already?).

> pbuilder and the usual sbuild+schroot setup have the disadvantage of
> requiring root privileges and crossing privilege boundaries, but vectis uses
> virtual machines (in practice this means kvm group membership or udev/logind
> uaccess, to get write access to /dev/kvm) and as namespace/container stuff
> gets more powerful and more trusted, I'd hope that we'll get a better ability
> to install build-dependencies and do builds in unprivileged containers.

That can be done with sbuild. With --chroot-mode=unshare, sbuild looks for
rootfs tarballs in ~/.cache/sbuild or you can give it your own tarball with the
--chroot option. Since you can create rootfs tarballs without sudo using
mmdebstrap, you can setup arbitrary chroots and build packages without any
process running with root privileges.

If you don't like the security implications of unshared user namespaces, then
you might want to try --chroot-mode=autopkgtest and
--autopkgtest-virt-server=qemu. Using --autopkgtest-virt-server-opts you can
then supply any qemu compatible system image. Using mmdebstrap with fakechroot
mode and guestfish you can create these system images without becoming root and
without having to enable kernel.unprivileged_userns_clone.

Or if lxc/lxd are the unprivileged containers you are talking about, then you
can just use --autopkgtest-virt-server=lxc and let sbuild do builds in your
existing lxc container.

Thanks!

cheers, josch

signature.asc (849 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Is running dpkg-buildpackage manually from the command line forbidden?

Johannes Schauer-3
In reply to this post by Sam Hartman-3
Hi Daniel, Sam and all,

Quoting Sam Hartman (2020-01-17 01:27:52)

> >>>>> "Daniel" == Daniel Schepler <[hidden email]> writes:
>
>     Daniel> (Incidentally, another offshoot was creating local patches to sbuild
>     Daniel> which add an operation mode using systemd-nspawn --ephemeral to start
>     Daniel> a container (along with the base being a BTRFS subvolume to speed up
>     Daniel> the cloning), systemd-run -M debian-sid-amd64-xxx
>     Daniel> [--property=PrivateNetwork=yes] cmd..., etc.  When I sent a message to
>     Daniel> [hidden email] there didn't seem to be any interest in
>     Daniel> having me clean up the patches and send them on.
>
> This sounds like a communication problem or something.  Having
> systemd-nspawn container support in sbuild would be really helpful.
> Would you be willing to try reaching out to the sbuild maintainers
> again?  If you don't get an answer, in a month or so, please reach out
> to me as DPL.  My job would not to be to convince the maintainer to
> accept your patches, but rather to facilitate communication so you can
> get a clear answer.  The DPL does that from time to time when things seem to
> have broken down.
yes, probably a communication problem. I think you are talking about [1] from
August 11 last year? I replied the same day, telling you to please file a bug
with your patches to continue discussion there. But then there was no response
from you anymore and I don't find your bug in the BTS. Maybe my mail got lost
somehow or I fail finding the bug you filed?

I'm also very excited about having yet another chroot backend being integrated
into sbuild! Though my first comment would be the same as I gave those asking
for a docker backend in #867176: maybe try adding that backend to autopkgtest
first. Then more people would profit from having that type of backend available
and no additional changes would be needed in sbuild because sbuild can already
use autopkgtest backends for package building.

Let me also use this opportunity to ask for help with maintaining sbuild. If
any of you reading this ever felt that sbuild was the thing that is worth your
time, please feel free to reach out to me. We need to make a new release fixing
some easy but important bugs that accumulated in the BTS. I would be very happy
to review and apply people's patches. :)

Thanks!

cheers, josch

[1] [hidden email]

signature.asc (849 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Is running dpkg-buildpackage manually from the command line forbidden?

Ansgar Burchardt-5
In reply to this post by Johannes Schauer-3
Johannes Schauer writes:
> My advice would also be to switch from debootstrap to mmdebstrap because the
> latter is able to create a chroot with only Essential:yes packages in it while
> debootstrap includes Priority:required packages as well. Alternatively,
> debootstrap could gain a variant only installing Essential:yes packages and
> their dependencies (why doesn't it have that already?).

Debian doesn't support systems that don't have "required" packages
installed.  So buildds should have them installed.  Policy states:
"Removing a required package may cause your system to become totally
broken and you may not even be able to use dpkg to put things back, so
only do so if you know what you are doing."

"Essential" packages just have additional requirements (in particular
essential packages must work even in the "unpacked" state).

Ansgar

Reply | Threaded
Open this post in threaded view
|

Re: Is running dpkg-buildpackage manually from the command line forbidden?

Simon Richter
In reply to this post by Daniel Schepler-6
Hi,

On Thu, Jan 16, 2020 at 08:50:25AM -0800, Daniel Schepler wrote:

> I've been running a manual test bootstrap of Debian (starting with
> cross-compiled packages amd64 -> i386 up to the point I was able to
> install debhelper), and posting a few bugs I've found along the way.
> These are where I found that having extra packages installed during
> the dpkg-buildpackage run either failed or resulted in broken
> packages.  (Some examples of the type of thing I mean: #948522,
> #887902.)

I would expect a sensible package build to not depend on package
relationships that aren't declared, in either way, but things in /usr/local
are outside the control of the package system.

So a package that enables an optional feature if a particular other package
is installed, but there is neither a build dependency nor a build conflict
declared is a bug to me.

The Build-Conflicts should still be avoided, by an explicit configure
option "disable this feature".

   Simon

Reply | Threaded
Open this post in threaded view
|

Re: Is running dpkg-buildpackage manually from the command line forbidden?

Guillem Jover
In reply to this post by Ansgar Burchardt-5
On Fri, 2020-01-17 at 11:12:50 +0100, Ansgar wrote:
> Johannes Schauer writes:
> > My advice would also be to switch from debootstrap to mmdebstrap because the
> > latter is able to create a chroot with only Essential:yes packages in it while
> > debootstrap includes Priority:required packages as well. Alternatively,
> > debootstrap could gain a variant only installing Essential:yes packages and
> > their dependencies (why doesn't it have that already?).
>
> Debian doesn't support systems that don't have "required" packages
> installed. So buildds should have them installed.

If these statements are based on the policy quote below, then I do
not agree. I don't see why e2fsprogs would need to be installed on
a chroot, as long as there's nothing depending on it, for example.

> Policy states:
> "Removing a required package may cause your system to become totally
> broken and you may not even be able to use dpkg to put things back, so
> only do so if you know what you are doing."

That seems wrong, or inaccurate at best. dpkg should never depend on
anything that is not part of the pseudo-essential set (strictly
speaking only Essential:yes + awk-virtual), or that it depends on
explicitly. So as long as a package has not been forced out, dpkg
must work.

Removing a required package (that is not Essential:yes) might indeed
render your system broken, but what broken means or in what context is
not qualified there. This could apply to systems that get booted for
example, but not to chroots. A package that relies on another package
that is Priority:required and not Essential:yes, and that it does not
depend on, is just broken.

Here's the current list of these packages on my system:

  $ aptitude -F '%p' search '~prequired !~E'
  debconf
  e2fsprogs
  gcc-10-base
  gcc-9-base
  libpam-modules
  libpam-modules-bin
  libpam-runtime
  mawk
  mount
  passwd
  tzdata

And most of these are actually part of the pseudo-essential set
anyway, and cannot be removed w/o force.

That passage in policy might have made sense some time ago, when
Priority:required almost matched the pseudo-essential set, and when we
didn't have a separation of "dpkg-essential" and "boot-essential".

Regards,
Guillem

Reply | Threaded
Open this post in threaded view
|

Re: Is running dpkg-buildpackage manually from the command line forbidden?

Helmut Grohne
In reply to this post by Daniel Schepler-6
Hi Daniel,

On Thu, Jan 16, 2020 at 12:04:12PM -0800, Daniel Schepler wrote:
> Also, by the way, the amd64 -> i386 cross built core packages largely
> worked OK, with the exception of gcc-9, which ended up with incorrect
> include-fixed/limits.h, and with a compiler that required -lssp when
> building with -fstack-protector-strong or -fstack-protector as almost
> all Debian packages do.  To anybody on the list: is there something I
> did wrong with the cross build there, which was to run
> "dpkg-buildpackage -a i386 -B -Pcross"?)

This sounds a lot like you're running into
https://gcc.gnu.org/bugzilla/show_bug.cgi?id=80677.

Cross building gcc tends to not work in my experience, see:
http://crossqa.debian.net/src/gcc-8
http://crossqa.debian.net/src/gcc-9

For gcc-9, the --host flag is not properly passed down to the gm2
component. This is reported as
https://gcc.gnu.org/bugzilla/show_bug.cgi?id=92336.

I think the expectation that it works is misguided.

Adding DEB_BUILD_OPTIONS=nocheck might increase your chances of a
successful build.

Quite certainly, more work is needed here. Unfortunately, we sorely lack
porters and everyone seems to expect that things just work without
anyone doing the work.

Helmut

Reply | Threaded
Open this post in threaded view
|

Re: Is running dpkg-buildpackage manually from the command line forbidden?

Helmut Grohne
In reply to this post by Daniel Schepler-6
Hi Daniel,

On Thu, Jan 16, 2020 at 08:50:25AM -0800, Daniel Schepler wrote:
> However, I've been getting push back on some of these, with
> maintainers of the opinion that it isn't actually a bug.  So, I
> thought I'd consult here to get more opinions on whether these are
> true bugs, or whether I'm at fault for trying to run dpkg-buildpackage
> manually instead of using it through pbuilder or sbuild.

Given that I'm also bootstrapping Debian (in a different setting), I'm
running into much the same problems. However, when I file bugs about
build failures in non-standard environments, I don't receive the
push-back that you describe. I cannot reproduce your experience. It's a
rare thing for maintainers to tell me that I'm filing a non-bug and I do
file a *lot* of bugs. Could it be that your sample is very small or
biased? Maybe you can give examples?

In any case, the consensus seems to be that FTBFS in a non-standard
environment is a bug, but not an RC one. And in general, you get to
discuss less with maintainers if your bug includes a patch. ;)

Helmut

Reply | Threaded
Open this post in threaded view
|

Re: Is running dpkg-buildpackage manually from the command line forbidden?

Daniel Schepler-6
In reply to this post by Johannes Schauer-3
On Fri, Jan 17, 2020 at 2:05 AM Johannes Schauer <[hidden email]> wrote:
> yes, probably a communication problem. I think you are talking about [1] from
> August 11 last year? I replied the same day, telling you to please file a bug
> with your patches to continue discussion there. But then there was no response
> from you anymore and I don't find your bug in the BTS. Maybe my mail got lost
> somehow or I fail finding the bug you filed?

It seems the reply must have gotten lost somehow - I don't see it anywhere.

> I'm also very excited about having yet another chroot backend being integrated
> into sbuild! Though my first comment would be the same as I gave those asking
> for a docker backend in #867176: maybe try adding that backend to autopkgtest
> first. Then more people would profit from having that type of backend available
> and no additional changes would be needed in sbuild because sbuild can already
> use autopkgtest backends for package building.

OK, I'll try to get the patch submitted either tonight or tomorrow.
(What I need to clean up is that it's interspersed with changes I made
to have it run with a personal distribution build I've been tinkering
with.  On a quick review, I now notice it's also interspersed with
changes to support using eatmydata only on the apt install commands
and only if not in schroot-update mode, instead of having to put it
into schroot config to apply to all commands - which seems reasonable
to split out into a separate patch to submit.  I also haven't yet
updated documentation files.)

One quick question: I don't see any mention of
$options->{'DISABLE_NETWORK'} in lib/Sbuild/ChrootAutopkgtest.pm,
whereas my new lib/Sbuild/ChrootNspawn.pm does support it.  If I'm not
missing something, then I wonder what it would take to add
DISABLE_NETWORK support to the autopkgtest sbuild engine.
--
Daniel Schepler

Reply | Threaded
Open this post in threaded view
|

Re: Is running dpkg-buildpackage manually from the command line forbidden?

Johannes Schauer-3
Hi,

Quoting Daniel Schepler (2020-01-17 17:58:09)
> > I'm also very excited about having yet another chroot backend being
> > integrated into sbuild! Though my first comment would be the same as I gave
> > those asking for a docker backend in #867176: maybe try adding that backend
> > to autopkgtest first. Then more people would profit from having that type
> > of backend available and no additional changes would be needed in sbuild
> > because sbuild can already use autopkgtest backends for package building.
> OK, I'll try to get the patch submitted either tonight or tomorrow.

thank you!! :)

> One quick question: I don't see any mention of $options->{'DISABLE_NETWORK'}
> in lib/Sbuild/ChrootAutopkgtest.pm, whereas my new lib/Sbuild/ChrootNspawn.pm
> does support it.  If I'm not missing something, then I wonder what it would
> take to add DISABLE_NETWORK support to the autopkgtest sbuild engine.

it would require some way to tell the autopkgtest backends to disable network.
To my knowledge that is not possible. So as of today, the only backend where
you can disable network for the build (not the apt install) is the "unshare"
backend.

More info here:
https://wiki.debian.org/sbuild#Disabling_network_access_for_dpkg-buildpackage

Thanks!

cheers, josch

signature.asc (849 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Is running dpkg-buildpackage manually from the command line forbidden?

Paul Wise via nm
In reply to this post by Johannes Schauer-3
On Fri, 2020-01-17 at 10:58 +0100, Johannes Schauer wrote:
> Quoting Simon McVittie (2020-01-16 19:47:02)
> > I think I dimly remember someone setting up "the buildd from hell" which
> > deliberately did this as a QA mechanism, but it doesn't seem to have
> > continued in any systematic way.
>
> is there more information about it somewhere? My inbox has only two emails from
> xnox and pabs (in CC) about it. How did it work?

I found a reference to it on the wiki:

https://wiki.debian.org/qa.debian.org/ArchiveTesting#TODO

Some discussions about it that I found:

Subject: rebuilding the archive in a dirty chroot: results
<[hidden email]>

Subject: Meaning of the "Altering package upload rules"
<[hidden email]>
<20080216181247.GA13475@garfield>

Subject: Use of the Build-Conflicts field
<CAKTje6F9os15U=B8S=LH=[hidden email]>

Subject: Re: Please drop anacron from task-desktop
<CANBHLUjRVOBWUZd=[hidden email]>

--
bye,
pabs

https://wiki.debian.org/PaulWise

signature.asc (849 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Is running dpkg-buildpackage manually from the command line forbidden?

Holger Levsen-2
In reply to this post by Paul Wise via nm
On Fri, Jan 17, 2020 at 03:02:06AM +0000, Paul Wise wrote:
> Personally, I think there is value in Daniel's work on bootstrapping
> Debian from other operating systems and Helmut's work on bootstrapping
> Debian from existing Debian architectures. Both are important projects
> and we need Debian and FLOSS in general to be more bootstrappable than
> it is now [...]

indeed!

> At some point I'd even like to see Debian
> bootstrapped from whatever comes out of the the Bootstrappable Builds
> project:
>
> https://bootstrappable.org/

Vagrant (cc:ed) was working on this at the last reproducible builds summit.


--
cheers,
        Holger

-------------------------------------------------------------------------------
               holger@(debian|reproducible-builds|layer-acht).org
       PGP fingerprint: B8BF 5413 7B09 D35C F026 FE9D 091A B856 069A AA1C


signature.asc (849 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Is running dpkg-buildpackage manually from the command line forbidden?

David Kalnischkies-4
In reply to this post by Guillem Jover
(Disclaimer: This is a xkcd:386-like response to this subthread)

> Here's the current list of these packages on my system:
>
>   $ aptitude -F '%p' search '~prequired !~E'

The list omits 'apt' as a libapt internally flags it as essential to
grant it the utmost protection by all clients along with its (due to
that) pseudo-essential dependencies both in terms of user actions as
well as (re)solver and installation ordering algorithms.

So, to see the "real" list, you need something like:
$ aptitude -F '%p' search '~prequired !~E' -o pkgCacheGen::ForceEssential=',' -o Dir::Cache=/dev/null

(The second -o is needed to prevent libapt from using its binary caches
and forces it to reparse everything in memory; the first -o is the knob
defaulting to 'apt' if unset. And yes, it is really ',' and you probably
don't want to know why and just accept it as meaning empty list)


Now, suggesting that apt is not an integral part of a Debian system and
could henceforth be removed is of course heresy! The only thing saving
you vile heretics is apts heavy involvement in the creation of these
chroots.


Best regards

David Kalnischkies

signature.asc (849 bytes) Download Attachment