RFC: Standardizing source package artifacts build paths

classic Classic list List threaded Threaded
27 messages Options
12
Reply | Threaded
Open this post in threaded view
|

RFC: Standardizing source package artifacts build paths

Guillem Jover-2
Hi!

We currently have many built artifacts being dropped directly under
debian/ or under tool specific directories such as debian/.debhelper/.
These have at least the following problems:

  - Make cleaning, an operation that requires executing code from the
    source package itself.
  - Require keeping up updating things like .gitignore or artifacts
    might end up being committed into a VCS.
  - Can stomp over existing files or other tools files, as these are
    generally not namespaced.
  - Clutter the source tree, with debian/ mixing generated and manual
    files.
  - dpkg-dev tools cannot assume the location of the binary package
    to be built, as that depends on the packaging logic and how that
    might drive the various commands.

These files include for example:

  - debian/files
  - debian/*substvars
  - debian/<binary-pkgname>/
  - debian/*debhelper.log
  - debian/.debhelper/
  - debian/tmp/

We'd like to standardize on a new set of artifact build pathnames
for our deb toolchain. These would have the following form:

  - debian/.build/upstream*

    These could be used for out-of-tree builds replacing current
    build-tree/ and similar directories.

  - debian/.build/fsys* (debian/.build/{stage,installed}* ?)

    These would replace the current debian/tmp/ and variants.

  - debian/.build/binary/<pkgname>

    These would be equivalent to the current debian/<pkgname>/.

  - debian/.build/<tool>/<pathname>

    These would be tool specific, for example:

    debian/.build/dpkg/files
    debian/.build/dpkg/files.d/<artifact>
    debian/.build/dpkg/substvars/<pkgname>

    debian/.build/debhelper/<...>

The use of a hidden directory is to reduce clutter and stomping over any
existing directories (there's for example a 3D engine called «build» :).
Most of this could be transparent to most packages, as it would be hidden
behind debhelper logic on a new compat level. There are really few
packages not using debhelper (either directly or indirectly via cdbs).
dpkg-dev tools would preserve backwards compatibility, where applicable,
or could enable this by default only on hypothetical new dpkg-dev compat
levels <https://wiki.debian.org/Teams/Dpkg/Spec/DpkgDevCompatLevel>.

This would give us at least the following properties:

  - We could just (say) .gitignore a single pathname.
  - Cleaning the Debian artifacts would imply doing just
    «rm -rf debian/.build/» (even though this could be abstracted via a
    new dpkg-<something>clean<something> command).
  - Namespacing would avoid any collisions.
  - Would make parallelization way easier and faster.
  - dpkg-dev tools could start defaulting to use package specfific
    directories, instead of requiring the user/tool to specify them.
  - Could make it possible (but not necessary) to move parts of the logic
    from debian/rules (such as calling dpkg-deb) into the build driver
    (see the upcoming proposal), further simplifiying the build process.

Thanks,
Niels and Guillem

Reply | Threaded
Open this post in threaded view
|

Re: RFC: Standardizing source package artifacts build paths

Simon McVittie-7
On Mon, 09 Mar 2020 at 10:09:46 +0100, Guillem Jover wrote:
> We'd like to standardize on a new set of artifact build pathnames
> for our deb toolchain. These would have the following form:
>
>   - debian/.build/upstream*
>
>     These could be used for out-of-tree builds replacing current
>     build-tree/ and similar directories.

Presumably the '*' means it's considered OK for packages that do separate
production/debug/udeb builds to use multiple directories matching that
glob?

For example, a simple debhelper build might eventually (in a future compat
level) default to "dh $@ --builddirectory=debian/.build/upstream",
but because src:glib2.0 does separate deb and udeb builds
(with non-essential features disabled in the udeb), it
would use "--builddirectory=debian/.build/upstream-deb"
and "--builddirectory=debian/.build/upstream-udeb" instead
of its current "--builddirectory=debian/build/deb" and
"--builddirectory=debian/build/udeb"?

>   - We could just (say) .gitignore a single pathname.
>   - Cleaning the Debian artifacts would imply doing just
>     «rm -rf debian/.build/» (even though this could be abstracted via a
>     new dpkg-<something>clean<something> command).
>   - Namespacing would avoid any collisions.
>   - Would make parallelization way easier and faster.
>   - dpkg-dev tools could start defaulting to use package specfific
>     directories, instead of requiring the user/tool to specify them.

This all seems good to have.

The Salsa-CI pipeline (which runs in a git clone of a package, and
currently uses debian/output/ for intermediate build stuff) could maybe
also move its scratch directory to debian/.build/salsa-ci.

Have you seen <https://bugs.launchpad.net/launchpad/+bug/1845159> and
related discussions in the context of sbuild and pbuilder? It seems
like a related topic. Briefly, there's a desire to be able to have
"artifacts" that are not part of the main build result, are produced by
both successful and failing builds, are kept for a while by the buildd
infrastructure, and are probably not uploaded to the Debian archive,
as a generalization of the build log.

For example, Autotools packages have ${builddir}/config.log, which is not
a useful thing to publish in a .deb (not at all reproducible!), but can
be useful to diff between successful and failing builds to debug why a
build failed. At the moment there's a special case in debhelper to cat
config.log if configure fails, but it is forgotten about after the build
if configure succeeds. Meson has ${builddir}/meson-logs/meson-log.txt,
which is analogous.

Similarly, Autotools packages normally output a test summary on success
or a more detailed log on failure (${builddir}/**/test-suite.log
describes all tests that failed, were partially or entirely skipped,
or were otherwise not 100% successful, and there are also detailed logs
that capture diagnostics from even the completely successful tests),
and Meson ${builddir}/meson-logs/testlog.txt is analogous.

At the moment the only artifact, other than the .changes file and the
files it references when the build is successful, is the build log itself,
which means the build process must either cat these other artifacts to
get them into the build log (resulting in large build logs where it's
difficult to find what you're looking for), or lose them forever.

Some packages have machine-readable test output that is used to track
regressions and intermittent failures between builds, for which cat'ing
the test output into the build log is less than ideal: to compare test
results between runs it's necessary to screen-scrape the test output
from among the unstructured/human-readable messages that make up most
of the build log. A few packages work around this by packaging their
test results, which usually makes them non-reproducible.

autopkgtest's ${AUTOPKGTEST_ARTIFACTS} (a directory into which tests are
encouraged to write or copy more detailed results than the ones shown
on stdout) and Gitlab-CI's artifacts:paths: parameter (a list of globs
matching paths in the built directory) are conceptually similar ideas.

For example, dpkg-buildpackage could perhaps read one glob per
line from debian/artifacts and hardlink matched files (if any) into
debian/.build/artifacts for collection by a "larger" framework like
sbuild or pbuilder, or individual packages could copy/link files into there
as they go, or debhelper build-system classes like Autotools and Meson
could know the names of common log files from their build system, or
some combination of those.

    smcv

Reply | Threaded
Open this post in threaded view
|

Re: RFC: Standardizing source package artifacts build paths

Sam Hartman-3
In reply to this post by Guillem Jover-2
I'm concerned about a leading . at least for:

* the debian/tmp replacement
* the replacement for the package install directories under debian.

I think that maintaining those directories such that ls shows them will
be more friendly for new maintainers.
So I'd prefer something like debian/build rather than debian/.build for
at least those directories.

I don't care as much about substvars, debhelper intermediates, debhelper
logs and the like.

Reply | Threaded
Open this post in threaded view
|

Re: RFC: Standardizing source package artifacts build paths

Weatherby,Gerard
FWIW:

We have been using debian packaging from a private repository for the last three years to configure virtual machines.

As a subversion shop, at least, the extraneous files generated by a build aren't an issue because we simply svn-clean after a build. I believe "git clean" does the equivalent operation.

I concur that hiding the build directory with a leading dot is less friendly. It takes a while to understand how package creation / building works, and hiding information from users will make it more difficult.

--
Gerard Weatherby| Application Architect
NMRbox | Department of Molecular Biology and Biophysics | UConn Health
263 Farmington Avenue, Farmington, CT 06030-6406
Phone: 860 679 8484
uchc.edu

________________________________________
From: Sam Hartman <[hidden email]>
Sent: Monday, March 9, 2020 9:23 AM
To: [hidden email]
Cc: [hidden email]; [hidden email]
Subject: Re: RFC: Standardizing source package artifacts build paths

*** Attention: This is an external email. Use caution responding, opening attachments or clicking on links. ***

I'm concerned about a leading . at least for:

* the debian/tmp replacement
* the replacement for the package install directories under debian.

I think that maintaining those directories such that ls shows them will
be more friendly for new maintainers.
So I'd prefer something like debian/build rather than debian/.build for
at least those directories.

I don't care as much about substvars, debhelper intermediates, debhelper
logs and the like.


Reply | Threaded
Open this post in threaded view
|

Re: RFC: Standardizing source package artifacts build paths

Niels Thykier
In reply to this post by Simon McVittie-7
Simon McVittie:

Hi :)

> On Mon, 09 Mar 2020 at 10:09:46 +0100, Guillem Jover wrote:
>> We'd like to standardize on a new set of artifact build pathnames
>> for our deb toolchain. These would have the following form:
>>
>>   - debian/.build/upstream*
>>
>>     These could be used for out-of-tree builds replacing current
>>     build-tree/ and similar directories.
>
> Presumably the '*' means it's considered OK for packages that do separate
> production/debug/udeb builds to use multiple directories matching that
> glob?
>
> For example, a simple debhelper build might eventually (in a future compat
> level) default to "dh $@ --builddirectory=debian/.build/upstream",
> but because src:glib2.0 does separate deb and udeb builds
> (with non-essential features disabled in the udeb), it
> would use "--builddirectory=debian/.build/upstream-deb"
> and "--builddirectory=debian/.build/upstream-udeb" instead
> of its current "--builddirectory=debian/build/deb" and
> "--builddirectory=debian/build/udeb"?
>

Yes.  :)

Among other, it will hopefully also enable debhelper to support multiple
builds natively or better, when we have a standardized path for these
things.

>>   - We could just (say) .gitignore a single pathname.
>>   - Cleaning the Debian artifacts would imply doing just
>>     «rm -rf debian/.build/» (even though this could be abstracted via a
>>     new dpkg-<something>clean<something> command).
>>   - Namespacing would avoid any collisions.
>>   - Would make parallelization way easier and faster.
>>   - dpkg-dev tools could start defaulting to use package specfific
>>     directories, instead of requiring the user/tool to specify them.
>
> This all seems good to have.
>
> The Salsa-CI pipeline (which runs in a git clone of a package, and
> currently uses debian/output/ for intermediate build stuff) could maybe
> also move its scratch directory to debian/.build/salsa-ci.
>
> Have you seen <https://bugs.launchpad.net/launchpad/+bug/1845159> and
> related discussions in the context of sbuild and pbuilder? It seems
> like a related topic. [...]
>
> For example, dpkg-buildpackage could perhaps read one glob per
> line from debian/artifacts and hardlink matched files (if any) into
> debian/.build/artifacts for collection by a "larger" framework like
> sbuild or pbuilder, or individual packages could copy/link files into there
> as they go, or debhelper build-system classes like Autotools and Meson
> could know the names of common log files from their build system, or
> some combination of those.
>
>     smcv
>

I could see how having a well-defined method for extracting artifacts
will be useful (also for CI purposes).

Though, can you elaborate a bit on why the above approach would be
better than a standard ENV variable a la AUTOPKGTEST_ARTIFACTS and some
easy way to declare additional artifacts to be extracted?
  I am fine either way, but I could image that the
debian/.build/artifacts will feature interact with e.g.
"dpkg-buildpackage -tc" where a AUTOPKGTEST_ARTIFACTS replacement would not.

~Niels

Reply | Threaded
Open this post in threaded view
|

Re: RFC: Standardizing source package artifacts build paths

Julian Andres Klode-4
In reply to this post by Guillem Jover-2
On Mon, Mar 09, 2020 at 10:09:46AM +0100, Guillem Jover wrote:
> We'd like to standardize on a new set of artifact build pathnames
> for our deb toolchain. [...]
[...]
> The use of a hidden directory is to reduce clutter and stomping over any

Love the hidden directory.

--
debian developer - deb.li/jak | jak-linux.org - free software dev
ubuntu core developer                              i speak de, en

Reply | Threaded
Open this post in threaded view
|

Re: RFC: Standardizing source package artifacts build paths

Sean Whitton
Hello Julian,

On Mon 09 Mar 2020 at 09:43PM +01, Julian Andres Klode wrote:

> On Mon, Mar 09, 2020 at 10:09:46AM +0100, Guillem Jover wrote:
>> We'd like to standardize on a new set of artifact build pathnames
>> for our deb toolchain. [...]
> [...]
>> The use of a hidden directory is to reduce clutter and stomping over any
>
> Love the hidden directory.

Fair enough, but what do you think about the counterarguments that have
been raised?  What's good about having it hidden?

Thanks.

--
Sean Whitton

signature.asc (847 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: RFC: Standardizing source package artifacts build paths

Simon McVittie-7
In reply to this post by Niels Thykier
On Mon, 09 Mar 2020 at 20:45:13 +0100, Niels Thykier wrote:

> Simon McVittie:
> > For example, dpkg-buildpackage could perhaps read one glob per
> > line from debian/artifacts and hardlink matched files (if any) into
> > debian/.build/artifacts for collection by a "larger" framework like
> > sbuild or pbuilder, or individual packages could copy/link files into there
> > as they go, or debhelper build-system classes like Autotools and Meson
> > could know the names of common log files from their build system, or
> > some combination of those.
>
> Though, can you elaborate a bit on why the above approach would be
> better than a standard ENV variable a la AUTOPKGTEST_ARTIFACTS and some
> easy way to declare additional artifacts to be extracted?

Mainly, I'd prefer something declarative with glob patterns (a bit like
debian/clean or Gitlab-CI's artifacts:paths) rather than having to write
logic like these pseudo-patches:

    -./configure
    +if ./configure; then
    +    : # OK
    +else
    +    error=$?
    +    if [ -n "$DPKG_ARTIFACTS" ]; then
    +        cp config.log "$DPKG_ARTIFACTS/config.log" || :
    +    fi
    +    exit "$error"
    +fi

    -make check
    +if make check; then
    +    : # OK
    +else
    +    error=$?
    +    if [ -n "$DPKG_ARTIFACTS" ]; then
    +        find tests/ -name '*.log' | something something xargs cp
    +    fi
    +    exit "$error"
    +fi

(doubly so if I have to sprinkle backslashes, tabs and double $ over
it to make it a valid Makefile), and I'm sure you think similarly from
the perspective of writing debhelper buildsystem classes.

In particular, if you want to capture Automake test logs without also
capturing unrelated files that happen to be named *.log, as far as
I'm aware there is no general way to do that without knowledge of the
specific package and how its tests are named or arranged, so individual
maintainers are likely to have to be involved in designating which logs
are of interest.

I don't have any particular opinion on whether artifacts should be
collected into debian/.build/artifacts/, into $DPKG_ARTIFACTS/, or
directly into some sort of archive (Gitlab and Jenkins seem to use zip
files, which have the advantage of being seekable, so web frontends
can presumably read individual logs directly out of the zip file if
that's desirable).

    smcv

Reply | Threaded
Open this post in threaded view
|

Standardized way of extracting additional build-time artefacts (was: Re: RFC: Standardizing source package artifacts build paths)

Niels Thykier
Simon McVittie:

> On Mon, 09 Mar 2020 at 20:45:13 +0100, Niels Thykier wrote:
>> Simon McVittie:
>>> For example, dpkg-buildpackage could perhaps read one glob per
>>> line from debian/artifacts and hardlink matched files (if any) into
>>> debian/.build/artifacts for collection by a "larger" framework like
>>> sbuild or pbuilder, or individual packages could copy/link files into there
>>> as they go, or debhelper build-system classes like Autotools and Meson
>>> could know the names of common log files from their build system, or
>>> some combination of those.
>>
>> Though, can you elaborate a bit on why the above approach would be
>> better than a standard ENV variable a la AUTOPKGTEST_ARTIFACTS and some
>> easy way to declare additional artifacts to be extracted?
>
> Mainly, I'd prefer something declarative with glob patterns (a bit like
> debian/clean or Gitlab-CI's artifacts:paths) rather than having to write
> logic like these pseudo-patches:
>
> [...]
>

Ack, I get the part of having a declarative mechanism for selecting files.

Just to clarify something related.  Should debhelper and other tools by
default archive "certain files of possible interest" (e.g. config.log)?
Or should we limit it to "on request only"?

The former makes it simpler for people that are interested in the
"default" parts but also bloats the archive for people that are
interested in just one file.

> I don't have any particular opinion on whether artifacts should be
> collected into debian/.build/artifacts/, into $DPKG_ARTIFACTS/, or
> directly into some sort of archive (Gitlab and Jenkins seem to use zip
> files, which have the advantage of being seekable, so web frontends
> can presumably read individual logs directly out of the zip file if
> that's desirable).
>
>     smcv
>

Thanks for clarifying.  This answered the question I was trying to write. :)

~Niels

Reply | Threaded
Open this post in threaded view
|

Re: Standardized way of extracting additional build-time artefacts (was: Re: RFC: Standardizing source package artifacts build paths)

Paul Wise via nm
On Tue, Mar 10, 2020 at 7:12 AM Niels Thykier wrote:

>

Standardized way of extracting additional build-time artefacts


This reminds me of the BYHAND stuff, I forget how that works though.

--
bye,
pabs

https://wiki.debian.org/PaulWise

Reply | Threaded
Open this post in threaded view
|

Re: Standardized way of extracting additional build-time artefacts (was: Re: RFC: Standardizing source package artifacts build paths)

Simon McVittie-7
In reply to this post by Niels Thykier
On Tue, 10 Mar 2020 at 08:10:55 +0100, Niels Thykier wrote:
> Just to clarify something related.  Should debhelper and other tools by
> default archive "certain files of possible interest" (e.g. config.log)?
> Or should we limit it to "on request only"?

I think it would probably make most sense for dpkg (which doesn't know
about specific build systems) to not archive anything by default, or to
archive only things it produced itself.

For debhelper it might make sense for build system classes to archive
well-known logs like Autotools' ${builddir}/config.log and Meson's
${builddir}/meson-logs/ by default, but probably not logs that are in
an unpredictable location like Autotools' test logs.

If there's an exclusion mechanism for packages that know a particular
artifact is not useful and monstrously large ("!meson-logs/big.log"?) then
it doesn't necessarily matter much either way. If artifacts aren't kept
forever then the damage from archiving too much will be temporary.

> The former makes it simpler for people that are interested in the
> "default" parts but also bloats the archive for people that are
> interested in just one file.

Have you seen the UI Gitlab-CI and Jenkins provide for this? If you look
at a Gitlab-CI job like <https://gitlab.gnome.org/GNOME/glib/-/jobs/624888>,
there's a Download link that gives you the complete bundle of
artifacts in a zip file, but there's also a Browse link like
<https://gitlab.gnome.org/GNOME/glib/-/jobs/624888/artifacts/browse>
that lets you look at individual logs online (which is often enough to
debug an issue). Jenkins has a similar system.

On Salsa, ci.debian.net (for tests with the needs-build restriction),
and similar systems, I think it would make most sense to drop the
artifacts into somewhere the "larger" CI system will pick them up, and
let the "larger" CI system handle browsing and expiry. On
buildd.debian.org, I think build logs are kept forever(?) but artifacts
should probably have some sort of expiration mechanism, similar to the
way ci.debian.net remembers test results indefinitely but discards old
logs after a while.

    smcv

Reply | Threaded
Open this post in threaded view
|

Re: Standardized way of extracting additional build-time artefacts (was: Re: RFC: Standardizing source package artifacts build paths)

Simon McVittie-7
In reply to this post by Paul Wise via nm
On Tue, 10 Mar 2020 at 07:19:59 +0000, Paul Wise wrote:
> On Tue, Mar 10, 2020 at 7:12 AM Niels Thykier wrote:
> > Standardized way of extracting additional build-time artefacts
>
> This reminds me of the BYHAND stuff, I forget how that works though.

I think how that works is that at the appropriate time during a successful
build, you run dpkg-distaddfile to insert extra entries that are not a
recognised file type (.deb, .udeb etc.) into debian/files?

The difference (as I understand it) is that BYHAND is for extra build
products that are listed in the .changes file and intended to be published
to Debian users via ftp.debian.org; whereas in this thread we're talking
about non-essential things that are produced as a side-effect of the
build, are potentially useful to Debian contributors for debugging and
analysis of the build process itself, but are not actually the "product".

Some important trade-offs are different. For example, for the build
products mentioned in the .changes file (whether .deb or BYHAND) we want
reproducible builds that don't capture unnecessary information like the
properties of the build system; whereas in build and test logs, we *do*
want to capture system-specific information in case it's relevant, for
example to help a Debian contributor to realise correlations like "this
test fails whenever we're building on a btrfs filesystem" that can help
them to find and fix bugs.

Similarly, we probably don't want to publish the build products to users
if the build(-time tests) failed (because we can't be confident that any
products that were already produced are good), although we might well
want to make them available through a contributor-oriented interface to
help to debug the failures; but we do want to publish build and test logs
to contributors, regardless of success or failure.

The .buildinfo file is arguably already in the same category as build
and test logs. We currently capture it in the .changes file and upload
it to ftp-master, but it isn't reproducible, and ftp-master doesn't
republish it through our user-facing interface (the archive). Ideally,
failed builds would capture their .buildinfo as well as their log for
subsequent analysis, although I don't know whether they actually do.

    smcv

Reply | Threaded
Open this post in threaded view
|

Re: Standardized way of extracting additional build-time artefacts (was: Re: RFC: Standardizing source package artifacts build paths)

Mattia Rizzolo-5
On Tue, Mar 10, 2020 at 09:07:57AM +0000, Simon McVittie wrote:
> On Tue, 10 Mar 2020 at 07:19:59 +0000, Paul Wise wrote:
> > On Tue, Mar 10, 2020 at 7:12 AM Niels Thykier wrote:
> > > Standardized way of extracting additional build-time artefacts
> >
> > This reminds me of the BYHAND stuff, I forget how that works though.
[…]
> Similarly, we probably don't want to publish the build products to users
> if the build(-time tests) failed (because we can't be confident that any
> products that were already produced are good), although we might well
> want to make them available through a contributor-oriented interface to
> help to debug the failures; but we do want to publish build and test logs
> to contributors, regardless of success or failure.

And this highlights one important aspect of such interface: such
artifacts would be collected even after a build failure.
That's not possible at all now.

> The .buildinfo file is arguably already in the same category as build
> and test logs. We currently capture it in the .changes file and upload
> it to ftp-master, but it isn't reproducible, and ftp-master doesn't
> republish it through our user-facing interface (the archive). Ideally,
> failed builds would capture their .buildinfo as well as their log for
> subsequent analysis, although I don't know whether they actually do.

That's somewhat of a tough argument, that I'd try to keep separate (it
has to do with the semantic meaning of a .buildinfo (i.e., it tries to
attach a *built artifacts* to the way it was build, a .buildinfo without
any hashes would be quite meaningless when tied to its original meaning.
Also, we do want it to be published, but we are still waiting for the
ftp-masters to tell us their distribution requirements...).

--
regards,
                        Mattia Rizzolo

GPG Key: 66AE 2B4A FCCF 3F52 DA18  4D18 4B04 3FCD B944 4540      .''`.
More about me:  https://mapreri.org                             : :'  :
Launchpad user: https://launchpad.net/~mapreri                  `. `'`
Debian QA page: https://qa.debian.org/developer.php?login=mattia  `-

signature.asc (849 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Standardized way of extracting additional build-time artefacts (was: Re: RFC: Standardizing source package artifacts build paths)

Mattia Rizzolo-5
In reply to this post by Niels Thykier
(note, this is a barely structured brain dump)

On Tue, Mar 10, 2020 at 08:10:55AM +0100, Niels Thykier wrote:
> >> Though, can you elaborate a bit on why the above approach would be
> >> better than a standard ENV variable a la AUTOPKGTEST_ARTIFACTS and some
> >> easy way to declare additional artifacts to be extracted?
> >
> > Mainly, I'd prefer something declarative with glob patterns (a bit like
> > debian/clean or Gitlab-CI's artifacts:paths) rather than having to write
> > logic like these pseudo-patches:

Same.
Having such directory be hidden inside
/build/foo-1.2.3/debian/.build/artefacts/ is kind of a mouthful, but if
that gets to be standardized I think it would be awesome (builders
(sbuild, pbuilder, …) decide on the first '/build/foo-1.2.3/' part of
the path and they know of it; package building happens with CWD in that
place, so build tools should just try to stick to relative paths
'./debian/.build/artefacts/'; everything should Just Work that way).

One thing that strikes me of this proposal, is that you were trying to
"hide" that .build directory from the maintainer; doing this would be
going against that design decision.  This is the only "concern" I have
with the proposal.  Probably this can be avoided by providing a dh_
helper.

> Ack, I get the part of having a declarative mechanism for selecting files.

And then builder could just take out the whole directory.  If that gets
to be (g|x|)zipped or not would be an implementation detail of the
builders (sbuild, pbuilder, …) and of whatever frontend (launchpad,
buildd + wanna-build, …) is used.

> Just to clarify something related.  Should debhelper and other tools by
> default archive "certain files of possible interest" (e.g. config.log)?
> Or should we limit it to "on request only"?

That would be some nice automatism indeed, but I think it's something
for "later".  If you do, please consider these bits:
 * naming the files: you risk clashing with maintainer-set file names
 * deciding on whether to put those files there only on failure or all
   the time

> The former makes it simpler for people that are interested in the
> "default" parts but also bloats the archive for people that are
> interested in just one file.

"bloating" is indeed important.  If we start doing this, frontends need
to decide on a retaining policy.  Do we want maintainers to have a say
on this?  Like, adding a metadata file to the artifacts to indicate any
interest on those files (this is a successful build: keep for x
days/keep until next successful build + y days, etc etc).

> > I don't have any particular opinion on whether artifacts should be
> > collected into debian/.build/artifacts/, into $DPKG_ARTIFACTS/, or
> > directly into some sort of archive (Gitlab and Jenkins seem to use zip
> > files, which have the advantage of being seekable, so web frontends
> > can presumably read individual logs directly out of the zip file if
> > that's desirable).
>
> Thanks for clarifying.  This answered the question I was trying to write. :)


I think I took care of those thoughts above, but to reiterate:
 * IMHO ./debian/.build/artefacts/ (or artifacts? :P) is a cool and
   accessible place for all interested software
 * perhaps, you could consider using
   ${DPKG_ARTEFACTS:-$PWD/debian/.build/artefacts} so that some builders
   can override the directory if they find it more convenient for some
   reason, but otherwise I'd rather stick to a stable, non-changable
   path.
 * I think eventual tarball/compression should be left as a matter for
   the build driver (sbuild, pbuilder, …).

--
regards,
                        Mattia Rizzolo

GPG Key: 66AE 2B4A FCCF 3F52 DA18  4D18 4B04 3FCD B944 4540      .''`.
More about me:  https://mapreri.org                             : :'  :
Launchpad user: https://launchpad.net/~mapreri                  `. `'`
Debian QA page: https://qa.debian.org/developer.php?login=mattia  `-

signature.asc (849 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: RFC: Standardizing source package artifacts build paths

Guillem Jover
In reply to this post by Sam Hartman-3
On Mon, 2020-03-09 at 09:23:46 -0400, Sam Hartman wrote:

> I'm concerned about a leading . at least for:
>
> * the debian/tmp replacement
> * the replacement for the package install directories under debian.
>
> I think that maintaining those directories such that ls shows them will
> be more friendly for new maintainers.
> So I'd prefer something like debian/build rather than debian/.build for
> at least those directories.
>
> I don't care as much about substvars, debhelper intermediates, debhelper
> logs and the like.

As mentioned on the proposal the use of a hidden directory is to
reduce clutter (getting the packaged bits alongside the generated
pathnames when listing the debian/ directory contents has always
seemed extremely distracting, which also makes it too easy for them
to end up on a VCS, f.ex.) and to avoid collisions with existing
packaging (due to temporary directory usage or due to staged package
directories).

There's also precedent with hidden directories with debhelper, even
for the staged directories for dbgsyms packages. Making internal stuff
non-hidden would make it more distracting, and if this would mean having
at least two hierarchies (one hidden and one non-hidden), that can be
even more confusing and is more work to cleanup and ignore, which goes
in the opposite direction of simplifying things IMO.

While it's true that we might need to use such pathnames in debian/rules
or debhelper fragment files (which some might consider ugly), IMO that
has always felt like a sign that there's something missing in our
packaging helpers/tools. To me this calls f.ex. for making dpkg-dev and
debhelper support build flavors (for multiple builds) or for debhelper
to automatically look in the relevant build directories when looking up
for files, so in addition to look for them in the installation stage
directory and the source package root directory, it would look there
too, removing the major needs for having to list the full pathnames
explicitly.

Thanks,
Guillem

Reply | Threaded
Open this post in threaded view
|

Re: RFC: Standardizing source package artifacts build paths

Marco d'Itri
On Mar 29, Guillem Jover <[hidden email]> wrote:

> While it's true that we might need to use such pathnames in debian/rules
> or debhelper fragment files (which some might consider ugly), IMO that
> has always felt like a sign that there's something missing in our
> packaging helpers/tools.
If you believe this then it means that you have been maintaining too
modern or too much simple packages.

https://salsa.debian.org/md/inn/-/blob/master/debian/rules
https://salsa.debian.org/md/inn2/-/blob/master/debian/rules
https://salsa.debian.org/md/kmod/-/blob/master/debian/rules
https://salsa.debian.org/md/binkd/-/blob/master/debian/rules
https://salsa.debian.org/md/libxcrypt/-/blob/master/debian/rules

--
ciao,
Marco

signature.asc (235 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: RFC: Standardizing source package artifacts build paths

Guillem Jover
On Sun, 2020-03-29 at 22:48:04 +0200, Marco d'Itri wrote:
> On Mar 29, Guillem Jover <[hidden email]> wrote:
> > While it's true that we might need to use such pathnames in debian/rules
> > or debhelper fragment files (which some might consider ugly), IMO that
> > has always felt like a sign that there's something missing in our
> > packaging helpers/tools.

> If you believe this then it means that you have been maintaining too
> modern or too much simple packages.

Aww thanks, you are, as always, way too kind, and you probably even
listed too many reasons there.

> https://salsa.debian.org/md/inn/-/blob/master/debian/rules
> https://salsa.debian.org/md/inn2/-/blob/master/debian/rules
> https://salsa.debian.org/md/kmod/-/blob/master/debian/rules
> https://salsa.debian.org/md/binkd/-/blob/master/debian/rules
> https://salsa.debian.org/md/libxcrypt/-/blob/master/debian/rules

Most of these show some of the things I had in mind. Which have not
been supported (easily) in the past or yet by our packaging toolchain,
like:

  - declarative file metadata,
  - renaming files,
  - flavor builds (like udeb vs no-udeb, or w/ features disabled in
    configure, etc.),

Some other usages there though, look like things that can really be
simplified in the packaging, by say, just:

  - using dh_install, dh_link and friends,
  - being explicit in things to install in the .install fragments
    instead of just adding whole hierarchies,
  - using a temporary destdir (like debhelper does by default) and only
    installing the desired pathnames to the final installation directory.

Other things there can be simplified by improving the upstream build
system, also by parametrizing things, and submitting these upstream or
just carrying as local patches.

So yes, thanks, to me these examples seem to still confirm that the
explicit usage of the staged directories is an indication of things that
need fixing or improving in either the packaging or the toolchain.

And, of course there are always going to be remaining sticking points
not covered by features I or others have in mind, but IMO their presence
will still mean there's something to improve somewhere.

Regards,
Guillem

Reply | Threaded
Open this post in threaded view
|

Re: RFC: Standardizing source package artifacts build paths

Andreas Metzler-2
In reply to this post by Sam Hartman-3
On 2020-03-09 Sam Hartman <[hidden email]> wrote:
> I'm concerned about a leading . at least for:

> * the debian/tmp replacement
> * the replacement for the package install directories under debian.

> I think that maintaining those directories such that ls shows them will
> be more friendly for new maintainers.
> So I'd prefer something like debian/build rather than debian/.build for
> at least those directories.

I would like to add a strong AOL. Please do not use a hidden toplevel
directory. debian/.build does not avoid clutter - it just makes it harder
to recognize a cluttered directory. Not "stomping over any existing
directories" is also a weak argument for ".build" vs any other name,
neither has a guarantee that there are no conflicts currently.

cu Andreas

Reply | Threaded
Open this post in threaded view
|

Re: RFC: Standardizing source package artifacts build paths

Niels Thykier
In reply to this post by Sam Hartman-3
Sam Hartman:

> I'm concerned about a leading . at least for:
>
> * the debian/tmp replacement
> * the replacement for the package install directories under debian.
>
> I think that maintaining those directories such that ls shows them will
> be more friendly for new maintainers.
> So I'd prefer something like debian/build rather than debian/.build for
> at least those directories.
>
> I don't care as much about substvars, debhelper intermediates, debhelper
> logs and the like.
>

Hi,

Guillem and I have debated this and come to a solution, which we believe
will be able to address the concerns about the path being "hidden". It
also enables us to simplify parts of the migration in some cases.

The short version of that solution is to enable debhelper to expose the
relevant directories where you want it and under debian/.build at the
same time (the latter being a symlink to the former)[1].
  This can be made to cover the directories you mention above, but also
e.g. the build directory for upstream builds, which was mentioned
elsewhere in this thread.

Based on the feedback so far, I will go with the following defaults in
debhelper:

 * build directories are exposed if -B/--builddirectory is used to
   request it.  If there is no explicit -B/--builddirectory, it will
   (in a new compat level) be hidden by default.

 * The staging directories a la d/tmp will be exposed if --destdir is
   used with dh_auto_install.
   - I will expose d/tmp by default for now but "parallel" d/tmp-X dirs
     are not (they currently require a --destdir to work as it is now).

 * d/<pkg> are exposed by default for now.


Any other debhelper artifact will move to a new path in a future compat
level (at this stage, we are talking debhelper/14 rather than debhelper/13).

~Niels

[1] Note that dpkg will migrate to always use the paths under
debian/.build.  This should not matter in practise unless you fiddle
with the symlinks that debhelper creates.

Reply | Threaded
Open this post in threaded view
|

Re: RFC: Standardizing source package artifacts build paths

Sean Whitton
Hello,

On Tue 14 Apr 2020 at 10:54AM +02, Niels Thykier wrote:

> Guillem and I have debated this and come to a solution, which we believe
> will be able to address the concerns about the path being "hidden". It
> also enables us to simplify parts of the migration in some cases.

Thank you for your continued efforts!

> The short version of that solution is to enable debhelper to expose the
> relevant directories where you want it and under debian/.build at the
> same time (the latter being a symlink to the former)[1].

Sorry, but don't you mean the former (non-dotdirs) being a symlink to
the latter (.build)?

--
Sean Whitton

signature.asc (847 bytes) Download Attachment
12