Bootstrappable Debian - proposal of needed changes

[Johannes Schauer]

Hi,

the following is an email written by Wookey and myself.

0. Introduction
===============

The Debian bootstrap build ordering tool Google Summer of Code project
[1] was continued even after the summer ended and recently reached a new
milestone by being able to create a final build order from a dependency
graph [2] for Debian Sid.

By now, all important tools and algorithms have been written [5] to
solve the following problems:

 - find source packages to which build profiles (reduced build
   dependencies) should be added

 - given enough source packages annotated with build profiles, generate
   a final build order which produces a full Debian archive from zero,
   starting from cross compiling a minimal system and natively compiling
   the rest, breaking build dependencies as necessary (and as possible)

Since Debian source packages do not (yet) contain enough meta data to
decide whether or not a build dependency can be dropped, USE flags of
Gentoo source packages were harvested [3] [6]. On top of that,
suggestions from Thorsten Glaser, Patrick McDermott and Daniel Schepler
were used. This way, our current results are hopefully not too far away
from reality,

While the theoretical results do look consistent, this has so far not
been completed in practice due to the following open issues:

 1. missing multiarch annotations prevent the multiarch cross build
dependencies of some source packages from being resolved correctly

 2. not all source packages of the minimal build system are cross
    compilable in practice yet

 3. no decision has been made on the syntax of the new control fields
    (build profiles) which are required for automated bootstrapping

 4. not enough source packages implement build profiles (this depends on
    3 being solved)


More details on this scheme are given at the DebianBootstrap wiki page
[8]. Work has been going on for a couple of years on this, evolving as
practical experience was gained, and input taken from more people.

We therefore make the following proposals (field names not set in stone)
in descending order of importance for us:


1. Build-Profiles
=================

The build profile format was proposed by Guillem Jover together with
other solutions he presented in this document [7] as part of bug#661538.
Build profiles extend the Build-Depends format with a syntax similar to
architecture restrictions but using < and > instead.

  Build-Depends: huge (>= 1.0) [i386 arm] <!embedded !stage1>, tiny

The build dependency "huge" would not be required by the source package
if it is built in the "embedded" or "stage1" profile. This mechanism
neatly allows for removed build-deps, replaced build-deps and added
build-deps, and an arbitrary number of possible 'profiles'.

Besides bootstrapping, these build profiles could also be used for
embedded builds, and to allow for changed buil-deps when cross-building.
One could also imagine that DEB_BUILD_OPTIONS=nodocs could be replaced
by a profile called "nodocs". Patches for dpkg (bug#661538) and dose3
implementing this syntax already exist.

This scheme supersedes an earlier version, (referred-to as 'staged'
builds), which used repeated Build-depends-StageN: lines. See the dpkg
bug#661538 for the evolution of this.

The profile labels are arbitrary but agreement on label usage is
necessary. For bootstrap automation we have been using 'stage1',
'stage2', etc which fits with existing custom in packages which already
have such internal mechanisms using DEB_STAGE (currently gcc, eglibc,
libselinux, gcj, gnat, gdc, linux [9]) These seem like sensible names so
we propose to stick with them. Other useful profiles can be defined in
the future.

The drawback of this syntax is that Build-Dep parsing tools need to be
updated to read/accept it, so uploads of source containing these
annotations cannot be done until the dpkg in buildds at least parses it.


2. Build-Profiles (extension 1)
===============================

When a source package is built with fewer build dependencies (cross,
embedded, stage1, nodocs...), then it often happens that it does not
build one or more of its binary packages at all (e.g. foo-gtk, foo-java,
foo-doc). While this is a minor nuisance during a half automated
bootstrap, a fully automated bootstrapping process needs to know which
binary packages a source package does not build if it is compiled in one
of its profiles. We therefore propose a new field for binary packages in
their control file which indicates for which profiles it builds.

  Builds-With-Profile: !stage1 !embedded

Different profile names are separated by spaces similar to the
Architecture field. A binary package with the above field would not be
built during the profile builds "stage1" or "embedded". Binary packages
which do not have this field would default to being built by every
profile. This field would mean a minor change to dpkg-gencontrol.


3. Build-profiles (extension 2)
===============================

A build profile is set either using a DEB_ environment variable or a
command-line option. DEB_STAGE has been used historically in a few
packages with staged build support, but that is specific to the
staged-builds purpose. For the more generic build-profiles
DEB_BUILD_PROFILE=<label> is proposed instead - (only 7 existing
packages would need to be changed - patches exist for some already).

Setting the build-profile causes dpkg-checkbuilddeps to use the modified
deps, dpkg-gencontrol to mark the built package with a new field:

  Built-With-Profile: stage1 cross

This new field is optional and just meant to mark binary packages such
that they can not accidentally make it to the archive. Another idea is
to encode this information in the package version by adding a ~stage1.
Using the field is more powerful as source packages can also be built
with multiple profiles activated at once and the field can store a list
of profile names. In above example, the binary package was built with
the cross profile activated for cross compilation and the stage1 profile
activated to break a build dependency cycle.

While this field is meant to make sure not to allow any profile built
binary package to be uploaded to the archive, it can also be abused to
only allow "some" build profiles to be uploaded. For example ubuntu
might generally forbid profile built binary packages to be uploaded
except for packages built with the "ubuntu" profile only. Or emdebian
might allow binary packages using the "embedded" profile. This would
allow unified binary packages which are able to build for different
targets. As only one unified binary package can satisfy the needs of
different purposes this can improve the quality of the package as only
one codebase has to be maintained. We already have this (using
dpkg-vendor) where changes only affect the rules file, but as soon as
build-dependency changes are needed that mechanism is insufficient. This
usage of build profiles is not part of this proposal but one of the
possibilities they offer besides allowing automated bootstrapping.


4. Unified field for extensions 1 & 2
=====================================


The Architecture field contains different information depending on its
context [10]. The syntax of profiles behaves similar to those of
architecture specifiers. An alternative name for the field names of the
last two items would therefore be a unified "Profile" field whose
meaning depends on its context:

  Profile: !stage1 !embedded
  Profile: stage1 cross

The first one would appear in binary packages in debian/control and
indicate which binary packages do or do not build with a specific
profile. The second one would appear in DEBIAN/control (the built binary
package) and indicate with which profiles the binary package was built.

This is our favoured option as Build-with-profile/Built-with-profile
will only be confused anyway, and if it works for 'Architecture' there
seems no reason why it's not sensible for 'Profile'.


5. Cross-Builds field
=====================

For even further automation and also for quality assurance, we propose
another new field for source packages which indicates whether or not
this source package is supposed to be cross compilable.

If Debian wants to incorporate the ability to being bootstrappable in its
policy, then there *must* be some packages which are cross compiled for
a minimal build system. Adding this header to those source packages
would make it a bug if they do not actually cross compile. Without this
header, cross compilation would be wishlist as usual.

Furthermore, cross compilation is one of the methods a porter can use to
break build dependency cycles. If some packages carry this new field
then not only could a porter decide quicker whether or not a source
package is cross compilable, an algorithm could also incorporate this
information to automatically break build dependency cycles for cross
compilation.

Some naming ideas:

Cross-Builds: Yes
Does-Cross-Build: Yes
Allows-Cross-Build: Yes


6. Conclusion
=============

If more automated bootstrapping of Debian is desired, then at least build
profiles (1.) should be introduced. For a fully automated bootstrapping of
Debian, the second item (extension 1) is needed as well. The third item
(extension 2) prevents accidental upload of binary packages that have
not been built fully. The last item (5.) adds further convenience to the
process but is not strictly needed.

We will now make an argument how Debian will benefit from allowing a fully
automated bootstrapping process:

 - obvious: it's the simplest possible way to bootstrap Debian for new architectures
 - no need for other distributions in the bootstrapping process (make Debian
   genuinely "universal")
 - better update lagging architectures
 - build packages for architectures that cannot build themselves
 - allow easy sub-arch builds, optimized for a specific CPU
 - continuously check the archive for bootstrappability as a QA measure
 
This mechanism also covers cross-compiler bootstraping. The eglibc, gcc,
and kernel packages already have the neceassary staged-build info, but
the build profiles (1.) part is also needed to specifiy the reduced
build-deps. The cross-toolchain bootstrap ceases to be a special case if
treated this way and just becomes packages to be built in stages using
the profiles mechansim, like many others in the base system (but for
build arch taregtting host, arch, rather than built for host-arch). See
the wiki article at [11].

The question remains of how many source packages would have to have the
proposed new fields added to them to make a full bootstrap from zero
possible. If the Gentoo USE flags were not too far off and assuming or
tools do the correct thing so far, then:

 - the number of source packages that has to be modified with the new
   fields is at maximum 83 (there might be a smaller set).

Another argument why a fully automated bootstrap of Debian might be
necessary is the growing problem size over the years [4]. If this trend
continues it will only become harder to implement the necessary meta
data in the future. If enough meta data is introduced now to make a
fully automated bootstrap possible, then any subsequent work will only
have to be incremental to that.

The main questions to this list are:

 - should Debian be bootstrappable in a fully automated fashion? We
   created the algorithms that can allow this to happen, we just need
   more meta data and a way to encode it

 - do the proposals for the needed fields sound convincing? Can they be
   improved? Do they have fundamental flaws?


cheers, josch and Wookey


Thanks to Thorsten Glaser and Patrick McDermott for feedback, and
numerous others along the way for help developing this scheme.

[1] http://wiki.debian.org/SummerOfCode2012/StudentApplications/JohannesSchauer
[2] http://lists.mister-muffin.de/pipermail/debian-bootstrap/2012-November/000425.html
[3] http://blog.mister-muffin.de/2012/10/10/using-gentoo-to-find-reduced-build-dependencies-for-debian-source-packages/
[4] http://blog.mister-muffin.de/2012/10/13/does-it-become-harder-to-bootstrap-debian-/
[5] https://gitorious.org/debian-bootstrap/bootstrap
[6] https://gitorious.org/debian-bootstrap/gen2deb
[7] http://www.hadrons.org/~guillem/debian/docs/embedded.proposal
[8] http://wiki.debian.org/DebianBootstrap
[9] http://codesearch.debian.net/search?q=DEB_STAGE
[10] http://www.debian.org/doc/debian-policy/ch-controlfields.html#s-f-Architecture
[11] http://wiki.debian.org/MultiarchCrossToolchains


--
To UNSUBSCRIBE, email to [hidden email]
with a subject of "unsubscribe". Trouble? Contact [hidden email]
Archive: http://lists.debian.org/20130115181840.GA16417@hoothoot

[Jakub Wilk-4]

* Johannes Schauer <[hidden email]>, 2013-01-15, 19:18:
>Build profiles extend the Build-Depends format with a syntax similar to
>architecture restrictions but using < and > instead.
>
>  Build-Depends: huge (>= 1.0) [i386 arm] <!embedded !stage1>, tiny
>
[...]
>
>The drawback of this syntax is that Build-Dep parsing tools need to be
>updated to read/accept it, so uploads of source containing these
>annotations cannot be done until the dpkg in buildds at least parses
>it.

Not only dpkg, but also wanna-build, sbuild, lintian, dak, and who knows
what else...

We've been historically very slow at adapting our software to these kind
of changes:

1) Architecture wildcards were implemented in dpkg in January 2006.
pbuilder added support for them in February... 2011.

2) Support for the :any qualifiers in Build-Depends was added to apt in
February 2010, and to dpkg in March 2012; AFAIK it's still not supported
by wanna-build.

--
Jakub Wilk


--
To UNSUBSCRIBE, email to [hidden email]
with a subject of "unsubscribe". Trouble? Contact [hidden email]
Archive: http://lists.debian.org/20130115232653.GA8205@...

[Charles Plessy-12]

Le Tue, Jan 15, 2013 at 07:18:40PM +0100, Johannes Schauer a écrit :
>
> The build profile format was proposed by Guillem Jover together with
> other solutions he presented in this document [7] as part of bug#661538.
> Build profiles extend the Build-Depends format with a syntax similar to
> architecture restrictions but using < and > instead.
>
>   Build-Depends: huge (>= 1.0) [i386 arm] <!embedded !stage1>, tiny

Hi Johannes,

It looks to me that the above is trying to implement the equivalent of
Recommends for build dependancies.  "The Recommends field should list packages
that would be found together with this one in all but unusual installations."

Are you entirely sure that you need to distinguish between profiles, instead of
having the source package build rules do the right things according to which
recommended packages have been installed ?  In that case, your example
could be reduced to:

Build-Depends: tiny
Build-Recommends: huge (>= 1.0)

Have a nice day,

--
Charles Plessy
Tsurumi, Kanagawa, Japan


--
To UNSUBSCRIBE, email to [hidden email]
with a subject of "unsubscribe". Trouble? Contact [hidden email]
Archive: http://lists.debian.org/20130115235806.GA24476@...

[Steve McIntyre]

Charles Plessy wrote:

>Le Tue, Jan 15, 2013 at 07:18:40PM +0100, Johannes Schauer a écrit :
>>
>> The build profile format was proposed by Guillem Jover together with
>> other solutions he presented in this document [7] as part of bug#661538.
>> Build profiles extend the Build-Depends format with a syntax similar to
>> architecture restrictions but using < and > instead.
>>
>>   Build-Depends: huge (>= 1.0) [i386 arm] <!embedded !stage1>, tiny
>
>Hi Johannes,
>
>It looks to me that the above is trying to implement the equivalent of
>Recommends for build dependancies.  "The Recommends field should list packages
>that would be found together with this one in all but unusual installations."
>
>Are you entirely sure that you need to distinguish between profiles, instead of
>having the source package build rules do the right things according to which
>recommended packages have been installed ?  In that case, your example
>could be reduced to:
>
>Build-Depends: tiny
>Build-Recommends: huge (>= 1.0)

No, not at all. This discussion has happened before, and
Build-Recommends has been suggested before. It's broken, leading to
non-deterministic package builds and associated insanity.

Explicit sets of profiles are an alternative that (it seems) are the
new preferred way to do bootstrapping, and also other optional (but
fully-specified) versions of packages. It's quite possible that it may
be necessary to have several stage profiles, depending on how big a
potential package loop you need to break. In this case, you'll need to
have <!stage1> through to <!stageN> listed for that package.

--
Steve McIntyre, Cambridge, UK.                                [hidden email]
"We're the technical experts.  We were hired so that management could
 ignore our recommendations and tell us how to do our jobs."  -- Mike Andrews


--
To UNSUBSCRIBE, email to [hidden email]
with a subject of "unsubscribe". Trouble? Contact [hidden email]
Archive: http://lists.debian.org/E1TvGhZ-0000LS-CC@...

[Steve Langasek]

On Wed, Jan 16, 2013 at 08:58:06AM +0900, Charles Plessy wrote:
> Le Tue, Jan 15, 2013 at 07:18:40PM +0100, Johannes Schauer a écrit :

> > The build profile format was proposed by Guillem Jover together with
> > other solutions he presented in this document [7] as part of bug#661538.
> > Build profiles extend the Build-Depends format with a syntax similar to
> > architecture restrictions but using < and > instead.

> >   Build-Depends: huge (>= 1.0) [i386 arm] <!embedded !stage1>, tiny

> Hi Johannes,

> It looks to me that the above is trying to implement the equivalent of
> Recommends for build dependancies.  "The Recommends field should list
> packages that would be found together with this one in all but unusual
> installations."

Recommends are totally useless for build-dependencies and this idea is a
non-starter.  Builds *must* be reproducible and *must not* rely on arbitrary
variations in the build environment.  The fundamental requirement for these
profiles is for bootstrapping; we must have a guarantee that at each stage
of the bootstrap, the output is well-formed to support the next stage of the
bootstrap, and that at the *end* of the bootstrap the result is a complete
package and not an arbitrary subset.

> Are you entirely sure that you need to distinguish between profiles,
> instead of having the source package build rules do the right things
> according to which recommended packages have been installed ?

Yes.

--
Steve Langasek                   Give me a lever long enough and a Free OS
Debian Developer                   to set it on, and I can move the world.
Ubuntu Developer                                    http://www.debian.org/
[hidden email]                                     [hidden email]

[Paul Wise]

Thanks a lot for your work on this! and to everyone else who worked on
or shaped the proposal.

On Wed, Jan 16, 2013 at 2:18 AM, Johannes Schauer wrote:

>  - should Debian be bootstrappable in a fully automated fashion? We
>    created the algorithms that can allow this to happen, we just need
>    more meta data and a way to encode it

That sounds useful, so yes. arm64 is on the way, it would be a nice
test case but I guess wookey/Sledge are onto that. The SH-5 CPU
architecture apparently exists but has no port. There are also the
architectures with open-source CPU designs OpenRISC and LatticeMico32
(LM32, used in the Milkymist SoC). So it is safe to say that there
could be new Debian ports in the future and that your work would be
very helpful in reducing the investment needed for new ports.

>  - do the proposals for the needed fields sound convincing? Can they be
>    improved? Do they have fundamental flaws?

As an interested bystander, it sounds good.

Agreed about using Profile rather than Build-with-profile/Built-with-profile.

I imagine that most packages will be cross-buildable. Packages outside
the set of bootstrap packages probably don't need to declare if they
are cross-buildable or not? Cross-building is a slightly larger topic
since there are things like arch:all firmware for ARM processors in
the same source package as the tools that upload that firmware. There
are also various packages in the archive (or potential ones) that are
cross-built for non-Debian architectures (like win32, SH-2 or bionic
libc).

If you think this might be interesting to announce more widely (as I
do), please add a paragraph to the next DeveloperNews, which will be
released on debian-devel-announce when we get 5 entries.

http://wiki.debian.org/DeveloperNews

I'm also sending a mail to LWN and the distributions mailing list (at
freedesktop), it might be interesting to developers of other
distributions, especially Gentoo :)

--
bye,
pabs

http://wiki.debian.org/PaulWise


--
To UNSUBSCRIBE, email to [hidden email]
with a subject of "unsubscribe". Trouble? Contact [hidden email]
Archive: http://lists.debian.org/CAKTje6FURVgsXQfXsqLCokHr7FRYF2rJz8a+kbNGP4qWz0WAsA@...

[wookey-4]

+++ Paul Wise [2013-01-16 11:52 +0800]:

> That sounds useful, so yes. arm64 is on the way, it would be a nice
> test case but I guess wookey/Sledge are onto that.

I intend to send an update mail on the state of this later this week.

> If you think this might be interesting to announce more widely (as I
> do), please add a paragraph to the next DeveloperNews, which will be
> released on debian-devel-announce when we get 5 entries.
>
> http://wiki.debian.org/DeveloperNews

Does asking d-devel for feedback count as news? Having this
functionality available for packagers would count as news... But I
agree that telling people about the concept is useful.

> I'm also sending a mail to LWN and the distributions mailing list (at
> freedesktop), it might be interesting to developers of other
> distributions, especially Gentoo :)

If doing that you might want to mention that Johannes will be talking
about his/our bootstrap work in the cross-distro room at FOSDEM.


Wookey
--
Principal hats:  Linaro, Emdebian, Wookware, Balloonboard, ARM
http://wookware.org/


--
To UNSUBSCRIBE, email to [hidden email]
with a subject of "unsubscribe". Trouble? Contact [hidden email]
Archive: http://lists.debian.org/20130116044126.GM5031@...

[Paul Wise]

On Wed, Jan 16, 2013 at 12:41 PM, Wookey wrote:

> I intend to send an update mail on the state of this later this week.

Excellent.

> Does asking d-devel for feedback count as news? Having this
> functionality available for packagers would count as news... But I
> agree that telling people about the concept is useful.

The "news" would be milestone mentioned in the intro.

> If doing that you might want to mention that Johannes will be talking
> about his/our bootstrap work in the cross-distro room at FOSDEM.

Sent a couple of followups, thanks for the suggestion.

--
bye,
pabs

http://wiki.debian.org/PaulWise


--
To UNSUBSCRIBE, email to [hidden email]
with a subject of "unsubscribe". Trouble? Contact [hidden email]
Archive: http://lists.debian.org/CAKTje6FEyOKS3oCZKn8qe11GBMp3N0XW1y3mObHonhweYMoJGg@...

[Neil Williams-4]

On Wed, 16 Jan 2013 00:26:53 +0100
Jakub Wilk <[hidden email]> wrote:

> * Johannes Schauer <[hidden email]>, 2013-01-15, 19:18:
> >Build profiles extend the Build-Depends format with a syntax similar to
> >architecture restrictions but using < and > instead.
> >
> >  Build-Depends: huge (>= 1.0) [i386 arm] <!embedded !stage1>, tiny
> >
> >The drawback of this syntax is that Build-Dep parsing tools need to be
> >updated to read/accept it, so uploads of source containing these
> >annotations cannot be done until the dpkg in buildds at least parses
> >it.
>
> Not only dpkg, but also wanna-build, sbuild, lintian, dak, and who knows
> what else...

It's about which ones need to change. lintian response rates are not
likely to be a problem - once this gets approved. dak doesn't
necessarily need to do anything - most bootstrapping happens outside
the main archive to prepare the ground for a move into debian-ports.
Beyond that point, none of the bootstrapping support is required.

sbuild can use a specified bootstrapping dependency resolver, e.g. the
one used to test the proposal itself. (As could pbuilder.)

Again, as bootstrapping doesn't happen inside the main archive (due to
changed dependencies and changed functionality which would cause
problems), there isn't necessarily anything wanna-build needs to do
with this other than to allow the syntax & ignore it.

> We've been historically very slow at adapting our software to these kind
> of changes:
>
> 1) Architecture wildcards were implemented in dpkg in January 2006.
> pbuilder added support for them in February... 2011.
>
> 2) Support for the :any qualifiers in Build-Depends was added to apt in
> February 2010, and to dpkg in March 2012; AFAIK it's still not supported
> by wanna-build.

There are a lot of people willing to put a lot of time into this
proposal, not just those who have joined the thread so far. Patches,
patches and a few NMU's...

Simply having the mechanism in place and the critical tools updated
will be a major win. There will still be enough changes needed in the
rest of the packages.

--


Neil Williams
=============
http://www.linux.codehelp.co.uk/

[Ansgar Burchardt-8]

On 01/16/2013 08:56, Neil Williams wrote:

> On Wed, 16 Jan 2013 00:26:53 +0100
> Jakub Wilk <[hidden email]> wrote:
>> Not only dpkg, but also wanna-build, sbuild, lintian, dak, and who knows
>> what else...
>
> It's about which ones need to change. lintian response rates are not
> likely to be a problem - once this gets approved. dak doesn't
> necessarily need to do anything - most bootstrapping happens outside
> the main archive to prepare the ground for a move into debian-ports.
> Beyond that point, none of the bootstrapping support is required.

If you want to use new Build-Depends features for packages in the main
archive, dak needs patches too: "dak rm -R" will warn if Build-Depends
are broken by a package removal. So it needs to be able to understand
the field.

python-apt would need to support the field for the same reason. And we
would need the support in stable (or backports) for dak to use it.

> sbuild can use a specified bootstrapping dependency resolver, e.g. the
> one used to test the proposal itself. (As could pbuilder.)

And the official buildd network would need to use these before any
package using build profiles in Build-Depends could be uploaded to the
main archive.

Ansgar


--
To UNSUBSCRIBE, email to [hidden email]
with a subject of "unsubscribe". Trouble? Contact [hidden email]
Archive: http://lists.debian.org/50F67199.5080202@...

[Neil Williams-4]

On Wed, 16 Jan 2013 10:23:37 +0100
Ansgar Burchardt <[hidden email]> wrote:

> On 01/16/2013 08:56, Neil Williams wrote:
> > On Wed, 16 Jan 2013 00:26:53 +0100
> > Jakub Wilk <[hidden email]> wrote:
> >> Not only dpkg, but also wanna-build, sbuild, lintian, dak, and who knows
> >> what else...
> >
> > It's about which ones need to change. lintian response rates are not
> > likely to be a problem - once this gets approved. dak doesn't
> > necessarily need to do anything - most bootstrapping happens outside
> > the main archive to prepare the ground for a move into debian-ports.
> > Beyond that point, none of the bootstrapping support is required.
>
> If you want to use new Build-Depends features for packages in the main
> archive, dak needs patches too: "dak rm -R" will warn if Build-Depends
> are broken by a package removal. So it needs to be able to understand
> the field.

That depends whether bootstrapping <foo> fields need to cover the
*inclusion* of extra Build-Depends only for bootstrapping which would
not normally be installed as a default Debian build. Otherwise, dak can
have a simpler patch to just allow but ignore <foo> content in
Build-Depends.

The main archive only needs to "carry" this extra information without
needing to act upon it. If dak needs patches to allow-and-ignore the
new information, that can be done. Most bootstrapping changes are to
turn off features by not build-depending on packages which can be
turned off in debian/rules.

I'm not sure if we are going to find this situation:

Source: foo
Build-Depends: bar <!embedded>, baz <+embedded>

(especially where bar does not depend on baz and therefore standard
Debian builds would not normally install it. If bar depends on baz
then this isn't a problem and dak can ignore all the <> content
without any effects.)

We may be able to implement dak support to allow-and-ignore and then
deal with the above corner-case later.

Wookey, Johannes: has anything come up so far which would trigger this
corner case?

> python-apt would need to support the field for the same reason. And we
> would need the support in stable (or backports) for dak to use it.

backports would be manageable.
 
> > sbuild can use a specified bootstrapping dependency resolver, e.g. the
> > one used to test the proposal itself. (As could pbuilder.)
>
> And the official buildd network would need to use these before any
> package using build profiles in Build-Depends could be uploaded to the
> main archive.

Isn't that also a case of allow-but-ignore or
allow-with-hardcoded-default ?

--


Neil Williams
=============
http://www.linux.codehelp.co.uk/

[Ian Jackson-2]

Johannes Schauer writes ("Bootstrappable Debian - proposal of needed changes"):
> the following is an email written by Wookey and myself.

Firstly, I want to say thank you!  This seems like excellent work to
me.

> 5. Cross-Builds field
> =====================
>
> For even further automation and also for quality assurance, we propose
> another new field for source packages which indicates whether or not
> this source package is supposed to be cross compilable.

Is it possible that packages might only cross build for certain
targets ?  Or only for certain hosts ?

> 6. Conclusion
> =============
...
> The question remains of how many source packages would have to have the
> proposed new fields added to them to make a full bootstrap from zero
> possible. If the Gentoo USE flags were not too far off and assuming or
> tools do the correct thing so far, then:
>
>  - the number of source packages that has to be modified with the new
>    fields is at maximum 83 (there might be a smaller set).

That sounds very manageable.

> The main questions to this list are:
>
>  - should Debian be bootstrappable in a fully automated fashion? We
>    created the algorithms that can allow this to happen, we just need
>    more meta data and a way to encode it

Yes, please.

>  - do the proposals for the needed fields sound convincing? Can they be
>    improved? Do they have fundamental flaws?

I haven't spotted anything hideously wrong.  I have three suggestions
for changes:

 * Packages should explicitly declare which profiles they support.
   This should be done with a Profile field in the source stanza
   of debian/control.

 * It should be made explicit in the spec that building occurs with
   zero or one profile, not several.

 * The concrete syntax in build-depends should not use < > but rather
   reuse the architecture qualification syntax.

In more detail:

I worry about this:
  Build-Depends: huge (>= 1.0) [i386 arm] <!embedded !stage1>, tiny

This takes the < > metacharacters and reserves them for build
profiles.  Metacharacters are very scarce and should only be used when
necessary.

And indeed the profiles work very much like architectures, just in a
different namespace and with different rules for defining whether a
profile applies to a particular build.

So I think it would be better to make using of this commonality, save
a metacharacter, and have a more regular syntax.  For example:

(a)
  Build-Depends: huge (>= 1.0) [i386 arm] [!profile:embedded !profile:stage1]

Another possibility is this:
(b)
  Build-Depends: huge (>= 1.0) [i386 arm] [profile: !embedded !stage1]

Or this:
(c)
  Build-Depends: huge (>= 1.0) [i386 arm] [!embedded !stage1]
(Since unknown arches and unknown profiles can be safely ignored as
"no match", the parser doesn't need to know which of the keywords are
profiles and which are architectures.)

All of these are a bit longer because they use an identifier rather
than a metacharacter to indicate that the qualifier is talking about
profiles.  I think this is the right tradeoff.  The arrangement could
be extended later by adding new keywords alongside "profile:", if we
wanted different qualifiers (whose application to a particular build
is specified in a different way to arches and profiles).

Of these I prefer (a) or (b).  (c) doesn't have a satisfactory answer
to the namespace registry.  (b) has a more irregular syntax but its
fields are slightly more readable and shorter than (a).

The semantics I would propose for my unified qualifier syntax are as
follows:

  * Multiple qualifiers (each with the syntax [...]) may occur in one
    build-dependency.  The dependency applies iff _all_ the
    qualifiers match.

  * A qualifier must contain either only positive labels
    (in which case it matches iff any label is true), or
    only negated ones (ie prefixed with !) (in which case it
    matches iff no label is true).

  * A label not known to the software processing the field
    is taken not to match.

  * (a) Labels are "<scope>:<value>", or "<arch>".
        Labels in a single qualifier must be in the same scope
        (but this need not be enforced by parsers).
        There is a registry of scopes, which defines the
        registry of labels within each scope.
    or
    (b) Qualifiers start with "<scope>:" iff the scope isn't
        architecture.  Label names are meaningful within scopes.
        There is a registry of scopes, which defines the
        registry of labels within each scope.
    or
    (c) Labels may be names of build profiles, architectures,
        or anything else.
        option (i) There is a registry of labels.
        option (ii) There is no registry of labels and the
            label namespace is a free-for-all.

 * We initially define one scope "profile", for build profiles.

   A build profile is an optional variation that can be applied
   to a particular package, for the purpose of reducing the
   build dependencies and/or avoiding the building of unneeded
   binary packages.

   The builder of a package may apply one single build profile.

   Build profile names are a controlled namespace and are allocated by
   the Debian project.  The following are defined:

      profile:stage1
      profile:stage2
            For multi-stage bootstrap.  A package may only
            declare support for stage2 if it supports stage1 as well.
            These are not a global ordering across the distribution.
            Rather, stage1 is the most limited build of this
            particular package.  stage2 is an intermediate bootstrap
            build, support for which is provided only if it is
            necessary to build this particular package three times.

Ian.


--
To UNSUBSCRIBE, email to [hidden email]
with a subject of "unsubscribe". Trouble? Contact [hidden email]
Archive: http://lists.debian.org/20726.45081.38709.233549@...

[Matthias Klose]

Am 16.01.2013 13:05, schrieb Neil Williams:

> The main archive only needs to "carry" this extra information without
> needing to act upon it. If dak needs patches to allow-and-ignore the new
> information, that can be done. Most bootstrapping changes are to turn off
> features by not build-depending on packages which can be turned off in
> debian/rules.
>
> I'm not sure if we are going to find this situation:
>
> Source: foo Build-Depends: bar <!embedded>, baz <+embedded>
>
> (especially where bar does not depend on baz and therefore standard Debian
> builds would not normally install it. If bar depends on baz then this isn't
> a problem and dak can ignore all the <> content without any effects.)
>
> We may be able to implement dak support to allow-and-ignore and then deal
> with the above corner-case later.
>
> Wookey, Johannes: has anything come up so far which would trigger this
> corner case?

cross building the gcc-4.x packages requires additional build dependencies on
{gobjc++,gccgo,gfortran}-<target-gnu-type> packages.


--
To UNSUBSCRIBE, email to [hidden email]
with a subject of "unsubscribe". Trouble? Contact [hidden email]
Archive: http://lists.debian.org/50F6B2E2.9000003@...

[Matthias Klose]

Am 15.01.2013 19:18, schrieb Johannes Schauer:

> This mechanism also covers cross-compiler bootstraping. The eglibc, gcc,
> and kernel packages already have the neceassary staged-build info, but
> the build profiles (1.) part is also needed to specifiy the reduced
> build-deps. The cross-toolchain bootstrap ceases to be a special case if
> treated this way and just becomes packages to be built in stages using
> the profiles mechansim, like many others in the base system (but for
> build arch taregtting host, arch, rather than built for host-arch). See
> the wiki article at [11].
>
> [11] http://wiki.debian.org/MultiarchCrossToolchains

Please stop calling this "MultiarchCrossToolchains". It was already wrong when
used in the Emdebian context, and on some GSOC pages. What you are apparently
aiming for is a cross compiler re-using the target libraries built (natively).
However this is not yet done, requires dependencies on "foreign" architectures
and only is a very special case, not helping for targets where these are not yet
available (like for the arm64 bootstrap). Having a cross-toolchain just built on
the host architecture (targeting the target architecture), should still be an
option.

Binutils, GCC, clang, and probably more packages in Debian are currently aware
about the multiarch locations, including native and cross compilers. So please
don't hijack this term for your very specific view on a cross compiler.

The so called "cross-toolchain bootstrap" is overly complex and probably should
not be handled by staged builds, but by building from a combined source tree.

  Matthias


--
To UNSUBSCRIBE, email to [hidden email]
with a subject of "unsubscribe". Trouble? Contact [hidden email]
Archive: http://lists.debian.org/50F6BB1F.7060602@...

[Matthias Klose]

Am 16.01.2013 14:50, schrieb Ian Jackson:

>  * We initially define one scope "profile", for build profiles.
>
>    A build profile is an optional variation that can be applied
>    to a particular package, for the purpose of reducing the
>    build dependencies and/or avoiding the building of unneeded
>    binary packages.
>
>    The builder of a package may apply one single build profile.
>
>    Build profile names are a controlled namespace and are allocated by
>    the Debian project.  The following are defined:
>
>       profile:stage1
>       profile:stage2
>             For multi-stage bootstrap.  A package may only
>             declare support for stage2 if it supports stage1 as well.
>             These are not a global ordering across the distribution.
>             Rather, stage1 is the most limited build of this
>             particular package.  stage2 is an intermediate bootstrap
>             build, support for which is provided only if it is
>             necessary to build this particular package three times.

this only takes care about packages with a reduced package set built, or
packages with reduced functionality. There are same cases missing:

 - extra build dependencies for cross builds, like for gcc-4.7:
   {gobjc++,gccgo,gfortran}-<target-gnu-type>
   profile:cross?

 - build dependencies for running the check target. Usually these
   dependencies can be ignored when cross-building packages, or when
   building a package natively with DEB_BUILD_OPTIONS=nocheck.
   profile:check? There are not few packages which can be easier
   cross-built when identifying and dropping these build dependencies.

So it does make sense to build with two profiles like stage1 & check.

  Matthias


--
To UNSUBSCRIBE, email to [hidden email]
with a subject of "unsubscribe". Trouble? Contact [hidden email]
Archive: http://lists.debian.org/50F6C07F.5050805@...

[Neil Williams-4]

On Wed, 16 Jan 2013 13:50:17 +0000
Ian Jackson <[hidden email]> wrote:

> > For even further automation and also for quality assurance, we propose
> > another new field for source packages which indicates whether or not
> > this source package is supposed to be cross compilable.
>
> Is it possible that packages might only cross build for certain
> targets ?  Or only for certain hosts ?

For certain host architectures: yes, definitely. Any source package
which includes assembly (there are more than most people expect) and any
package which then has a build dependency on a binary package
built from the source package(s) containing assembly and so on.
(This is a separate discussion, please don't hijack this
thread for the pros and cons of assembly in packages.)

Also, cross-building of kernels other than Linux and the packages
which build-depend on the headers for such kernels is often not
supported by cross-building toolchains.

These failures are recursive and it can be very hard to identify why
package A wouldn't cross-build until the bootstrapping process shows
that it absolutely requires a package N levels down the dependency
chain which fails to cross build.

--


Neil Williams
=============
http://www.linux.codehelp.co.uk/

[Johannes Schauer]

Hi,

On Wed, Jan 16, 2013 at 01:50:17PM +0000, Ian Jackson wrote:
> > 5. Cross-Builds field
> > =====================
> >
> > For even further automation and also for quality assurance, we propose
> > another new field for source packages which indicates whether or not
> > this source package is supposed to be cross compilable.
>
> Is it possible that packages might only cross build for certain
> targets ?  Or only for certain hosts ?

Yes. But for the intended purpose of this field, it does not make much
sense to encode the information on which architecture and for which
architecture a source package cross compiles.

If a source package would not compile on some architectures or for some
architectures, then it would be Cross-Builds:No.

If the Cross-Builds field is being introduced, then only a few base
source packages *must* carry this field. For all other source packages,
this field would be optional.

The tools we developed can always be used to calculate which source
packages are necessary to cross compile a minimal build system.
Therefor, this field would only be:

 - a more obvious way for the maintainer of a source package to know
   that his/her package is one of those that must be cross compilable
   for any upcoming architecture.

 - a more obvious way for a bootstrapper to see which native build
   dependency cycles could easily be solved by cross compiling some
   source packages

The field is therefor rather low priority compared to the other ideas.

> I haven't spotted anything hideously wrong.  I have three suggestions
> for changes:
>
>  * Packages should explicitly declare which profiles they support.
>    This should be done with a Profile field in the source stanza
>    of debian/control.

Good idea - this would make it even more analogous to the "Architecture"
field and its meaning in source and binary stanzas in ./debian/control.

>  * It should be made explicit in the spec that building occurs with
>  zero or one profile, not several.

As far as I understood (but I'm not the one who actually cross-compiles
things in real life), different dependencies might be needed during
cross compilation of some source packages. If that source package must
be cross compiled for a minimal base system and if it also must be built
with reduced build dependencies, then two build profiles at the same
time are necessary. In this case a "cross" and "stage1" profile.

>  * The concrete syntax in build-depends should not use < > but rather
>  reuse the architecture qualification syntax.

You basically propose to extend the architecture qualification syntax
from a single disjunction into a conjunctive normal form expression. If
any number of disjunctions is allowed, your proposal will also support
more than one profile at the same time.

What would an example of a different namespace than "profile:" be?

>From the perspective of dependency analysis, your proposal seems to be
able to carry all information that is needed for it. I leave it up to
you guys to decide on which you like better.

Thanks for your input!

cheers, josch


--
To UNSUBSCRIBE, email to [hidden email]
with a subject of "unsubscribe". Trouble? Contact [hidden email]
Archive: http://lists.debian.org/20130116160543.GA30841@hoothoot

[Johannes Schauer]

Hi,

On Wed, Jan 16, 2013 at 04:00:15PM +0100, Matthias Klose wrote:

> this only takes care about packages with a reduced package set built,
> or packages with reduced functionality. There are same cases missing:
>
>  - extra build dependencies for cross builds, like for gcc-4.7:
>    {gobjc++,gccgo,gfortran}-<target-gnu-type>
>    profile:cross?
>
>  - build dependencies for running the check target. Usually these
>    dependencies can be ignored when cross-building packages, or when
>    building a package natively with DEB_BUILD_OPTIONS=nocheck.
>    profile:check? There are not few packages which can be easier
>    cross-built when identifying and dropping these build dependencies.
>
> So it does make sense to build with two profiles like stage1 & check.

You probably mean a "nocheck" profile?

Your first example indeed demonstrates why multiple profiles are useful
to be enabled at once.

The second example can be accomplished with only one profile by marking
all dependencies that are not being needed by a "nocheck" profile as not
being needed by the "stage1" profile as well.

So instead of:

  Build-Depends: foo <!stage1>, bar <!nocheck>

and then needing two profiles at the same time, one could have:

  Build-Depends: foo <!stage1>, bar <!nocheck !stage1>

Though I agree that the first option looks more maintainable.

There is also the idea of a "nodocs" profile which would work like
DEB_BUILD_OPTIONS=nodocs.

Whether or not "nocheck" and "nodocs" can/should become build profiles
is of course still to be debated.

An argument for them becoming build profiles is, that the analysis
algorithm would then be able to just choose the less dangerous "nodocs"
or "nocheck" profile instead of a "stage1" profile if they break a
dependency cycle. How safe/dangerous choosing a profile is, could be
evaluated by the number of binary packages which are not built with a
given profile but which are needed as build dependencies by other source
packages. While a "stage1" profile is likely to not build binary
packages which are needed somewhere else, the "nodocs" and "nocheck"
profiles will likely not lead to needed binary packages not being
compiled.

So "nocheck" and "nodocs" profiles could allow "safer" reduced builds.

An argument against "nodocs" might be, that since (most?) *-doc packages
are Architecture:all, the build dependencies that are needed by them to
be built are already/should be included in Build-Depends-Indep anyways.
The current algorithms discard the Build-Depends-Indep field as
Architecture:all binary packages do naturally not need to be built
during the bootstrapping process.

cheers, josch


--
To UNSUBSCRIBE, email to [hidden email]
with a subject of "unsubscribe". Trouble? Contact [hidden email]
Archive: http://lists.debian.org/20130116162652.GB30841@hoothoot

[wookey-4]

+++ Matthias Klose [2013-01-16 15:37 +0100]:

> Am 15.01.2013 19:18, schrieb Johannes Schauer:
> > This mechanism also covers cross-compiler bootstraping. The eglibc, gcc,
> > and kernel packages already have the neceassary staged-build info, but
> > the build profiles (1.) part is also needed to specifiy the reduced
> > build-deps. The cross-toolchain bootstrap ceases to be a special case if
> > treated this way and just becomes packages to be built in stages using
> > the profiles mechansim, like many others in the base system (but for
> > build arch taregtting host, arch, rather than built for host-arch). See
> > the wiki article at [11].
> >
> > [11] http://wiki.debian.org/MultiarchCrossToolchains
>
> Please stop calling this "MultiarchCrossToolchains". It was already wrong when
> used in the Emdebian context, and on some GSOC pages. What you are apparently
> aiming for is a cross compiler re-using the target libraries built (natively).

The target libraries can be built natively or cross-built. Clearly
cross-building is necessary when the target architecture does not yet
exist (e.g. arm64). The point is that the library dependencies use the
multiarch file locations and mechanisms, rather than extra copies in
different locations.

I agree this is separate from multiarch path support in the
cross-toolchain for finding libraries and headers for non-toolchain
libs. What term would you like me to use for this cross-toolchain
arangement? multiarch-libraryCrossToolchains?
PureMultiarchCrossToolchains? FullyMultiarchedCrossToolchains?

Yes it's a bit confusing, because multiarch affects various aspects
of the toolchain. I'm happy to use a different name to avoid
confusion, this just seems to me to be the logical conclusion of
'mulitarching the cross toolchains' (see discussion below), so it
seemed a reasonable name.

> However this is not yet done,

It was done in Thibg's GSOC project for 4.7.2-4. Yes it needs more
work to properly integrate, test, and make bootstrappable, but we have a
working implementation (modulo keeping up with your amazing rate of
new uploads :-). (For which kudos, even if it does make it hard to
keep up!)

> requires dependencies on "foreign" architectures

It does.

> and only is a very special case,

This is where we seem to disagree. I think it should be the standard
case (for binary distro cross-toolchains).

> not helping for targets where these are not yet
> available (like for the arm64 bootstrap). Having a cross-toolchain just built on
> the host architecture (targeting the target architecture), should still be an
> option.

This is still an option - you just cross-build the target libraries as
part of the bootstrap. The wiki page gives the runes.

I am not convinced of the benefit in also maintaining toolchains with
these libraries installed twice in different locations, now that we
have multiarch to make this unnecessary. But I would
like to see much wider discussion of this issue so we can reach
conclusions as a project on what the default cross-toochain setup
should be, and what non-default methods are also worth maintaining in the
packaging.

The main disadvantage of making use of the multiarch libraries is the
need to keep libgcc, libstdc++ etc in version lockstep across
architectures. But this applies to multiarch in general for all
libraries, and is something we manage in Debian as part of being a
distro. It is something to carefully consider. Is the reduced
independence of the cross-toolchain library versions a fatal flaw?

The main advantage is that the code you build against is actually the
code the toolchain was built against, not some other code that happens
to be installed in the expected location. Also build-dependencies work
automatically without having to have 'equivs' packages for libgcc1,
libstc++6-dev etc until the real native ones are built. And having one
copy of libgcc1:<targetarch>, libstdc++:<targetarch> on the system
instead of two seems like good practice to me.

Another advantage is that for normal cross-toolchain builds you don't
need to do a 3-stage bootstrap build, because you already have the
target-arch libc, libgcc1, libstc++ etc available so you just need to
rebuild gcc without staging. Much quicker and simpler. This is
(fundamentally) how the emdebian toolchain builds worked for many
years.

Using multiarch libraries also greatly simplifies the gcc packaging
(if we are able to drop multilib equivalents as a result). Again we
disagree on this point.

I think we should be encouaging people to install arm-linux-gnueabi
and arm-linux-gnueabihf cross-compilers if they want to target both
those arches, not install one or the other, each of which is
multilibbed to output code for both, with a corresponding collection
of libgcc's. I realise that upstream is not yet convinced of the
benefits of this view of the world, but I think we should be using it
and demonstrating its effectiveness (assuming we believe that
multiarch is actually the right way to be dealing with multiple ABIs)

I am happy to be persuaded that this is madness with real examples of
why we can't/shouldn't do things this way (and I can see that
deprecating multilib support where it is well-established and code is
co-runnable (x86_64 and i386) is not going to happen), but I've been
doing it for years for arm and it seems sensible to me.

> Binutils, GCC, clang, and probably more packages in Debian are currently aware
> about the multiarch locations, including native and cross compilers. So please
> don't hijack this term for your very specific view on a cross compiler.

That is using multiarch, as opposed to being multiarched themselves. I
agree we should have terms to distinguish between these concepts to
avoid confusion. How about 'Multiarch-compatible' and 'Pure-multiarch'?

> The so called "cross-toolchain bootstrap" is overly complex and probably should
> not be handled by staged builds, but by building from a combined source tree.

It's a rare case where more than one stage is needed (both libc and
gcc have stage1 and stage2 before the final full build), but there is
no way to avoid this process for a new architecture. The question is
simply how it is implemented.

It can be done just the same as other packages using profile builds,
or it can be special-cased as is currently done for the ubuntu builds.
That works by each of binutils, linux, eglibc and gcc binary builds
producing a -source package, and having a special package
(<targetarch>-cross-toolchain-base) which uses
all of those sources to step through the binutils, linux stage1, gcc
stage1, eglibc stage1, gcc stage2, eglibc stage2, builds, using
dpkg-cross to munge library locations.

This latter works but I disagree with the assertion that it is simpler
than making these packages the same as others that have profile builds:
The normal source packages are used (instead of binary -source
packages), and the build-deps encode the profile/stage builds needed,
and their ordering. Then any tool that can order builds for
bootstrapping can now also do toolchain bootstrapping. Why is that not
a desirable goal?


As you can see if you've made it to the bottom of this long mail,
there are some quite involved questions about how we should be
implementing cross-toolchains in debian. I would love to have wider
input from the project on these questions, and am not sure what the
best forum for that is.

I guess 'this mailing list' for now. With final decision at debconf
once people have got their use-cases and arguments aired?

I have a vision of a 'fully multiarched' cross-toolchain, but the
toolchain maintainer doesn't really share it. Ultimately it's his job not
mine, and perhaps that is the clinching argument, but it would be good
to hear from some other people too.

Wookey
--
Principal hats:  Linaro, Emdebian, Wookware, Balloonboard, ARM
http://wookware.org/


--
To UNSUBSCRIBE, email to [hidden email]
with a subject of "unsubscribe". Trouble? Contact [hidden email]
Archive: http://lists.debian.org/20130116163005.GO5031@...

[Colin Watson]

On Tue, Jan 15, 2013 at 07:18:40PM +0100, Johannes Schauer wrote:
> Besides bootstrapping, these build profiles could also be used for
> embedded builds, and to allow for changed buil-deps when cross-building.

On a related point, see:

  http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=695287

I've been thinking, and talking to Wookey, about how to do this sanely.
The best idea I have is for those packages for which we have -triplet
versions (gcc-x.y et al, pkg-config, and so on) to have an extra control
field called something like "Cross-Tool-Available: yes".  sbuild and
dpkg-checkbuilddeps could then use the presence of this field to know
that they need to translate "Build-Depends: gcc-4.6" into
"Build-Depends: gcc-4.6:native, gcc-4.6-arm-linux-gnueabihf" when
cross-compiling to armhf, say.  Given my recent sbuild patches, it's
very easy for sbuild to do this once we have agreement on how it should
detect when to do so.

This is cleaner than any of the other options I've come up with: it
doesn't require hardcoding a list of "toolchain packages" that have
special cross versions; it would allow us to stop having to shove
pkg-config-HOST into cross-build chroots; and it wouldn't require
dpkg-checkbuilddeps to violate layers by looking in the apt cache, at
least as long as the available file is up to date.  Does it seem sane to
people?

> While this field is meant to make sure not to allow any profile built
> binary package to be uploaded to the archive, it can also be abused to
> only allow "some" build profiles to be uploaded. For example ubuntu
> might generally forbid profile built binary packages to be uploaded
> except for packages built with the "ubuntu" profile only.

I'm not sure we (Ubuntu) would actually want to do this, FWIW.  It
wouldn't buy us much since all our binaries are required to be built on
our own buildds anyway.  That said, the ability to have conditional
build-dependencies for Ubuntu and its descendant dpkg-vendors would be
valuable; but I'm wary of feature creep.

> 5. Cross-Builds field
> =====================
>
> For even further automation and also for quality assurance, we propose
> another new field for source packages which indicates whether or not
> this source package is supposed to be cross compilable.

I don't think we should do this.  Instead, we should have a cross-buildd
that regularly tests whether packages are cross-buildable, and over time
we should expect packages considerably beyond just the base system to be
cross-buildable.  I want to be able to take any package that I might
find on (for the sake of argument ...) an Ubuntu phone and expect that I
can cross-build it cleanly; I don't want cross-building to be this
strange thing that only works for a few exceptional packages, and I
think that having this field sets that expectation.  The more packages
that just work, the easier it will be to do anything related to
cross-building.

This shouldn't be unrealistic, although it's certainly ambitious.
http://people.canonical.com/~cjwatson/cross/armhf/raring/ shows about
35% of Ubuntu main cross-building cleanly right now, and there are lots
of cross-build and multiarch patches sitting in the Debian BTS (I've
forwarded and/or uploaded everything I've done) which would improve the
state of a similar core-ish set of Debian source packages to something
similar.  There are really only a handful of underlying causes
representing many of the remaining failures - and yes, I know these are
fairly hard (perl, python, gobject-introspection, ...) but they aren't
intractable.

> If Debian wants to incorporate the ability to being bootstrappable in its
> policy, then there *must* be some packages which are cross compiled for
> a minimal build system. Adding this header to those source packages
> would make it a bug if they do not actually cross compile. Without this
> header, cross compilation would be wishlist as usual.

This would be better done by way of an external list of the packages
that need to cross-build cleanly.

> Furthermore, cross compilation is one of the methods a porter can use to
> break build dependency cycles. If some packages carry this new field
> then not only could a porter decide quicker whether or not a source
> package is cross compilable, an algorithm could also incorporate this
> information to automatically break build dependency cycles for cross
> compilation.

While this is true, having stared at similar things myself in the past,
I've come to believe that it's a better use of time to just make
everything in sight cross-buildable than to spend lots of effort trying
to algorithmically decide what might work or not.

> If more automated bootstrapping of Debian is desired, then at least build
> profiles (1.) should be introduced.

I am strongly in favour of something occupying the general position of
build profiles; there are several otherwise-intractable cross-building
problems that will require something like them.  I don't have much
interest in the specifics so will refrain from suggesting another colour
for the bikeshed. :-)

> The question remains of how many source packages would have to have the
> proposed new fields added to them to make a full bootstrap from zero
> possible. If the Gentoo USE flags were not too far off and assuming or
> tools do the correct thing so far, then:
>
>  - the number of source packages that has to be modified with the new
>    fields is at maximum 83 (there might be a smaller set).

https://wiki.ubuntu.com/CrossBuilding/BuilddChroot (thanks, Matthias)
suggests that we are not that far from being able to at least get to a
sensible minimal chroot with a modicum of hacking.

--
Colin Watson                                       [[hidden email]]


--
To UNSUBSCRIBE, email to [hidden email]
with a subject of "unsubscribe". Trouble? Contact [hidden email]
Archive: http://lists.debian.org/20130116174130.GA3045@...