Binary blobs in source packages

classic Classic list List threaded Threaded
16 messages Options
Reply | Threaded
Open this post in threaded view
|

Binary blobs in source packages

Thomas Koch-11
Hi,

I just made a fool of myself on the simple-build-tool list by claiming that
Debian would build scala without scala. I only checked debian/rules and
debian/control and since scala is in main, I assumed that I must be right.

However scala comes with a bytecode-compiled scala compiler in lib/ which is
required only during build but not installed in the binary packages.

I already opened reportbug to fill a serious Debian Policy violation, but
actually I couldn't find anything in the policy about it.

So is it ok to ship binaries in the source package that are only required
during build? Can I do the same with simple-build-tool, which requires itself
to build?

Regards,

Thomas Koch, http://www.koch.ro


--
To UNSUBSCRIBE, email to [hidden email]
with a subject of "unsubscribe". Trouble? Contact [hidden email]
Archive: http://lists.debian.org/201112131323.26313.thomas@...

Reply | Threaded
Open this post in threaded view
|

Re: Binary blobs in source packages

Reinhard Tartler-5
On Di, Dez 13, 2011 at 13:23:25 (CET), Thomas Koch wrote:

> Hi,
>
> I just made a fool of myself on the simple-build-tool list by claiming that
> Debian would build scala without scala. I only checked debian/rules and
> debian/control and since scala is in main, I assumed that I must be right.
>
> However scala comes with a bytecode-compiled scala compiler in lib/ which is
> required only during build but not installed in the binary packages.
>
> I already opened reportbug to fill a serious Debian Policy violation, but
> actually I couldn't find anything in the policy about it.
>
> So is it ok to ship binaries in the source package that are only required
> during build? Can I do the same with simple-build-tool, which requires itself
> to build?

I had a similar situation in the aspectc++ package, which is a
source-to-source compiler (from AspectC++ to C++).

Background: AspectC++ requires a library called Puma for parsing C/C++
Code. While Puma itself is written in C++, the Visual C++ and Gnu
Extensions to the C++ language are written with aspects. This means that
you need the 'ac++' compiler to "weave" Puma.

Solution: Upstream provides so-called "woven" sources, which includes a
"pre-woven" Puma library. In the Debian package I use these sources to
compile a "boostrap" ac++ binary, then start over from start, weave Puma
with the "bootstrap" ac++ binary, recompile Puma, throw away the
boostrap ac++ binary, and finally compile ac++.

Additional benefit: I have now tested that the ac++ compiler actually
works on all architectures!

Implementation:
http://bazaar.launchpad.net/~siretart/aspectc++/debian/view/head:/debian/rules
(line 41-56)

I'm pretty sure that this approach can be appleid to both the
'simple-build-tool' and 'scala' packages.

Cheers,
Reinhard

--
Gruesse/greetings,
Reinhard Tartler, KeyID 945348A4


--
To UNSUBSCRIBE, email to [hidden email]
with a subject of "unsubscribe". Trouble? Contact [hidden email]
Archive: http://lists.debian.org/87iplkzm3l.fsf@...

Reply | Threaded
Open this post in threaded view
|

Re: [Pkg-scala-maint] Binary blobs in source packages

Mehdi Dogguy-3
In reply to this post by Thomas Koch-11
On 12/13/2011 01:23 PM, Thomas Koch wrote:
>
> So is it ok to ship binaries in the source package that are only
> required during build? Can I do the same with simple-build-tool,
> which requires itself to build?
>

Depends on the need. It is quite common for compilers to have some
binaries to do the bootstrapping. Scala uses that since some parts of
the compiler are written in scala. And, of course, I make sure that I
ship new binaries only in the Debian package. Another example is OCaml
which needs an ocamlc to bootstrap itself.

I'm not sure yet if it is really justified for simple-build-tool. I was
planning to have a look at it, but didn't find time to actually do that.
IIRC, another issue is that simple-build-tool requires network access
during the build… I guess it can be fooled somehow but didn't check yet.

Regards,

--
Mehdi Dogguy مهدي الدڤي
http://dogguy.org/


--
To UNSUBSCRIBE, email to [hidden email]
with a subject of "unsubscribe". Trouble? Contact [hidden email]
Archive: http://lists.debian.org/4EE7938B.2000609@...

Reply | Threaded
Open this post in threaded view
|

Re: [Pkg-scala-maint] Binary blobs in source packages

Steve Langasek
On Tue, Dec 13, 2011 at 07:03:55PM +0100, Mehdi Dogguy wrote:
> On 12/13/2011 01:23 PM, Thomas Koch wrote:

> > So is it ok to ship binaries in the source package that are only
> > required during build? Can I do the same with simple-build-tool,
> > which requires itself to build?

> Depends on the need. It is quite common for compilers to have some
> binaries to do the bootstrapping. Scala uses that since some parts of
> the compiler are written in scala. And, of course, I make sure that I
> ship new binaries only in the Debian package. Another example is OCaml
> which needs an ocamlc to bootstrap itself.

I think the traditional expectation here is that compilers will do their
initial bootstrap using an out-of-archive binary, and that once in the
archive, they'll be maintained using a self-build-depends instead.

Shipping no-longer-needed bootstrap binaries in the source package is a
borderline case.  It's probably ok if we have the source needed to build
that binary.  But we ought not need to use it for the actual package build.

--
Steve Langasek                   Give me a lever long enough and a Free OS
Debian Developer                   to set it on, and I can move the world.
Ubuntu Developer                                    http://www.debian.org/
[hidden email]                                     [hidden email]

signature.asc (845 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: [Pkg-scala-maint] Binary blobs in source packages

Mehdi Dogguy-3
On 12/13/2011 07:26 PM, Steve Langasek wrote:

> On Tue, Dec 13, 2011 at 07:03:55PM +0100, Mehdi Dogguy wrote:
>> On 12/13/2011 01:23 PM, Thomas Koch wrote:
>
>>> So is it ok to ship binaries in the source package that are only
>>>  required during build? Can I do the same with simple-build-tool,
>>>  which requires itself to build?
>
>> Depends on the need. It is quite common for compilers to have some
>> binaries to do the bootstrapping. Scala uses that since some parts
>> of the compiler are written in scala. And, of course, I make sure
>> that I ship new binaries only in the Debian package. Another
>> example is OCaml which needs an ocamlc to bootstrap itself.
>
> I think the traditional expectation here is that compilers will do
> their initial bootstrap using an out-of-archive binary, and that once
> in the archive, they'll be maintained using a self-build-depends
> instead.
>

You mean having a circular build-dependency? That isn't great :/
I've seen some packages doing that (don't recall which right now) but
didn't like it, tbh.

Regards,

--
Mehdi Dogguy مهدي الدڤي
http://dogguy.org/


--
To UNSUBSCRIBE, email to [hidden email]
with a subject of "unsubscribe". Trouble? Contact [hidden email]
Archive: http://lists.debian.org/4EE79983.2040209@...

Reply | Threaded
Open this post in threaded view
|

Re: [Pkg-scala-maint] Binary blobs in source packages

Steve Langasek
On Tue, Dec 13, 2011 at 07:29:23PM +0100, Mehdi Dogguy wrote:
> > I think the traditional expectation here is that compilers will do
> > their initial bootstrap using an out-of-archive binary, and that once
> > in the archive, they'll be maintained using a self-build-depends
> > instead.

> You mean having a circular build-dependency?

Yes.

> That isn't great :/

<shrug> It's how self-hosting compilers work.  That's how the gcc package
works, too. :)

--
Steve Langasek                   Give me a lever long enough and a Free OS
Debian Developer                   to set it on, and I can move the world.
Ubuntu Developer                                    http://www.debian.org/
[hidden email]                                     [hidden email]

signature.asc (845 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: [Pkg-scala-maint] Binary blobs in source packages

Joachim Breitner
In reply to this post by Mehdi Dogguy-3
Hi,

Am Dienstag, den 13.12.2011, 19:29 +0100 schrieb Mehdi Dogguy:
> You mean having a circular build-dependency? That isn't great :/
> I've seen some packages doing that (don't recall which right now) but
> didn't like it, tbh.

ghc does, for instance.

Greetings,
Joachim

--
Joachim "nomeata" Breitner
Debian Developer
  [hidden email] | ICQ# 74513189 | GPG-Keyid: 4743206C
  JID: [hidden email] | http://people.debian.org/~nomeata

signature.asc (205 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: [Pkg-scala-maint] Binary blobs in source packages

Adam Borowski-3
In reply to this post by Steve Langasek
On Tue, Dec 13, 2011 at 10:45:21AM -0800, Steve Langasek wrote:

> On Tue, Dec 13, 2011 at 07:29:23PM +0100, Mehdi Dogguy wrote:
> > > I think the traditional expectation here is that compilers will do
> > > their initial bootstrap using an out-of-archive binary, and that once
> > > in the archive, they'll be maintained using a self-build-depends
> > > instead.
>
> > You mean having a circular build-dependency?
>
> <shrug> It's how self-hosting compilers work.  That's how the gcc package
> works, too. :)

GCC can be built using any other C compiler, though, so there's no freeness
or security issue (Ken Thompson can explain why this is important).

In some cases (like Smalltalk) this degenerates into a binary quine rather
than providing real source.


Do you remember my joke package "goodbye" a while ago?  One of ideas in the
resulting discussion was to put a quoted ELF object into debian/rules, that
compiles itself from a provided source (so the letter of the Policy and DFSG
is fulfilled), doing some Ken Thompsonese modifications on the way -- hey,
these are allowed in the archive as required by compilers you're talking
about.  Somehow, this idea met the most revulsion, I wonder why.

--
1KB // Yo momma uses IPv4!


--
To UNSUBSCRIBE, email to [hidden email]
with a subject of "unsubscribe". Trouble? Contact [hidden email]
Archive: http://lists.debian.org/20111213193017.GA25968@...

Reply | Threaded
Open this post in threaded view
|

Re: [Pkg-scala-maint] Binary blobs in source packages

Mehdi Dogguy-3
In reply to this post by Steve Langasek
On 12/13/2011 07:26 PM, Steve Langasek wrote:

> On Tue, Dec 13, 2011 at 07:03:55PM +0100, Mehdi Dogguy wrote:
>> On 12/13/2011 01:23 PM, Thomas Koch wrote:
>
>>> So is it ok to ship binaries in the source package that are only
>>> required during build? Can I do the same with simple-build-tool,
>>> which requires itself to build?
>
>> Depends on the need. It is quite common for compilers to have some
>>  binaries to do the bootstrapping. Scala uses that since some
>> parts of the compiler are written in scala. And, of course, I make
>> sure that I ship new binaries only in the Debian package. Another
>> example is OCaml which needs an ocamlc to bootstrap itself.
>
> I think the traditional expectation here is that compilers will do
> their initial bootstrap using an out-of-archive binary, and that
> once in the archive, they'll be maintained using a
> self-build-depends instead.
>

oh, and actually, this is not guaranteed to always work… this is at
least the case with OCaml which performs rather strict checks during
compilation. (This happened already in the past where some core type
changed).

Regards,

--
Mehdi Dogguy مهدي الدڤي
http://dogguy.org/


--
To UNSUBSCRIBE, email to [hidden email]
with a subject of "unsubscribe". Trouble? Contact [hidden email]
Archive: http://lists.debian.org/4EE7ABDB.5020407@...

Reply | Threaded
Open this post in threaded view
|

Re: [Pkg-scala-maint] Binary blobs in source packages

Philipp Kern-4
In reply to this post by Joachim Breitner
On 2011-12-13, Joachim Breitner <[hidden email]> wrote:
> Am Dienstag, den 13.12.2011, 19:29 +0100 schrieb Mehdi Dogguy:
>> You mean having a circular build-dependency? That isn't great :/
>> I've seen some packages doing that (don't recall which right now) but
>> didn't like it, tbh.=20
> ghc does, for instance.

And we had some fun where it was about to disappear from the archive due
to a bad upload.  It would've required manual pseudo-binNMUs on all
architectures to get it back.

But then I don't see how you could avoid circular build-dependencies
with compilers written in their own language.  fpc/fp-compiler does the
same.

Kind regards
Philipp Kern


--
To UNSUBSCRIBE, email to [hidden email]
with a subject of "unsubscribe". Trouble? Contact [hidden email]
Archive: http://lists.debian.org/slrnjefdeu.6s1.trash@...

Reply | Threaded
Open this post in threaded view
|

Re: [Pkg-scala-maint] Binary blobs in source packages

Lars Wirzenius-5
On Tue, Dec 13, 2011 at 08:30:22PM +0000, Philipp Kern wrote:
> But then I don't see how you could avoid circular build-dependencies
> with compilers written in their own language.  fpc/fp-compiler does the
> same.

You can avoid it by having a bootstrap compiler written in another
suitable language, which is capable (even if just) of building the
real compiler. So you write two compilers, A and B. A is written
in, say, AWK or Perl, and compiles the source code for B into
executable code. The executable B is then used to actually compile
software written in the language in question. Indeed, during the
package build, you would build the compiler first with A, and then
again with B, and the binary package would contain just the binary
built with B.

The benefit of the simple bootstrapping compiler A is that it can
often be written quite simply (and, indeed, might not be a real
compiler at all, just an interpreter), and does not need, for example,
to have any optimization phase at all. The only program it ever
needs to compile is B.

Even so, writing A is a fair bit of effort, and many language
implementers don't do that. Which makes things harder for distros,
but nobody cares about the poor distro developer.

--
Freedom-based blog/wiki/web hosting: http://www.branchable.com/

signature.asc (501 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: [Pkg-scala-maint] Binary blobs in source packages

Stéphane Glondu-3
In reply to this post by Philipp Kern-4
Le 13/12/2011 21:30, Philipp Kern a écrit :
> But then I don't see how you could avoid circular build-dependencies
> with compilers written in their own language.  fpc/fp-compiler does the
> same.

OCaml, F# (and Scala, it seems) do that by targetting a bytecode for
which there exists an independent interpreter (resp. OCaml's own
bytecode, .NET, Java). A (cross-platform) binary is then bundled with
the sources and is used for initial bootstrap.

In the OCaml case, a bytecode compiler (ocamlc) is bundled in the
upstream tarball, and the build system recompiles it until it reaches a
fixpoint. There are also native code backends for selected architectures
that can be compiled easily once everything has been compiled in bytecode.

--
Stéphane


--
To UNSUBSCRIBE, email to [hidden email]
with a subject of "unsubscribe". Trouble? Contact [hidden email]
Archive: http://lists.debian.org/4EE7C1EF.8060508@...

Reply | Threaded
Open this post in threaded view
|

Re: [Pkg-scala-maint] Binary blobs in source packages

wookey-4
In reply to this post by Steve Langasek
+++ Steve Langasek [2011-12-13 10:45 -0800]:

> On Tue, Dec 13, 2011 at 07:29:23PM +0100, Mehdi Dogguy wrote:
> > > I think the traditional expectation here is that compilers will do
> > > their initial bootstrap using an out-of-archive binary, and that once
> > > in the archive, they'll be maintained using a self-build-depends
> > > instead.
>
> > You mean having a circular build-dependency?
>
> Yes.
>
> > That isn't great :/
>
> <shrug> It's how self-hosting compilers work.  That's how the gcc package
> works, too. :)

This question is part of the general subject of 'making debian
automatically bootstrapable'.

Lots of languages have this properly of being written in themselves.
However they do generally have some way of getting themselves
bootstrapped.

Languages are generally a degenerate case of circular build-deps with
a very short loop (i.e just on themselves).

My prefered way to deal with the circular dependency is for at least
one package in a loop to have a 'bootstrap-build' supported in the
rules file which does not have a circular dependency (defined by
Build-Depends-Stage1 in control). So a tool which bootstraps debian
from scratch understands to build this 'not the real package'
bootstrap version first, then go on to build the full version once
dependencies are satisfied. The gcc packaging already implements
exactly this, although not using the exact machinery proposed in
http://wiki.debian.org/DebianBootstrap (That doc is out of date,
better info in my debconf talk
http://wookware.org/talks/bootstrappable-Debconf11.tar.gz )

For some packages this might mean that a $language-minimal package is
really needed to avoid the circular build-dep.

I anyone is aware of packages where it really isn't possible to do an
automatic bootstrap without a circular dependency (for the initial
bootstrap build), I would like to know about it.

Wookey
--
Principal hats:  Linaro, Emdebian, Wookware, Balloonboard, ARM
http://wookware.org/


--
To UNSUBSCRIBE, email to [hidden email]
with a subject of "unsubscribe". Trouble? Contact [hidden email]
Archive: http://lists.debian.org/20111214174425.GM28838@...

Reply | Threaded
Open this post in threaded view
|

Re: [Pkg-scala-maint] Binary blobs in source packages

Thomas Goirand-3
On 12/15/2011 01:44 AM, Wookey wrote:
> defined by
> Build-Depends-Stage1 in control
Hi,

I watched the debconf11 video about bootstraping video (yes,
I in Banja Luka, but regrettably didn't attend this one).
My understanding of it was that last summer, there was no

Build-Depends-Stage1

defined in the policy. Has this changed? How does it work?
Where is this documented?

Thomas


--
To UNSUBSCRIBE, email to [hidden email]
with a subject of "unsubscribe". Trouble? Contact [hidden email]
Archive: http://lists.debian.org/4EE97FBB.8050909@...

Reply | Threaded
Open this post in threaded view
|

Re: [Pkg-scala-maint] Binary blobs in source packages

Joachim Breitner
In reply to this post by wookey-4
Hi,

Am Mittwoch, den 14.12.2011, 17:44 +0000 schrieb Wookey:
> I anyone is aware of packages where it really isn't possible to do an
> automatic bootstrap without a circular dependency (for the initial
> bootstrap build), I would like to know about it.

again, GHC comes to mind. When I ported it to s390x, I had to use an old
version (6.4.2), find some 64bit patches that were used by someone to
port it to powerpc64 (https://slyfox.ath.cx/ghc/patches/6.4.2/) and
built it partly on s390, copied the generated C files, and then finished
the build on s390x. Then I used that to build 6.10.1, working around a
bug by manually building one file of the compiler without optimization.
And then I build 7.0.4 (the current version) using that file.

Certainly not something that is worth automating...

If you get the impression that upstream does not care too much about
exotic architectures – then you are right. Unfortunately, that is
nothing that we can change easily.

Greetings,
Joachim

--
Joachim "nomeata" Breitner
Debian Developer
  [hidden email] | ICQ# 74513189 | GPG-Keyid: 4743206C
  JID: [hidden email] | http://people.debian.org/~nomeata

signature.asc (205 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Binary blobs in source packages

Goswin von Brederlow-2
In reply to this post by Thomas Koch-11
Thomas Koch <[hidden email]> writes:

> Hi,
>
> I just made a fool of myself on the simple-build-tool list by claiming that
> Debian would build scala without scala. I only checked debian/rules and
> debian/control and since scala is in main, I assumed that I must be right.
>
> However scala comes with a bytecode-compiled scala compiler in lib/ which is
> required only during build but not installed in the binary packages.
>
> I already opened reportbug to fill a serious Debian Policy violation, but
> actually I couldn't find anything in the policy about it.
>
> So is it ok to ship binaries in the source package that are only required
> during build? Can I do the same with simple-build-tool, which requires itself
> to build?
>
> Regards,
>
> Thomas Koch, http://www.koch.ro

What you could do is use the existing previous version of scala to
build the new one and thereby avoid any precompiled binaries.

That has the obvious disadvantage that if the previous scala is broken
you need to manually build and upload a fixed one.


I don't think though that there is anything legally or morally wrong
with shipping and using a binary blob for bootstrapping where
neccessary. Important is that you include the sources for the blob and
that you can build the blob from scratch, which means you testing that
regulary before uploads.

A nice test is to build the blob and package repeadetly and check if the
result remains the same. Build compiler1 with upstream blob, build blob1
with compiler1, build compiler2 with blob1, build blob2 with compiler2.
Now blob1 and blob2 should be identical (other than timestamps and
such).

MfG
        Goswin


--
To UNSUBSCRIBE, email to [hidden email]
with a subject of "unsubscribe". Trouble? Contact [hidden email]
Archive: http://lists.debian.org/87d3bo6d8f.fsf@...