Bad interaction between pbuilder/debhelper/dpkg-buildinfo/dpkg-genchanges and dak on security-master

classic Classic list List threaded Threaded
14 messages Options
Reply | Threaded
Open this post in threaded view
|

Bad interaction between pbuilder/debhelper/dpkg-buildinfo/dpkg-genchanges and dak on security-master

Yves-Alexis Perez-2
Hi,

I recently had a problem with an upload to the archive, likely due to bad
interaction between the tooling I use to build packages and the archive
manager.

I usually build my packages using pbuilder, with SOURCE_ONLY_CHANGES=yes in
.pbuilderrc, so pbuilder will ask to generate a _source.changes along with the
_<arch>.changes files.

When uploading to the archive, I do a complete (meaning arch-any + arch-all)
build to be sure, then upload only the _source.changes so everything gets
rebuilt by the autobuilders. It worked just fine for every upload I did to sid
and experimental.

However, I recently did that for an upload targeted at stretch-security, and
unfortunately this caused a problem on security-master, where dak couldn't
process the build by the amd64 autobuilder because an _amd64.buildinfo file
was already present. It was part of my upload because it was included in the
_source.changes file generated during the pbuilder run.

I'm not completely sure what needs fixing here, but:

- when doing a complete build, something will generate a _<arch>.buildinfo
file (using dpkg-genbuildinfo), maybe its part of debhelper, I'm not sure what
control I, dpkg or pbuilder have on this;
- nothing generates a _source.buildinfo file;
- when pbuilder generates the source changes file, it calls dpkg-genchanges -S
after the build, which apparently includes any buildinfo file unconditionnaly
(http://sources.debian.net/src/dpkg/1.18.24/scripts/dpkg-genchanges.pl/#L310)

So few questions:

- would it make sense to have a _source.buildinfo when building a package?
- would it make sense to not include _<arch>.buildinfo when generating a
_source.changes?
- should the archive accept a _source.changes file with arch specific stuff
inside (it's does currently, although the security archive failed later)?

I yet didn't file any bug (to pbuidler, debhelper or dpkg-dev) because I'm
really unsure where the problem(s) lies.

Any help appreciated here. Please keep on CC, I'm not subscribed to debian-
devel anymore.

Regards,
--
Yves-Alexis

signature.asc (499 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Bad interaction between pbuilder/debhelper/dpkg-buildinfo/dpkg-genchanges and dak on security-master

Yves-Alexis Perez-2
Sorry for the fail on FTP-masters email address (which also got the mail
bounced from alioth). Replying to keep threading consistent but quoting the
whole mail below).

On Mon, 2017-07-03 at 14:49 +0200, Yves-Alexis Perez wrote:

> Hi,
>
> I recently had a problem with an upload to the archive, likely due to bad
> interaction between the tooling I use to build packages and the archive
> manager.
>
> I usually build my packages using pbuilder, with SOURCE_ONLY_CHANGES=yes in
> .pbuilderrc, so pbuilder will ask to generate a _source.changes along with
> the
> _<arch>.changes files.
>
> When uploading to the archive, I do a complete (meaning arch-any + arch-all)
> build to be sure, then upload only the _source.changes so everything gets
> rebuilt by the autobuilders. It worked just fine for every upload I did to
> sid
> and experimental.
>
> However, I recently did that for an upload targeted at stretch-security, and
> unfortunately this caused a problem on security-master, where dak couldn't
> process the build by the amd64 autobuilder because an _amd64.buildinfo file
> was already present. It was part of my upload because it was included in the
> _source.changes file generated during the pbuilder run.
>
> I'm not completely sure what needs fixing here, but:
>
> - when doing a complete build, something will generate a _<arch>.buildinfo
> file (using dpkg-genbuildinfo), maybe its part of debhelper, I'm not sure
> what
> control I, dpkg or pbuilder have on this;
> - nothing generates a _source.buildinfo file;
> - when pbuilder generates the source changes file, it calls dpkg-genchanges
> -S
> after the build, which apparently includes any buildinfo file
> unconditionnaly
> (http://sources.debian.net/src/dpkg/1.18.24/scripts/dpkg-genchanges.pl/#L310
> )
>
> So few questions:
>
> - would it make sense to have a _source.buildinfo when building a package?
> - would it make sense to not include _<arch>.buildinfo when generating a
> _source.changes?
> - should the archive accept a _source.changes file with arch specific stuff
> inside (it's does currently, although the security archive failed later)?
>
> I yet didn't file any bug (to pbuidler, debhelper or dpkg-dev) because I'm
> really unsure where the problem(s) lies.
>
> Any help appreciated here. Please keep on CC, I'm not subscribed to debian-
> devel anymore.
>
> Regards,
--
Yves-Alexis

signature.asc (499 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Bad interaction between pbuilder/debhelper/dpkg-buildinfo/dpkg-genchanges and dak on security-master

Ian Jackson-2
In reply to this post by Yves-Alexis Perez-2
Yves-Alexis Perez writes ("Bad interaction between pbuilder/debhelper/dpkg-buildinfo/dpkg-genchanges and dak on security-master"):
> However, I recently did that for an upload targeted at stretch-security, and
> unfortunately this caused a problem on security-master, where dak couldn't
> process the build by the amd64 autobuilder because an _amd64.buildinfo file
> was already present. It was part of my upload because it was included in the
> _source.changes file generated during the pbuilder run.

We had a related conversation on debian-dpkg recently.

There were some reasons discussed why publishing the uploader's
_ARCH.buildinfo, even of a source-only upload, may be useful to some
people.

However, given that the autobuilder's _ARCH.buildinfo actually
corresponds to the published binaries, that clearly must take
precedence.

So I think publishing the uploader's .buildinfo for
built-but-unpublished binaries requires being able to store and
reproduce multiple .buildinfo files for a single ARCH.

> So few questions:
>
> - would it make sense to have a _source.buildinfo when building a package?

Separately, we discussed the nature of _source.buildinfo.  Currently
dpkg-genchanges will generate a _source.buildinfo if asked to make a
source-only .changes file.  I think this is usually wrong.

(And the fact that build-dependencies might affect the generated
_source package_ is a bug in our source code management systems.)

Perhaps we need to invent the concept of _source+ARCH.buildinfo or
something.

Ian.

Reply | Threaded
Open this post in threaded view
|

Re: Bad interaction between pbuilder/debhelper/dpkg-buildinfo/dpkg-genchanges and dak on security-master

Philipp Kern-6
[ Correcting ftp-master's email address, but keeping the large list of
recipients for some reason. ]

On 2017-07-03 16:00, Ian Jackson wrote:

> Yves-Alexis Perez writes ("Bad interaction between
> pbuilder/debhelper/dpkg-buildinfo/dpkg-genchanges and dak on
> security-master"):
>> However, I recently did that for an upload targeted at
>> stretch-security, and
>> unfortunately this caused a problem on security-master, where dak
>> couldn't
>> process the build by the amd64 autobuilder because an _amd64.buildinfo
>> file
>> was already present. It was part of my upload because it was included
>> in the
>> _source.changes file generated during the pbuilder run.
> We had a related conversation on debian-dpkg recently.
>
> There were some reasons discussed why publishing the uploader's
> _ARCH.buildinfo, even of a source-only upload, may be useful to some
> people.
>
> However, given that the autobuilder's _ARCH.buildinfo actually
> corresponds to the published binaries, that clearly must take
> precedence.

Is the buildinfo actually published today? I don't see it in the pool.
As I would've had some use for them at work I was sort of curious if
they could be ingested automatically.

Kind regards and thanks
Philipp Kern

Reply | Threaded
Open this post in threaded view
|

Re: Bad interaction between pbuilder/debhelper/dpkg-buildinfo/dpkg-genchanges and dak on security-master

Mattia Rizzolo-5
On Mon, Jul 03, 2017 at 07:00:20PM +0200, Philipp Kern wrote:
> [ Correcting ftp-master's email address, but keeping the large list of
> recipients for some reason. ]

really…  that's just a ftp-master issue IMHO, definitely not due to
debhelper much less by pbuilder…

> Is the buildinfo actually published today? I don't see it in the pool. As I
> would've had some use for them at work I was sort of curious if they could
> be ingested automatically.

That's kind of OT, but:

Not yet.  We people from the reproducible team couldn't find a way to
usefully talk to ftp-masters people, whom never replied to any of the
questions in the thread at #763822 (they only did some quick comments on
IRC, and we have been left on guessing what they would like…).

Anyhow, .buildinfo files are stored in ftp-master, just not exported to
the mirrors, you can find them in
coccia.debian.org:/srv/ftp-master.debian.org/<something>.

--
regards,
                        Mattia Rizzolo

GPG Key: 66AE 2B4A FCCF 3F52 DA18  4D18 4B04 3FCD B944 4540      .''`.
more about me:  https://mapreri.org                             : :'  :
Launchpad user: https://launchpad.net/~mapreri                  `. `'`
Debian QA page: https://qa.debian.org/developer.php?login=mattia  `-

signature.asc (849 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Bad interaction between pbuilder/debhelper/dpkg-buildinfo/dpkg-genchanges and dak on security-master

Philipp Kern-6
On 07/03/2017 07:06 PM, Mattia Rizzolo wrote:
> On Mon, Jul 03, 2017 at 07:00:20PM +0200, Philipp Kern wrote:
>> [ Correcting ftp-master's email address, but keeping the large list of
>> recipients for some reason. ]
>
> really…  that's just a ftp-master issue IMHO, definitely not due to
> debhelper much less by pbuilder…

Pruning accordingly (I'm not sure why the mails were sent there in the
first place).

>> Is the buildinfo actually published today? I don't see it in the pool. As I
>> would've had some use for them at work I was sort of curious if they could
>> be ingested automatically.
>
> That's kind of OT, but:
>
> Not yet.  We people from the reproducible team couldn't find a way to
> usefully talk to ftp-masters people, whom never replied to any of the
> questions in the thread at #763822 (they only did some quick comments on
> IRC, and we have been left on guessing what they would like…).
>
> Anyhow, .buildinfo files are stored in ftp-master, just not exported to
> the mirrors, you can find them in
> coccia.debian.org:/srv/ftp-master.debian.org/<something>.
So I suppose we talk about 13 GB[1] of static content in about 1.7M
files. Is that something that could be distributed through
static.debian.org if there are concerns around inodes for the main
mirrors? Given that they would be accessed mostly rarely[2]?

Kind regards
Philipp Kern

[1] 7.7kB (75%ile as mentioned in the referenced bug) * 55000 binary
packages * 10 architectures * 3 versions - so quite conservatively
[2] So supposedly a CDN wouldn't bring a lot of benefit as individual
files aren't likely to be hit frequently.


signature.asc (523 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Bad interaction between pbuilder/debhelper/dpkg-buildinfo/dpkg-genchanges and dak on security-master

Yves-Alexis Perez-2
In reply to this post by Mattia Rizzolo-5
On Mon, 2017-07-03 at 19:06 +0200, Mattia Rizzolo wrote:
> On Mon, Jul 03, 2017 at 07:00:20PM +0200, Philipp Kern wrote:
> > [ Correcting ftp-master's email address, but keeping the large list of
> > recipients for some reason. ]
>
> really…  that's just a ftp-master issue IMHO, definitely not due to
> debhelper much less by pbuilder…

I fine with that answer. My point was just that, should ftpmasters say that it
was unsupported to have an _<arch>.buildinfo file inside a _source.changes,
then something was wrong in the build toolchain.

Regards,
--
Yves-Alexis

signature.asc (499 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Bad interaction between pbuilder/debhelper/dpkg-buildinfo/dpkg-genchanges and dak on security-master

Yves-Alexis Perez-2
On Mon, 2017-07-03 at 20:49 +0200, Yves-Alexis Perez wrote:

> On Mon, 2017-07-03 at 19:06 +0200, Mattia Rizzolo wrote:
> > On Mon, Jul 03, 2017 at 07:00:20PM +0200, Philipp Kern wrote:
> > > [ Correcting ftp-master's email address, but keeping the large list of
> > > recipients for some reason. ]
> >
> > really…  that's just a ftp-master issue IMHO, definitely not due to
> > debhelper much less by pbuilder…
>
> I fine with that answer. My point was just that, should ftpmasters say that it
> was unsupported to have an _<arch>.buildinfo file inside a _source.changes,
> then something was wrong in the build toolchain.
I hope that ftpmasters will reply here, but just in case:

[08:06:05] (ansgar): Corsac: I fixed it by changing the filename and resigning
the buildd's changes.  But please don't upload _amd64.buildinfo unless you include amd64 binaries.

I didn't manually generate this _sources.changes and I didn't ask to include
_amd64.buildinfo. So something does need fixing in the build chain, whether
it's pbuilder, debhelper or dpkg-dev or a combination thereof.

Regards,
--
Yves-Alexis

signature.asc (499 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Bad interaction between pbuilder/debhelper/dpkg-buildinfo/dpkg-genchanges and dak on security-master

James Clarke-2
On 9 Jul 2017, at 15:19, Yves-Alexis Perez <[hidden email]> wrote:

> On Mon, 2017-07-03 at 20:49 +0200, Yves-Alexis Perez wrote:
>> On Mon, 2017-07-03 at 19:06 +0200, Mattia Rizzolo wrote:
>>> On Mon, Jul 03, 2017 at 07:00:20PM +0200, Philipp Kern wrote:
>>>> [ Correcting ftp-master's email address, but keeping the large list of
>>>> recipients for some reason. ]
>>>
>>> really…  that's just a ftp-master issue IMHO, definitely not due to
>>> debhelper much less by pbuilder…
>>
>> I fine with that answer. My point was just that, should ftpmasters say that it
>> was unsupported to have an _<arch>.buildinfo file inside a _source.changes,
>> then something was wrong in the build toolchain.
>
> I hope that ftpmasters will reply here, but just in case:
>
> [08:06:05] (ansgar): Corsac: I fixed it by changing the filename and resigning
> the buildd's changes.  But please don't upload _amd64.buildinfo unless you include amd64 binaries.
>
> I didn't manually generate this _sources.changes and I didn't ask to include
> _amd64.buildinfo. So something does need fixing in the build chain, whether
> it's pbuilder, debhelper or dpkg-dev or a combination thereof.

Having the _amd64.buildinfo included in a _source.changes created by
dpkg-genchanges -S in a tree which has done a source+binary build is an
intended feature. You've done the build, so by uploading the _amd64.buildinfo
you are announcing that you were able to produce those build results in the
specified environment, and in theory it allows anyone to compare the buildd's
results to what you claim to have been able to build, without you ever having
to upload the binaries (yes, throwing away binary uploads would allow you to do
this, but *you would still want to upload and keep the _amd64.buildinfo
otherwise you have nothing to compare against and you might as well have just
done a source-only upload*). Now, the issue here is not its presence, but its
name; however, I'd argue this is the correct name for it; it *is* a buildinfo
file for an amd64 build. Uploading a source-only changes file called
_amd64.changes has been done many times in the past (and used to be what you
would get with pbuilder pre-stretch) and never posed an issue, I guess because
the .changes files were thrown away(?), though I seem to recall in some cases
there were issues? Anyway, I don't especially care whether the _amd64.buildinfo
gets renamed (copied) by dpkg-genchanges -S, or whether dak is fixed to allow
multiple buildinfo files for the same arch (maybe renaming the file itself);
perhaps in either case it could include part of one of its hashes in its name,
so you can still find it solely from the data in changes file? The file names
used to have something like that; what happened to it? I guess debsign gets in
the way of that, as it signs the file inline rather than with a detached
signature, changing the hash and thus the name the file should have...

Regards,
James

Reply | Threaded
Open this post in threaded view
|

Re: Bad interaction between pbuilder/debhelper/dpkg-buildinfo/dpkg-genchanges and dak on security-master

Yves-Alexis Perez-2
On Sun, 2017-07-09 at 15:41 +0100, James Clarke wrote:
>  You've done the build, so by uploading the _amd64.buildinfo
> you are announcing that you were able to produce those build results in the
> specified environment, and in theory it allows anyone to compare the buildd's
> results to what you claim to have been able to build, without you ever having
> to upload the binaries (yes, throwing away binary uploads would allow you to do
> this, but *you would still want to upload and keep the _amd64.buildinfo
> otherwise you have nothing to compare against and you might as well have just
> done a source-only upload*).

Actually, I've done the build just to be sure what I'm uploading does build.
But I'm doing a source-only upload, I'm uploading only the _sources.changes
that pbuilder requests generating (with SOURCE_ONLY_CHANGES=yes in
.pbuilderrc), but the build does produce and _amd64.changes too (which I don't
touch).

Regards,
--
Yves-Alexis

signature.asc (499 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Bad interaction between pbuilder/debhelper/dpkg-buildinfo/dpkg-genchanges and dak on security-master

Thorsten Glaser
In reply to this post by James Clarke-2
James Clarke dixit:

>file for an amd64 build. Uploading a source-only changes file called
>_amd64.changes has been done many times in the past (and used to be what you
>would get with pbuilder pre-stretch) and never posed an issue, I guess because
>the .changes files were thrown away(?), though I seem to recall in some cases

Both wrong, the buildd uploads were REJECTed as the _arch.changes
already existed in the archive.

>>> I fine with that answer. My point was just that, should ftpmasters say that it
>>> was unsupported to have an _<arch>.buildinfo file inside a _source.changes,
>>> then something was wrong in the build toolchain.

>> [08:06:05] (ansgar): Corsac: I fixed it by changing the filename and resigning
>> the buildd's changes.  But please don't upload _amd64.buildinfo unless you include amd64 binaries.
>> it's pbuilder, debhelper or dpkg-dev or a combination thereof.

I just drop the buildinfo file from the _source.changes manually.

I believe there’s an unclearness about what exactly a buildinfo file
is and what exactly the arch qualifier in its name stands for, while
there *is* a clear definition that each filename may occur in the
archive at most once.

I think it’s best that the persons responsible for creating those
buildinfo files comment on the scope of then, after which it should
be decided whether

1) *_source.changes should never list a buildinfo file, or
2) *_source.changes may list a *_source.buildinfo file, or
3) something else that does not work with the way the archive works.

bye,
//mirabilos
--
15:39⎜«mika:#grml» mira|AO: "mit XFree86® wär’ das nicht passiert" - muhaha
15:48⎜<thkoehler:#grml> also warum machen die xorg Jungs eigentlich alles
kaputt? :)    15:49⎜<novoid:#grml> thkoehler: weil sie als Kinder nie den
gebauten Turm selber umschmeissen durften? -- ~/.Xmodmap wonders…

Reply | Threaded
Open this post in threaded view
|

Re: Bad interaction between pbuilder/debhelper/dpkg-buildinfo/dpkg-genchanges and dak on security-master

Ian Jackson-2
In reply to this post by Yves-Alexis Perez-2
James Clarke writes ("Re: Bad interaction between pbuilder/debhelper/dpkg-buildinfo/dpkg-genchanges and dak on security-master"):
> Having the _amd64.buildinfo included in a _source.changes created by
> dpkg-genchanges -S in a tree which has done a source+binary build is an
> intended feature.

Wait, what ?  You're telling me that dpkg-genchanges will pick up
random .buildinfo files it happens to find lying around ?  And that
this is deliberate ?

There is no way the tools can tell whether the _amd64.buildinfo it
finds even relates to the same source code, or anything.

Ian.

Reply | Threaded
Open this post in threaded view
|

Re: Bad interaction between pbuilder/debhelper/dpkg-buildinfo/dpkg-genchanges and dak on security-master

James Clarke-2
On 12 Jul 2017, at 10:30, Ian Jackson <[hidden email]> wrote:
> James Clarke writes ("Re: Bad interaction between pbuilder/debhelper/dpkg-buildinfo/dpkg-genchanges and dak on security-master"):
>> Having the _amd64.buildinfo included in a _source.changes created by
>> dpkg-genchanges -S in a tree which has done a source+binary build is an
>> intended feature.
>
> Wait, what ?  You're telling me that dpkg-genchanges will pick up
> random .buildinfo files it happens to find lying around ?  And that
> this is deliberate ?

No, only files mentioned in debian/files, which will be emptied for a
subsequent build.

> There is no way the tools can tell whether the _amd64.buildinfo it
> finds even relates to the same source code, or anything.

Well, the same is true for the .dsc if you modify the source and then do a
dpkg-genchanges :)

James

Reply | Threaded
Open this post in threaded view
|

Re: Bad interaction between pbuilder/debhelper/dpkg-buildinfo/dpkg-genchanges and dak on security-master

Ian Jackson-2
James Clarke writes ("Re: Bad interaction between pbuilder/debhelper/dpkg-buildinfo/dpkg-genchanges and dak on security-master"):
> On 12 Jul 2017, at 10:30, Ian Jackson <[hidden email]> wrote:
> > Wait, what ?  You're telling me that dpkg-genchanges will pick up
> > random .buildinfo files it happens to find lying around ?  And that
> > this is deliberate ?
>
> No, only files mentioned in debian/files, which will be emptied for a
> subsequent build.

That makes more sense.  So this oddness with different (and
partially-overlapping) changes files is something pbuilder has done,
perhaps accidentally.

I find myself agreeing with Thorsten:

   I think it's best that the persons responsible for creating those
   buildinfo files comment on the scope of then, after which it should
   be decided whether

   1) *_source.changes should never list a buildinfo file, or
   2) *_source.changes may list a *_source.buildinfo file, or
   3) something else that does not work with the way the archive works.

The .buildinfo files were invented as part of the reproducible builds
project, but the decision to include a .buildinfo which talks about
binary packages, even in a source-only upload, was taken by the dpkg
maintainer.

My view from Thorsten's options is (1).

Ian.