Re: Enabling -ffile-prefix-map by default

classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

Re: Enabling -ffile-prefix-map by default

Vagrant Cascadian-9
On 2020-06-03, Benjamin Barenblat wrote:
> About two years ago, Guillem Jover added support for -ffile-prefix-map
> to dpkg [1] (thanks!).

Wow, hard to believe that was nearly two years ago!

Thanks for bringing this up!


> Related discussion [2] concluded that the option
> should be disabled by default and enabled after Reproducible Builds had
> trialed it for a while. Having enabled it in some of my recent
> packaging, I’m curious whether it may be time to switch this flag to be
> enabled by default. What do you think?

It helps maybe hundreds of packages, and we've identified a small (16 at
the moment) number packages for which it causes issues:

  https://tests.reproducible-builds.org/debian/issues/unstable/ftbfs_due_to_f-file-prefix-map_issue.html

But most of those packages still FTBFS on bullseye and buster where we
don't set the flag (or vary the build path)...

There may also be others that fail but haven't been identified.

I recall this thread where it broke test suites of some packages:

  https://alioth-lists.debian.net/pipermail/reproducible-builds/Week-of-Mon-20190204/011101.html


I believe maintainers can override this flag in their packages if
needed? If so, I'd be inclined to explore setting it by default!


live well,
  vagrant

signature.asc (233 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Enabling -ffile-prefix-map by default

Chris Lamb -2
Vagrant Cascadian wrote:

> I believe maintainers can override this flag in their packages if
> needed? If so, I'd be inclined to explore setting it by default!

I would be very much in favour of enabling this at the earliest
opportunity.

Regarding maintainer control over this I have two remarks to make. To
answer Vagrant's question of whether they can override this (ie.
disable it in the case of it becoming a default) then unless I am
missing something that would be possible via the usual dpkg-buildflags
mechanism.

Secondly (and related to the previous remark as well as being another
reason to enable this soon) many maintainers are explicitly enabling
the fixfilepath feature area of dpkg-buildflags right now as we
incentivise making their package appear reproducible on the
Reproducible Build's testing framework.

My point here is that if were to enable this flag anyway, doing it
before yet more packages add this boilerplate would avoid even more
cluttering of debian/rules files. (Many packages are still parsing
debian/changelog and manually exporting SOURCE_DATE_EPOCH.)


Regards,

--
      ,''`.
     : :'  :     Chris Lamb
     `. `'`      [hidden email] 🍥 chris-lamb.co.uk
       `-

Reply | Threaded
Open this post in threaded view
|

Re: Enabling -ffile-prefix-map by default

Guillem Jover
Hi!

On Wed, 2020-06-03 at 11:59:32 -0700, Vagrant Cascadian wrote:

> On 2020-06-03, Benjamin Barenblat wrote:
> > Related discussion [2] concluded that the option
> > should be disabled by default and enabled after Reproducible Builds had
> > trialed it for a while. Having enabled it in some of my recent
> > packaging, I’m curious whether it may be time to switch this flag to be
> > enabled by default. What do you think?
>
> It helps maybe hundreds of packages, and we've identified a small (16 at
> the moment) number packages for which it causes issues:
>
>   https://tests.reproducible-builds.org/debian/issues/unstable/ftbfs_due_to_f-file-prefix-map_issue.html

Have these got bugs filed?

> But most of those packages still FTBFS on bullseye and buster where we
> don't set the flag (or vary the build path)...

Ah OK, so it does not look bad at all. :)

> There may also be others that fail but haven't been identified.
>
> I recall this thread where it broke test suites of some packages:
>
>   https://alioth-lists.debian.net/pipermail/reproducible-builds/Week-of-Mon-20190204/011101.html

But I assume this would have been detected while building?

On Thu, 2020-06-04 at 22:58:05 -0000, Chris Lamb wrote:
> Vagrant Cascadian wrote:
> > I believe maintainers can override this flag in their packages if
> > needed? If so, I'd be inclined to explore setting it by default!

As I mentioned last time, I've got no concerns with enabling this. I
think we should just follow the usual steps described at:

  <https://wiki.debian.org/Teams/Dpkg/FAQ#Q:_Can_we_add_support_for_new_default_build_flags_to_dpkg-buildflags.3F>

Most of which have already been done now in the repro infra, I think? So
it'd just mostly be a matter of someone sending a note to d-d proposing
the change, providing the numbers, waiting a bit in case someone has
concerns, then I'm happy to flip the default. :)

> Regarding maintainer control over this I have two remarks to make. To
> answer Vagrant's question of whether they can override this (ie.
> disable it in the case of it becoming a default) then unless I am
> missing something that would be possible via the usual dpkg-buildflags
> mechanism.

Indeed.

> Secondly (and related to the previous remark as well as being another
> reason to enable this soon) many maintainers are explicitly enabling
> the fixfilepath feature area of dpkg-buildflags right now as we
> incentivise making their package appear reproducible on the
> Reproducible Build's testing framework.
>
> My point here is that if were to enable this flag anyway, doing it
> before yet more packages add this boilerplate would avoid even more
> cluttering of debian/rules files. (Many packages are still parsing
> debian/changelog and manually exporting SOURCE_DATE_EPOCH.)

Yes, adding this to packages to then remove when the default change,
does not seem like effort well spent, so let's get this moving. :)

Thanks,
Guillem

Reply | Threaded
Open this post in threaded view
|

Re: Enabling -ffile-prefix-map by default

Chris Lamb -2
Hi Guillem,

> > I recall this thread where it broke test suites of some packages:
> >
> >   https://alioth-lists.debian.net/pipermail/reproducible-builds/Week-of-Mon-20190204/011101.html
>
> But I assume this would have been detected while building?

It would have been detected during build, yes. (It is unlikely,
although not technically impossible, that the adjustment of __FILE__
can affect the runtime behaviour of a package yet not at build time.)

> Most of which have already been done now in the repro infra, I think? So
> it'd just mostly be a matter of someone sending a note to d-d proposing
> the change, providing the numbers, waiting a bit in case someone has
> concerns, then I'm happy to flip the default. :)

Vagrant, could you go ahead with this?


Regards,

--
      ,''`.
     : :'  :     Chris Lamb
     `. `'`      [hidden email] 🍥 chris-lamb.co.uk
       `-

Reply | Threaded
Open this post in threaded view
|

Re: Enabling -ffile-prefix-map by default

Vagrant Cascadian-9
In reply to this post by Vagrant Cascadian-9
On 2020-06-03, Vagrant Cascadian wrote:
> On 2020-06-03, Benjamin Barenblat wrote:
>> Related discussion [2] concluded that the option
>> should be disabled by default and enabled after Reproducible Builds had
>> trialed it for a while. Having enabled it in some of my recent
>> packaging, I’m curious whether it may be time to switch this flag to be
>> enabled by default. What do you think?
>
> It helps maybe hundreds of packages

I can't prove it isn't fixed in some other way, but my guess is this is
in the ballpark of 500-700, based on the currently reproducible builds
from packages marked with this issue:

  https://tests.reproducible-builds.org/debian/issues/unstable/gcc_captures_build_path_issue.html


> and we've identified a small (16 at the moment) number packages for
> which it causes issues:
>
>   https://tests.reproducible-builds.org/debian/issues/unstable/ftbfs_due_to_f-file-prefix-map_issue.html
>
> But most of those packages still FTBFS on bullseye and buster where we
> don't set the flag (or vary the build path)...

On closer look, I was mistaken; tests.reproducible-builds.org sets
reproducible=+all on all distributions... even though we only vary the
build path in unstable and experimental.

We'd be able to get more accurate data if we remove the flag for
bullseye and manually schedule all of those packages, at least
temporarily... anyone have concerns with doing that and waiting for the
builds to complete before bringing this up on debian-devel?


> There may also be others that fail but haven't been identified.
>
> I recall this thread where it broke test suites of some packages:
>
>   https://alioth-lists.debian.net/pipermail/reproducible-builds/Week-of-Mon-20190204/011101.html

Many of the identified ones are named k*, so probably directly
related...


> I believe maintainers can override this flag in their packages if
> needed? If so, I'd be inclined to explore setting it by default!

Overriding the flags in 16 packages doesn't sound like a *huge* effort
to me, although though from a quick glance, many of them from KDE
related teams:

  https://udd.debian.org/dmd/?debian-qt-kde%40lists.debian.org
  https://udd.debian.org/dmd/?pkg-kde-extras%40lists.alioth.debian.org


live well,
  vagrant

signature.asc (233 bytes) Download Attachment