Re: Bug#926242: jenkins.debian.org: Please test reproducibility status of Debian Installer images

classic Classic list List threaded Threaded
31 messages Options
12
Reply | Threaded
Open this post in threaded view
|

Re: Bug#926242: jenkins.debian.org: Please test reproducibility status of Debian Installer images

Chris Lamb -2
Cyril Brulebois wrote:

> I'm a little wary with possibly merging this late in the release cycle,
> so I'd rather get see that looked at after Buster is out.

I naturally understand your hesitation but on the other hand I would
truly love to see this in buster. Indeed, we may actually have done
enough work to boast about having reproducible installer images for
the upcoming release (!) although without testing on our more-
comprehensive testing framework it is difficult to tell at the
moment...

Devil's advocate: this is surely unlikely to break the release of
buster itself? I mean, for the "final" official buster builds, that is?

> With extra apologies since I've just broken the context of your patch
> by removing the volatile references

No problem, updated patch is:

commit c27a34ba97e028c5b57a35470f6ecb82ad1d9ffb
Author: Chris Lamb <[hidden email]>
Date:   Wed Jun 5 21:20:34 2019 +0100

    Include arguments from sources.list(5) such as [check-valid-until=no], etc. (Re: #926242)
   
    We print three columns if we have "[options]", otherwise we just print
    two as before.

diff --git a/build/util/gen-sources.list.udeb b/build/util/gen-sources.list.udeb
index 7fa40ac5a..f8415fb05 100755
--- a/build/util/gen-sources.list.udeb
+++ b/build/util/gen-sources.list.udeb
@@ -36,10 +36,9 @@ get_mirrors() {
  [ -s $file ] || continue
  grep '^deb[[:space:]]' $file | \
    grep -v '^deb[[:space:]]\+cdrom:' | \
-   sed 's,^deb \[[^]]*\] ,deb ,' | \
    grep -v 'security.debian.org' | \
    grep '[[:space:]]main' | \
-   awk '{print $1 " " $2}' | \
+   awk '{ print (substr($2, 0, 1) == "[") ? $1 " " $2 " " $3 : $1 " " $2 }' | \
    sed 's,^deb file,deb copy,' | \
    sed 's,/* *$,,'
  done


Best wishes,

--
      ,''`.
     : :'  :     Chris Lamb
     `. `'`      [hidden email] 🍥 chris-lamb.co.uk
       `-

Reply | Threaded
Open this post in threaded view
|

Re: Bug#926242: jenkins.debian.org: Please test reproducibility status of Debian Installer images

Cyril Brulebois-4
Chris Lamb <[hidden email]> (2019-06-05):
> I naturally understand your hesitation but on the other hand I would
> truly love to see this in buster. Indeed, we may actually have done
> enough work to boast about having reproducible installer images for
> the upcoming release (!) although without testing on our more-
> comprehensive testing framework it is difficult to tell at the
> moment...
>
> Devil's advocate: this is surely unlikely to break the release of
> buster itself? I mean, for the "final" official buster builds, that is?

That's exactly the point: I don't think it's unlikely. And it can't be
tested until it reaches the buildds. At which point, seeing breakages
isn't exactly what we want when a release is about to happen.

As a middle ground, how does the following sound? First test this in
unstable with the first alpha for bullseye, and possible backport it
in a point release?


Cheers,
--
Cyril Brulebois ([hidden email])            <https://debamax.com/>
D-I release manager -- Release team member -- Freelance Consultant

signature.asc (849 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Bug#926242: jenkins.debian.org: Please test reproducibility status of Debian Installer images

Chris Lamb -2
Hi Cyril,

> > Devil's advocate: this is surely unlikely to break the release of
> > buster itself? I mean, for the "final" official buster builds, that is?
>
> That's exactly the point: I don't think it's unlikely. […]

Could you elaborate on this bit? sources.list(5) options aren't /that/
common, in my experience. I ACK the lack of visibility until it hits
buildds, thanks for explaining that.

> As a middle ground, how does the following sound? First test this in
> unstable with the first alpha for bullseye, and possible backport it
> in a point release?

Ah, a point release might be the thing. Indeed, we might need some
other tiny changes to be reproducible that only become visible once we
can do the aforementioned extensive testing.

In the meantime, Mattia: I wonder if we could hack something specific
for src:debian-installer in addition to the network access exception?

For example, my reading of:

     683         if [ "$(MIRROR)x" != "x" ]; then \
     684                 echo "deb $(MIRROR) $(USE_UDEBS_FROM) $(UDEB_COMPONENTS)"; \
     685                 if [ "$(USE_UNRELEASED)" = 1 ]; then \
     686                         echo "deb $(MIRROR) unreleased $(UDEB_COMPONENTS)"; \
     687                 fi \
     688         else \
     689                 gen-sources.list.udeb "$(SYSTEM_SOURCES_LIST)" $(USE_UDEBS_FROM) $(UDEB_COMPONENTS) $(USE_PROPOSED_UPDATES); \
     690                 if [ "$(USE_UNRELEASED)" = 1 ]; then \
     691                         gen-sources.list.udeb "$(SYSTEM_SOURCES_LIST)" unreleased $(UDEB_COMPONENTS); \
     692                 fi \
     693         fi) > $@

… suggests that if we could export "MIRROR" pointing to (literally)
"[check-valid-until=no] http://deb.debian.org/debian" (!).


Regards,

--
      ,''`.
     : :'  :     Chris Lamb
     `. `'`      [hidden email] 🍥 chris-lamb.co.uk
       `-

Reply | Threaded
Open this post in threaded view
|

Re: Bug#926242: jenkins.debian.org: Please test reproducibility status of Debian Installer images

Philip Hands
In reply to this post by Chris Lamb -2
"Chris Lamb" <[hidden email]> writes:

> [adding [hidden email] to CC]
>
> Hi Colin,
>
>> This is all from dubious memory, but I suspect my setup at the time was
>> roughly an amd64 system with:
>>
>>   deb [arch=amd64] <local partial mirror>
>>   deb <official mirror>
>>
>> ... on the grounds that my local partial mirror didn't have space for
>> both amd64 and i386.
>
> Apologies for the delay in getting back to you all here.
>
> I've got this working locally here although we require the following
> change to the gen-sources.list.udeb script. Basically, we need print
> three columns if we have "[options]", otherwise we just print two:
>
> diff --git a/build/util/gen-sources.list.udeb b/build/util/gen-sources.list.udeb
> index 539345a45..ac416266a 100755
> --- a/build/util/gen-sources.list.udeb
> +++ b/build/util/gen-sources.list.udeb
> @@ -36,10 +36,9 @@ get_mirrors() {
>   [ -s $file ] || continue
>   grep '^deb[[:space:]]' $file | \
>     grep -v '^deb[[:space:]]\+cdrom:' | \
> -   sed 's,^deb \[[^]]*\] ,deb ,' | \
>     grep -v '\(security.debian.org\|volatile.debian.\(net\|org\)\)' | \
>     grep '[[:space:]]main' | \
> -   awk '{print $1 " " $2}' | \
> +   awk '{ print (substr($2, 0, 1) == "[") ? $1 " " $2 " " $3 : $1 " " $2 }' | \
>     sed 's,^deb file,deb copy,' | \
>     sed 's,/* *$,,'
>   done
>
> How does this look to you? Shell "golf" suggestions welcome,
> naturally. (I tried a few sed variants but it got a bit messy.)
I failed to resist that, so I _think_ this sed command implements the
same effect as those greps/seds & awk:

  sed -ne '/^deb[[:space:]]\+cdrom/d;
           /\(security.debian.org\|volatile.debian.\(net\|org\)\)/d;
           /^deb[[:space:]]\+.*[[:space:]]\+main/{
             s,\(deb[[:space:]]\+\(\[[^]]*\][[:space:]]\+\|\)[^[:space:]]*\).*$,\1,;
             s,^deb file,deb copy,;
             s,/* *$,,;
             p
           }'

I only tested that with a few fairly simple test cases, so hopefully
you've got some nice test data.

BTW I note that in the original (and therefore in this too) that the
exclusion of cdrom: and the 'deb file' to 'deb copy' bits only work if
there's no [option] bit in the line -- was that an oversight?

Also, I'm asking myself: Why all the [[:space:]]\+ stuff if one then
seds for '^deb file' and '/* *$' at the end? -- I think one should
choose to do one or the other of those throughout.

As you say though, the sed for handling the optional bit is rather nasty
to read.  I'm now wondering if the whole thing might not be better done
in a single awk invocation...

Cheers, Phil.
--
|)|  Philip Hands  [+44 (0)20 8530 9560]  HANDS.COM Ltd.
|-|  http://www.hands.com/    http://ftp.uk.debian.org/
|(|  Hugo-Klemm-Strasse 34,   21075 Hamburg,    GERMANY

signature.asc (847 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Bug#926242: jenkins.debian.org: Please test reproducibility status of Debian Installer images

Philip Hands
Philip Hands <[hidden email]> writes:
...

> I failed to resist that, so I _think_ this sed command implements the
> same effect as those greps/seds & awk:
>
>   sed -ne '/^deb[[:space:]]\+cdrom/d;
>            /\(security.debian.org\|volatile.debian.\(net\|org\)\)/d;
>            /^deb[[:space:]]\+.*[[:space:]]\+main/{
>              s,\(deb[[:space:]]\+\(\[[^]]*\][[:space:]]\+\|\)[^[:space:]]*\).*$,\1,;
>              s,^deb file,deb copy,;
>              s,/* *$,,;
>              p
>            }'
I seem to have neglected to type the line I had intended saying "If
this change needs to wait for post-buster anyway, then we could cut down
the number of sed/grep invocations".

This was not meant as a suggestion that we should do this sort of thing
this late, particularly since the suggested patch may actually be broken
if it's supposed to be able to deal with, e.g.:

  deb [arch=amd64] file:///...

because that won't currently end up as:

  deb [arch=amd64] copy:///...

so I completely sympathise with Cyril's caution.

Cheers, Phil.
--
|)|  Philip Hands  [+44 (0)20 8530 9560]  HANDS.COM Ltd.
|-|  http://www.hands.com/    http://ftp.uk.debian.org/
|(|  Hugo-Klemm-Strasse 34,   21075 Hamburg,    GERMANY

signature.asc (847 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Bug#926242: jenkins.debian.org: Please test reproducibility status of Debian Installer images

Chris Lamb -2
Philip Hands wrote:

> BTW I note that in the original (and therefore in this too) that the
> exclusion of cdrom: and the 'deb file' to 'deb copy' bits only work if
> there's no [option] bit in the line -- was that an oversight?

I would guess so, yes.

In light of that (and whilst my shell is a little rusty) but how about
we just make this all more explicit instead of abusing sed/awk?

For example:

get_mirrors () {
        local file
        local line
        for file in $@; do
                while read line
                do
                        if ! echo "$line" | grep -qs '^deb[[:space:]]'; then
                                continue
                        fi

                        local options=
                        local uri="$(echo "$line" | cut -d' ' -f2)"
                        local dist="$(echo "$line" | cut -d' ' -f3)"
                        local components="$(echo "$line" | cut -d' ' -f4-)"

                        case "$uri" in
                        \[*)
                                options="$uri "
                                uri="$(echo "$line" | cut -d' ' -f3)"
                                dist="$(echo "$line" | cut -d' ' -f4)"
                                components="$(echo "$line" | cut -d' ' -f5-)"
                                ;;
                        esac

                        case "$uri" in
                        cdrom:*|*security.debian.org*|*volatile.debian.*)
                                continue
                                ;;
                        file:*)
                                uri="$(echo "${uri}" | sed 's,^file,copy,')"
                                ;;
                        esac

                        echo "deb ${options}${uri} ${dist} ${components}"
                done < $file
        done
}


Best wishes,

--
      ,''`.
     : :'  :     Chris Lamb
     `. `'`      [hidden email] 🍥 chris-lamb.co.uk
       `-

Reply | Threaded
Open this post in threaded view
|

Re: [rb-general] Bug#926242: jenkins.debian.org: Please test reproducibility status of Debian Installer images

Chris Lamb -2
Chris Lamb wrote:

> In light of that (and whilst my shell is a little rusty) but how about
> we just make this all more explicit instead of abusing sed/awk?
>
> For example:

[…]

So, I heard a vague rumour that this "buster" thing was released? I
was thus wondering whether we could apply my patch from:

  https://bugs.debian.org/926242#127
 
:)


Best wishes,

--
      ,''`.
     : :'  :     Chris Lamb
     `. `'`      [hidden email] 🍥 chris-lamb.co.uk
       `-

Reply | Threaded
Open this post in threaded view
|

Re: [rb-general] Bug#926242: jenkins.debian.org: Please test reproducibility status of Debian Installer images

Holger Levsen-2
On Mon, Jul 08, 2019 at 10:27:02AM -0300, Chris Lamb wrote:
> So, I heard a vague rumour that this "buster" thing was released? I
> was thus wondering whether we could apply my patch from:
>   https://bugs.debian.org/926242#127

https://bugs.debian.org/926242#117 makes me think this is not to be
applied against jenkins.debian.net.git? So, a.) a full (git) patch is nicer
than just some inline code in a bug report because b.) this also reveals
where to apply against.


--
tschau,
        Holger

-------------------------------------------------------------------------------
               holger@(debian|reproducible-builds|layer-acht).org
       PGP fingerprint: B8BF 5413 7B09 D35C F026 FE9D 091A B856 069A AA1C

signature.asc (849 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: [rb-general] Bug#926242: jenkins.debian.org: Please test reproducibility status of Debian Installer images

Chris Lamb -2
Hi Holger,

> > So, I heard a vague rumour that this "buster" thing was released? I
> > was thus wondering whether we could apply my patch from:
[…]
> https://bugs.debian.org/926242#117 makes me think this is not to be
> applied against jenkins.debian.net.git?

Pre-buster, perhaps? I don't quite see why we cannot and should not
fix it "upstream" in d-i instead? Indeed, fixing it on our Jenkins
instance would surely be rather ugly and essentially involve special-
casing, hardcoding a patch, etc. etc. Ew.

(If we should fix it in d-i, I'll create a MR or similar but I'll
save that until there is some form of ACK...)


Regards,

--
      ,''`.
     : :'  :     Chris Lamb
     `. `'`      [hidden email] 🍥 chris-lamb.co.uk
       `-

Reply | Threaded
Open this post in threaded view
|

Re: [rb-general] Bug#926242: jenkins.debian.org: Please test reproducibility status of Debian Installer images

Holger Levsen-2
On Mon, Jul 08, 2019 at 11:47:34AM -0300, Chris Lamb wrote:
> > > So, I heard a vague rumour that this "buster" thing was released? I
> > > was thus wondering whether we could apply my patch from:
> […]
> > https://bugs.debian.org/926242#117 makes me think this is not to be
> > applied against jenkins.debian.net.git?
> Pre-buster, perhaps? I don't quite see why we cannot and should not
> fix it "upstream" in d-i instead?

yes, though #926242 is a bug against jenkins.d.o|n and not against d-i/.

> Indeed, fixing it on our Jenkins
> instance would surely be rather ugly and essentially involve special-
> casing, hardcoding a patch, etc. etc. Ew.

yeah

> (If we should fix it in d-i, I'll create a MR or similar but I'll
> save that until there is some form of ACK...)

*nods*


--
tschau,
        Holger

-------------------------------------------------------------------------------
               holger@(debian|reproducible-builds|layer-acht).org
       PGP fingerprint: B8BF 5413 7B09 D35C F026 FE9D 091A B856 069A AA1C

signature.asc (849 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: [rb-general] Bug#926242: jenkins.debian.org: Please test reproducibility status of Debian Installer images

Cyril Brulebois-4
In reply to this post by Chris Lamb -2
Chris Lamb <[hidden email]> (2019-07-08):

> Chris Lamb wrote:
>
> > In light of that (and whilst my shell is a little rusty) but how about
> > we just make this all more explicit instead of abusing sed/awk?
> >
> > For example:
>
> […]
>
> So, I heard a vague rumour that this "buster" thing was released? I
> was thus wondering whether we could apply my patch from:
>
>   https://bugs.debian.org/926242#127
>  
> :)
My current plan is (1) breathing a little, (2) getting the needed
bugfixes into 10.1.


Cheers,
--
Cyril Brulebois ([hidden email])            <https://debamax.com/>
D-I release manager -- Release team member -- Freelance Consultant

signature.asc (849 bytes) Download Attachment
12