Bug#928026: release-notes: document the state of security support for golang packages in Buster

classic Classic list List threaded Threaded
26 messages Options
12
Reply | Threaded
Open this post in threaded view
|

Bug#928026: release-notes: document the state of security support for golang packages in Buster

Holger Levsen-2
package: release-notes
x-debbugs-cc: [hidden email], [hidden email], [hidden email], [hidden email], [hidden email]

On Sat, Apr 20, 2019 at 11:07:34PM +0200, Moritz Mühlenhoff wrote:

> > Are there other concerns or warnings and
> > should they already be mentioned in the release notes?
>
> There has been no visible movement on the issues with Go as mentioned in
> https://lists.debian.org/debian-release/2018/07/msg00002.html (and
> this dates back much further, initial discussions were from 2016 or
> earlier).
>
> This is already an issue in Stretch (e.g. #922170), but will be much
> worse in Buster, so unless someone reliably commits to work on
> this ASAP the available options are to drop everything Go apart
> from the toolchain packages from buster or exclude of all that mess
> from security updates so that people know what they can expect.
 
filing a bug (in coordination with Moritz) so this doesnt get forgotten.

Also, I believe this bug should be cloned and reassigned to
src:debian-security-support as well, so it's also documented there. Will
do so once the bug arrives back.


--
tschau,
        Holger

-------------------------------------------------------------------------------
               holger@(debian|reproducible-builds|layer-acht).org
       PGP fingerprint: B8BF 5413 7B09 D35C F026 FE9D 091A B856 069A AA1C

signature.asc (849 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Bug#928026: security support for golang packages in Buster

Shengjing Zhu-3
Please CC [hidden email] and me.

Hi the security team and release team,

On Sat, Apr 20, 2019 at 11:07:34PM +0200, Moritz Mühlenhoff wrote:

> There has been no visible movement on the issues with Go as mentioned in
> https://lists.debian.org/debian-release/2018/07/msg00002.html (and
> this dates back much further, initial discussions were from 2016 or
> earlier).
>
> This is already an issue in Stretch (e.g. #922170), but will be much
> worse in Buster, so unless someone reliably commits to work on
> this ASAP the available options are to drop everything Go apart
> from the toolchain packages from buster or exclude of all that mess
> from security updates so that people know what they can expect.
>
IIUC, there're two concerns for Go packages.

1. the way to detect what packages need to be rebuilt if a Go package
   has been fixed.

   It should be easy in Buster. All the Go binary program packages (which
   are not arch:all) have a Built-Using filed. This filed records all
   the static linked libraries(include direct and indirect).

   So a similar sql script like
   https://ftp-master.debian.org/users/ansgar/outdated-built-using.txt
   should work, to filter the packages which need rebuild.

   And yes, we are aware the use of Built-Using filed is against policy
   now... #921284. IMHO, this cloud be transited to other filed in next
   release.

2. binNMU without full source upload for security-master.

   It's still not possible, and I don't know there's any effort to
   change the dak.

   But I want to know how security team handles other static linked
   languages, like rust, haskell, ocaml, etc.

   It's not the issue for only Go packages.

   The easiest probably is to binNMU in stable-pu.

--
Shengjing Zhu

signature.asc (499 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Bug#928026: security support for golang packages in Buster

Paul Gevers-4
Hi,

On 27-04-2019 09:31, Shengjing Zhu wrote:
> Please CC [hidden email] and me.

Done.

[...]

> IIUC, there're two concerns for Go packages.

[...]

> 2. binNMU without full source upload for security-master.
>
>    It's still not possible, and I don't know there's any effort to
>    change the dak.
>
>    But I want to know how security team handles other static linked
>    languages, like rust, haskell, ocaml, etc.

With respect to binNMU'ing, static linking is not a problem, only
arch:all is. Most haskell (4 vs 1048) and ocaml (21 vs 233) aren't
arch:all. haskell and ocaml have a framework in place to at least know
the status in unstable/testing. See e.g. the "permanent trackers" at
https://release.debian.org/transitions/ I don't know yet what this means
for security support. Neither do I know what it means for rust.

>    It's not the issue for only Go packages.

But most haskell and ocaml packages can be binNMU'd.

>    The easiest probably is to binNMU in stable-pu.

I don't understand what you mean by this last sentence. You mean to not
do a binNMU but a full NMU for all the arch:all packages? I think the
problem of the security team is that they don't want to commit to that.

[bug 928227]
On 05-05-2019 18:00, Shengjing Zhu wrote:> Hi,

[...]

>> On Tue, Apr 30, 2019 at 05:07:57PM +0800, Drew Parsons wrote:
>>> Please unblock package golang-golang-x-net-dev
>>>
>>> Upstream has provided patches addressing security issues
>>> CVE-2018-17846 / CVE-2018-17847 / CVE-2018-17848
>>> (Debian bug #911795).
>>
>> How will unblocking this fix these issues? golang-golang-x-net-dev is
embedded
>> in a number of packages in buster. If they are not updated, the
unblock will
>> not fix anything. How will this be handled?
>>
>
> All the reverse depends need binNMU.
> Since the Go packages are using(abusing) Built-Using tag, probably the
> release team will binNMU all outdated Built-Using packages at this
> period(before release)?

I think the rebuild (or at least a big chunk of it) has already been
done. And, as noted above, that we can't binNMU arch:all yet. Will you
source upload those and add the list to bug 928227 and tell us which
additional packages need to be scheduled for a binNMU?

Just wondering, does anybody already have tooling/scripts/urls do check
the current status? If not, I'll cook up something to assess the
situation for myself. I'll update bug 928227 when I have some data.

> Maybe we can keep the conversation at
> https://lists.debian.org/msgid-search/20190427073148.GA7478@debian ?

Done.

Paul


signature.asc (499 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Bug#928026: security support for golang packages in Buster

Shengjing Zhu-3
On Wed, May 8, 2019 at 2:45 PM Paul Gevers <[hidden email]> wrote:

>
> Hi,
>
> On 27-04-2019 09:31, Shengjing Zhu wrote:
> > Please CC [hidden email] and me.
>
> Done.
>
> [...]
>
> > IIUC, there're two concerns for Go packages.
>
> [...]
>
> > 2. binNMU without full source upload for security-master.
> >
> >    It's still not possible, and I don't know there's any effort to
> >    change the dak.
> >
> >    But I want to know how security team handles other static linked
> >    languages, like rust, haskell, ocaml, etc.
>
> With respect to binNMU'ing, static linking is not a problem, only
> arch:all is. Most haskell (4 vs 1048) and ocaml (21 vs 233) aren't
> arch:all. haskell and ocaml have a framework in place to at least know
> the status in unstable/testing. See e.g. the "permanent trackers" at
> https://release.debian.org/transitions/ I don't know yet what this means
> for security support. Neither do I know what it means for rust.
>
> >    It's not the issue for only Go packages.
>
> But most haskell and ocaml packages can be binNMU'd.
>
> >    The easiest probably is to binNMU in stable-pu.
>
> I don't understand what you mean by this last sentence. You mean to not
> do a binNMU but a full NMU for all the arch:all packages? I think the
> problem of the security team is that they don't want to commit to that.
>

All the arch:all golang packages, are only meant for building other
arch:any packages.
Let's say one arch:all package(take golang-golang-x-net-dev for
example) has security bug, we need:
1. fix the golang-golang-x-net-dev, do an usual upload.
2. since golang-golang-x-net-dev is statically embedded in other
arch:any packages, rebuild/binNMU these packages.

These steps usually work, when in unstable.

But for stable release, we need to do 1&2 in security-master. 1 is
possible, just like normal packages.
After 1 is done in security-master, we need to rebuild the reverse
depends. However binNMU is not possible in security-master. The
secrity-master doesn't share orig tarball with ftp-master. So in
security-master, full-NMU should be done for *many* packages.

And, other arch:all packages are not involved in these steps. Because
these packages are not affected, since they only contain the their own
Go source code files.

> [bug 928227]
> On 05-05-2019 18:00, Shengjing Zhu wrote:> Hi,
>
> [...]
>
> >> On Tue, Apr 30, 2019 at 05:07:57PM +0800, Drew Parsons wrote:
> >>> Please unblock package golang-golang-x-net-dev
> >>>
> >>> Upstream has provided patches addressing security issues
> >>> CVE-2018-17846 / CVE-2018-17847 / CVE-2018-17848
> >>> (Debian bug #911795).
> >>
> >> How will unblocking this fix these issues? golang-golang-x-net-dev is
> embedded
> >> in a number of packages in buster. If they are not updated, the
> unblock will
> >> not fix anything. How will this be handled?
> >>
> >
> > All the reverse depends need binNMU.
> > Since the Go packages are using(abusing) Built-Using tag, probably the
> > release team will binNMU all outdated Built-Using packages at this
> > period(before release)?
>
> I think the rebuild (or at least a big chunk of it) has already been
> done. And, as noted above, that we can't binNMU arch:all yet. Will you
> source upload those and add the list to bug 928227 and tell us which
> additional packages need to be scheduled for a binNMU?
>
> Just wondering, does anybody already have tooling/scripts/urls do check
> the current status? If not, I'll cook up something to assess the
> situation for myself. I'll update bug 928227 when I have some data.
>

There's no tool yet, but some SQL scripts like
http://paste.debian.net/1082099/

As show in the result, for
golang-golang-x-net-dev/1:0.0+git20181201.351d144+dfsg-3, probably 66
packages need binNMU.

--
Shengjing Zhu

Reply | Threaded
Open this post in threaded view
|

Bug#928026: security support for golang packages in Buster

Moritz Mühlenhoff-2
In reply to this post by Paul Gevers-4
On Wed, May 08, 2019 at 08:45:30AM +0200, Paul Gevers wrote:

> > 2. binNMU without full source upload for security-master.
> >
> >    It's still not possible, and I don't know there's any effort to
> >    change the dak.
> >
> >    But I want to know how security team handles other static linked
> >    languages, like rust, haskell, ocaml, etc.
>
> With respect to binNMU'ing, static linking is not a problem, only
> arch:all is. Most haskell (4 vs 1048) and ocaml (21 vs 233) aren't
> arch:all. haskell and ocaml have a framework in place to at least know
> the status in unstable/testing. See e.g. the "permanent trackers" at
> https://release.debian.org/transitions/ I don't know yet what this means
> for security support. Neither do I know what it means for rust.

There's the additional issue that ftp-master and security-master don't
share tarballs; binNMUs are only possible for packages which are on
security-master, so we'd need to do manual source uploads for every
affected go package.

> But most haskell and ocaml packages can be binNMU'd.

This issue has been around for Haskell and Ocaml for a long time, but it
hasn't been an issue in practice. Ideally we can use the same mechanism
found for Go, also for Ocaml or Haskell if the need should ever arise.

For Go it's different, it has already bitten us for the limited subset
of Go applications in Stretch (and #922170 which is needed to kludge
around it), but with the increased footprint of Go stuff in buster we
need something better.

The same issue will hit us with Rust as well, but for buster we have
only two relevant packages (librsvg and firefox which use vendored
libs, which seems manageable). For bullseye we also need a proper, generic
solution for Rust.

> I think the
> problem of the security team is that they don't want to commit to that.

Yeah, that won't work, this could amount to hundreds of packages.

Cheers,
        Moritz

Reply | Threaded
Open this post in threaded view
|

Bug#928026: security support for golang packages in Buster

Paul Wise via nm
In reply to this post by Paul Gevers-4
On Wed, May 8, 2019 at 2:45 PM Paul Gevers wrote:

> With respect to binNMU'ing, static linking is not a problem, only
> arch:all is. Most haskell (4 vs 1048) and ocaml (21 vs 233) aren't
> arch:all. haskell and ocaml have a framework in place to at least know
> the status in unstable/testing. See e.g. the "permanent trackers" at
> https://release.debian.org/transitions/ I don't know yet what this means
> for security support. Neither do I know what it means for rust.

I think there is something that the release/security teams might be
missing here:

The Go/Rust arch:all packages (the golang-*-dev ones) contain only
source code (ideally things would support build-deps on foo:src, the
current Go/Rust binary packages are a workaround for that being
missing), so they do not need to be changed after a security upload.
Only the packages containing statically linked Go/Rust code need to be
binNMUed and those ones should be arch:any since they contain
architecture-specific binaries. In addition the arch:all packages do
not have Built-Using, only the statically linked ones do.

So the workflow seems to be quite manageable, modulo the
security-master binNMU issue: fix the security issue where it
originated, then binNMU anything that has Built-Using on any version
less than the fixed version.

--
bye,
pabs

https://wiki.debian.org/PaulWise

Reply | Threaded
Open this post in threaded view
|

Bug#928026: security support for golang packages in Buster

Shengjing Zhu-3
In reply to this post by Moritz Mühlenhoff-2
Hi the security team,

On Thu, May 9, 2019 at 1:53 AM Moritz Muehlenhoff <[hidden email]> wrote:
[...]
>
> There's the additional issue that ftp-master and security-master don't
> share tarballs; binNMUs are only possible for packages which are on
> security-master, so we'd need to do manual source uploads for every
> affected go package.
>

I probably lack of some historical background, have you ever think of
merging ftp-master and security-master?

--
Shengjing Zhu

Reply | Threaded
Open this post in threaded view
|

Bug#928026: security support for golang packages in Buster

Salvatore Bonaccorso-4
Hi,

On Fri, May 10, 2019 at 10:44:13AM +0800, Shengjing Zhu wrote:

> Hi the security team,
>
> On Thu, May 9, 2019 at 1:53 AM Moritz Muehlenhoff <[hidden email]> wrote:
> [...]
> >
> > There's the additional issue that ftp-master and security-master don't
> > share tarballs; binNMUs are only possible for packages which are on
> > security-master, so we'd need to do manual source uploads for every
> > affected go package.
> >
>
> I probably lack of some historical background, have you ever think of
> merging ftp-master and security-master?

The security team does not manage dak on security-master, this is
actually ftp-masters domain. The separation has as disadantages and
advantages, one which comes to my mind idepenently on the aspect of
the one beeing security-master is to have a fallback updateing channel
in case one or the other cannot be used.

But okay that's not the point.

There is #823820 for one Built-Using aspect, but the idea might be
possible to generalize to have on every time orig tarballs from
main archive available on security-master as well.

Regards,
Salvatore

Reply | Threaded
Open this post in threaded view
|

Bug#928026: technical solutions enabling binNMUs in the security archive (support of golang packages)

Paul Gevers-4
In reply to this post by Shengjing Zhu-3
Hi all,

[@ftp-master: we're discussing the biggest blocker for picking a buster
release date]

TL;DR; With this mail I'd like to ask technical background and/or
reasoning of why the security archive can't do binNMUs for packages that
weren't sourcefully uploaded there and to search for concrete potential
solutions (I assume there are more) to solve the technical problem of
binNMU-ing in the security archive.

On 08-05-2019 18:33, Shengjing Zhu wrote:
>>> 2. binNMU without full source upload for security-master.
>>>
>>>    It's still not possible, and I don't know there's any effort to
>>>    change the dak.

I am all to new into this business, so ignore my ignorance and please
educate me on any mistake. Can somebody please try to explain what the
exact problem is or point me to documentation or earlier discussion that
do that? I'll try to write down potential descriptions as I see them
(from quite a bit of guessing).

IIUC from this thread so far, ftp-master and security-master are two
separate archives [1]. The security-master archive only contains sources
and binaries that were uploaded to security. A potential solution to the
problem at hand seems to be to sync all sources from ftp-master into
security-master, but I guess that is undesirable because of the massive
size increase of the security archive in that case. If this is going to
be the solution, is this still dak domain?

Another solution already raised by Shengjing is to merge the archives. I
*guess* that is undesirable due to the fact that the security archive
often has embargoed sources and binaries. Am I right there?

Another direction, as I am wondering: when packages are build for the
security archive, the binary packages from the regular archive are
available to the buildds, right? So is the problem really that the
security archive doesn't have the sources, or could/should wanna-build
(or the code really responsible for creating the proper binNMU input) be
enhanced to support this use case with the archives as they are?

I have been thinking that code could be made to do the required stuff
out-of-band and basically do sourcefull uploads automatically when
required for a set of packages. It feels like a great hack, but I guess
such a tool could cover the time until the archive tooling can properly
support it.

[Here your solution direction]

Paul

[1] I wonder how that matches with the drawing on
https://wiki.debian.org/DebianDak as that suggests that the security
archive is just another pool like unstable and testing.


signature.asc (499 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Bug#928227: Bug#928026: security support for golang packages in Buster

Paul Gevers-4
In reply to this post by Shengjing Zhu-3
Hi Shengjing,

On 08-05-2019 18:33, Shengjing Zhu wrote:
> On Wed, May 8, 2019 at 2:45 PM Paul Gevers <[hidden email]> wrote:
> There's no tool yet, but some SQL scripts like
> http://paste.debian.net/1082099/
>
> As show in the result, for
> golang-golang-x-net-dev/1:0.0+git20181201.351d144+dfsg-3, probably 66
> packages need binNMU.

@Shengjing, do you care to add the code and list to this bug? You're
paste expired, so I can't reproduce what you did.

@RT: Would it be a good extension to ben (the tool that generates the
transition pages) to support the Built-Using (or whatever the future
replacement for this purpose) field/syntax, such that we can also track
this kind of thing?

Paul


signature.asc (499 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Bug#928227: Bug#928026: security support for golang packages in Buster

Shengjing Zhu-3
On Sat, May 18, 2019 at 5:30 PM Paul Gevers <[hidden email]> wrote:

>
> Hi Shengjing,
>
> On 08-05-2019 18:33, Shengjing Zhu wrote:
> > On Wed, May 8, 2019 at 2:45 PM Paul Gevers <[hidden email]> wrote:
> > There's no tool yet, but some SQL scripts like
> > http://paste.debian.net/1082099/
> >
> > As show in the result, for
> > golang-golang-x-net-dev/1:0.0+git20181201.351d144+dfsg-3, probably 66
> > packages need binNMU.
>
> @Shengjing, do you care to add the code and list to this bug? You're
> paste expired, so I can't reproduce what you did.
>

tweak a little bit,

http://paste.debian.net/1082006/

--
Shengjing Zhu

Reply | Threaded
Open this post in threaded view
|

Bug#928227: Bug#928026: security support for golang packages in Buster

Shengjing Zhu-3
On Sat, May 18, 2019 at 5:41 PM Shengjing Zhu <[hidden email]> wrote:

>
> On Sat, May 18, 2019 at 5:30 PM Paul Gevers <[hidden email]> wrote:
> >
> > Hi Shengjing,
> >
> > On 08-05-2019 18:33, Shengjing Zhu wrote:
> > > On Wed, May 8, 2019 at 2:45 PM Paul Gevers <[hidden email]> wrote:
> > > There's no tool yet, but some SQL scripts like
> > > http://paste.debian.net/1082099/
> > >
> > > As show in the result, for
> > > golang-golang-x-net-dev/1:0.0+git20181201.351d144+dfsg-3, probably 66
> > > packages need binNMU.
> >
> > @Shengjing, do you care to add the code and list to this bug? You're
> > paste expired, so I can't reproduce what you did.
> >
>
> tweak a little bit,
>
> http://paste.debian.net/1082006/

And FTR,

$ cat test.sql
SELECT
  s.source AS source,
  s2.version AS built_using_version,
  ar.arch_string AS arch

FROM extra_src_references esr
JOIN binaries b ON esr.bin_id = b.id
JOIN source s ON b.source = s.id
JOIN architecture ar ON ar.id = b.architecture
JOIN bin_associations ba ON esr.bin_id = ba.bin
JOIN source s2 ON esr.src_id = s2.id
JOIN suite su ON ba.suite = su.id

WHERE s2.version < '1:0.0+git20181201.351d144+dfsg-3'
  AND s2.source = 'golang-golang-x-net-dev'
  AND su.suite_name = 'testing'

GROUP BY s.source, s2.version, ar.arch_string


--
Shengjing Zhu

Reply | Threaded
Open this post in threaded view
|

Bug#928227: unblock: golang-golang-x-net-dev/1:0.0+git20181201.351d144+dfsg-3

Paul Gevers-4
Hi Shengjijng,

And how do we prevent that binNMU'ing all these reverse dependencies
don't pick up other golang changes in unstable? Can you tell us if there
are any? We don't want to accept more unnecessary packages.

Paul


signature.asc (499 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Bug#928227: unblock: golang-golang-x-net-dev/1:0.0+git20181201.351d144+dfsg-3

Shengjing Zhu-3
On Sat, May 18, 2019 at 10:53 PM Paul Gevers <[hidden email]> wrote:
>
> Hi Shengjijng,
>
> And how do we prevent that binNMU'ing all these reverse dependencies
> don't pick up other golang changes in unstable? Can you tell us if there
> are any? We don't want to accept more unnecessary packages.

Is it possible to binNMU against testing(like testing-proposed-update)?

I'm sure most packages can't migrate to testing after binNMU in unstable.
like etcd, nomad, snapd, umoci... which build-depends
golang-golang-x-sys and/or golang-github-mattn-go-isatty(which are
updated in unstable unexpectedly).


--
Shengjing Zhu

Reply | Threaded
Open this post in threaded view
|

Bug#928227: unblock: golang-golang-x-net-dev/1:0.0+git20181201.351d144+dfsg-3

Paul Gevers-4
Hi Shengjing,

On 18-05-2019 17:10, Shengjing Zhu wrote:

> On Sat, May 18, 2019 at 10:53 PM Paul Gevers <[hidden email]> wrote:
>>
>> Hi Shengjijng,
>>
>> And how do we prevent that binNMU'ing all these reverse dependencies
>> don't pick up other golang changes in unstable? Can you tell us if there
>> are any? We don't want to accept more unnecessary packages.
>
> Is it possible to binNMU against testing(like testing-proposed-update)?
>
> I'm sure most packages can't migrate to testing after binNMU in unstable.
> like etcd, nomad, snapd, umoci... which build-depends
> golang-golang-x-sys and/or golang-github-mattn-go-isatty(which are
> updated in unstable unexpectedly).
Can those packages be reverted to have the content of buster, e.g. with
a +really version? Than we can schedule the binNMU's in unstable and
have them migrate with these changelog only change packages.

Paul


signature.asc (499 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Bug#928227: unblock: golang-golang-x-net-dev/1:0.0+git20181201.351d144+dfsg-3

Shengjing Zhu-3
On Sun, May 19, 2019 at 4:06 AM Paul Gevers <[hidden email]> wrote:

>
> Hi Shengjing,
>
> On 18-05-2019 17:10, Shengjing Zhu wrote:
> > On Sat, May 18, 2019 at 10:53 PM Paul Gevers <[hidden email]> wrote:
> >>
> >> Hi Shengjijng,
> >>
> >> And how do we prevent that binNMU'ing all these reverse dependencies
> >> don't pick up other golang changes in unstable? Can you tell us if there
> >> are any? We don't want to accept more unnecessary packages.
> >
> > Is it possible to binNMU against testing(like testing-proposed-update)?
> >
> > I'm sure most packages can't migrate to testing after binNMU in unstable.
> > like etcd, nomad, snapd, umoci... which build-depends
> > golang-golang-x-sys and/or golang-github-mattn-go-isatty(which are
> > updated in unstable unexpectedly).
>
> Can those packages be reverted to have the content of buster, e.g. with
> a +really version? Than we can schedule the binNMU's in unstable and
> have them migrate with these changelog only change packages.
>

This could cause another mess. Since we are close to buster release, I
hope this can be avoid...

1. I'm not sure how many golang dev packages started a "trasition" in unstable,

```
zhsj@coccia:~$ cat golang-divarication.sql
SELECT
 s.source AS source,
 s.version AS version_testing,
 s2.version AS version_unstable

FROM source s
JOIN source_metadata sm ON sm.src_id = s.id
JOIN metadata_keys mk ON mk.key_id = sm.key_id
JOIN source s2 on s2.source = s.source
JOIN src_associations sa ON s.id = sa.source
JOIN suite su ON sa.suite = su.id
JOIN src_associations sa2 ON s2.id = sa2.source
JOIN suite su2 ON sa2.suite = su2.id

WHERE
 su.suite_name = 'testing'
 AND su2.suite_name = 'unstable'
 AND s.version != s2.version
 AND mk.key IN ('Maintainer', 'Uploaders')
 AND sm.value like '%pkg-go%'

GROUP BY s.source, s.version, s2.version
zhsj@coccia:~$ psql service=projectb -f golang-divarication.sql |cat
                source                  |         version_testing
    |         version_unstable
-----------------------------------------+----------------------------------+-----------------------------------
go-dep                                  | 0.5.0-1
    | 0.5.1-1
goiardi                                 | 0.11.9-2
    | 0.11.9-3
gokey                                   | 0.0~git20181023.b4e2780-1
    | 0.0~git20190103.40eba7e-1
golang-gitaly-proto                     | 0.123.0+dfsg-2
    | 1.22.0+dfsg-1
golang-github-alecthomas-chroma         | 0.6.2-1
    | 0.6.3-1
golang-github-bep-debounce              | 1.1.0-1
    | 1.2.0-1
golang-github-disintegration-imaging    | 1.5.0-1
    | 1.6.0-1
golang-github-mattn-go-isatty           | 0.0.4-1
    | 0.0.7-1
golang-github-rackspace-gophercloud     |
1.0.0+git20160603.920.934dbf8-1  | 1.0.0+git20161013.1012.e00690e8-1
golang-github-seccomp-libseccomp-golang | 0.9.0-1
    | 0.9.0-2
golang-github-spf13-afero               | 1.2.1-1
    | 1.2.2-1
golang-github-spf13-jwalterweatherman   | 1.0.0+git20181028.94f6ae3-1
    | 1.1.0-1
golang-github-spf13-viper               | 1.3.1-1
    | 1.3.2-1
golang-golang-x-image                   | 0.0~git20181116.cd38e80-1
    | 0.0~git20190321.3fc05d4-1
golang-golang-x-net-dev                 |
1:0.0+git20181201.351d144+dfsg-2 | 1:0.0+git20181201.351d144+dfsg-3
golang-golang-x-sys                     | 0.0~git20181228.9a3f9b0-1
    | 0.0~git20190412.9773273-1
hugo                                    | 0.54.0-1
    | 0.55.4-1
hugo                                    | 0.54.0-1
    | 0.55.5-1
influxdb                                | 1.6.4-1
    | 1.1.1+dfsg1-4
mtail                                   | 3.0.0~rc19-2
    | 3.0.0~rc24.1-1
sia                                     | 1.3.0-1.1~deb10u1
    | 1.3.0-1.1
(21 rows)
```

(And please take another note that some leaf/binary packages, like
hugo, influxdb, mtail.. already uploaded new version in unstable...)

2. more packages need rebuild

```
zhsj@coccia:~$ cat golang-dep.sql
SELECT
 s.source AS source

FROM extra_src_references esr
JOIN binaries b ON esr.bin_id = b.id
JOIN source s ON b.source = s.id
JOIN architecture ar ON ar.id = b.architecture
JOIN bin_associations ba ON esr.bin_id = ba.bin
JOIN source s2 ON esr.src_id = s2.id
JOIN suite su ON ba.suite = su.id

WHERE
-- s2.source IN ('golang-golang-x-sys', 'golang-golang-x-net-dev',
'golang-github-spf13-viper', 'golang-github-mattn-go-isatty')
 s2.source IN (
SELECT
 s.source AS source

FROM source s
JOIN source_metadata sm ON sm.src_id = s.id
JOIN metadata_keys mk ON mk.key_id = sm.key_id
JOIN source s2 on s2.source = s.source
JOIN src_associations sa ON s.id = sa.source
JOIN suite su ON sa.suite = su.id
JOIN src_associations sa2 ON s2.id = sa2.source
JOIN suite su2 ON sa2.suite = su2.id

WHERE
 su.suite_name = 'testing'
 AND su2.suite_name = 'unstable'
 AND s.version != s2.version
 AND mk.key IN ('Maintainer', 'Uploaders')
 AND sm.value like '%pkg-go%'

GROUP BY s.source
)
 AND su.suite_name = 'testing'

GROUP BY s.source
zhsj@coccia:~$ psql service=projectb -f golang-dep.sql |cat
                   source
----------------------------------------------
abci
acbuild
acmetool
amazon-ecr-credential-helper
aptly
autodeb
balboa
burrow
certspotter
chasquid
cloudsql-proxy
consul
consulfs
continuity
coyim
debiman
debos
dh-make-golang
dnscrypt-proxy
dnss
docker-registry
docker.io
elvish
etcd
ethflux
fever
fscrypt
fzf
g10k
git-lfs
gitlab-workhorse
go-cve-dictionary
go-dep
go-exploitdb
go-wire
gobuster
gocryptfs
goiardi
gokey
golang-github-appc-docker2aci
golang-github-appc-spec
golang-github-cloudflare-cfssl
golang-github-grpc-ecosystem-grpc-gateway
golang-github-hashicorp-serf
golang-github-influxdata-tail
golang-github-mendersoftware-mender-artifact
golang-github-spf13-cobra
golang-github-tdewolff-minify
golang-github-vbatts-tar-split
golang-github-xenolf-lego
golang-github-xordataexchange-crypt
golang-github-yosssi-ace
golang-gogottrpc
golang-golang-x-tools
google-cloud-print-connector
gopass
gost
gosu
goval-dictionary
gron
hub
hugo
influxdb
irtt
jid
kcptun
mender-cli
mender-client
merkleeyes
mongo-tools
morty
mtail
ncbi-entrez-direct
nomad
notary
obfs4proxy
packer
pk4
prometheus
prometheus-alertmanager
prometheus-apache-exporter
prometheus-bind-exporter
prometheus-bird-exporter
prometheus-blackbox-exporter
prometheus-haproxy-exporter
prometheus-mailexporter
prometheus-mysqld-exporter
prometheus-node-exporter
prometheus-postgres-exporter
prometheus-pushgateway
prometheus-snmp-exporter
prometheus-sql-exporter
pt-websocket
rawdns
rclone
reflex
restic
rkt
runc
sia
singularity-container
slinkwatch
snapd
stenographer
syncthing
toxiproxy
umoci
vuls
webhook
wuzz
(110 rows)
```

--
Shengjing Zhu

Reply | Threaded
Open this post in threaded view
|

Bug#928026: technical solutions enabling binNMUs in the security archive (support of golang packages)

Ansgar Burchardt-8
In reply to this post by Paul Gevers-4
Paul Gevers writes:

> On 08-05-2019 18:33, Shengjing Zhu wrote:
>>>> 2. binNMU without full source upload for security-master.
>>>>
>>>>    It's still not possible, and I don't know there's any effort to
>>>>    change the dak.
>
> I am all to new into this business, so ignore my ignorance and please
> educate me on any mistake. Can somebody please try to explain what the
> exact problem is or point me to documentation or earlier discussion that
> do that? I'll try to write down potential descriptions as I see them
> (from quite a bit of guessing).
>
> IIUC from this thread so far, ftp-master and security-master are two
> separate archives [1]. The security-master archive only contains sources
> and binaries that were uploaded to security. A potential solution to the
> problem at hand seems to be to sync all sources from ftp-master into
> security-master, but I guess that is undesirable because of the massive
> size increase of the security archive in that case. If this is going to
> be the solution, is this still dak domain?

I though about importing the full source to security-master already for
a different reason: `Built-Using` leads to a similar problem as binNMUs
in that uploads require source that is not already present in the
archive.

It is not necessary to push all sources to the public mirrors.

> Another solution already raised by Shengjing is to merge the archives. I
> *guess* that is undesirable due to the fact that the security archive
> often has embargoed sources and binaries. Am I right there?

That doesn't work as dak doesn't try to keep secrets.  There are various
ways information would be leaked about embargoed issues (mails,
database, web interface (rmadison), ...).

I personally also don't find it too bad to have a fallback: if one of
the hosts is broken at the same time we have to release a critical
update, we can still do so by publishing via the "wrong" archive.

Ansgar

Reply | Threaded
Open this post in threaded view
|

Bug#928227: unblock: golang-golang-x-net-dev/1:0.0+git20181201.351d144+dfsg-3

Paul Gevers-4
In reply to this post by Shengjing Zhu-3
Hi Shengjing,

On 19-05-2019 17:01, Shengjing Zhu wrote:

> On Sun, May 19, 2019 at 4:06 AM Paul Gevers <[hidden email]> wrote:
>> On 18-05-2019 17:10, Shengjing Zhu wrote:
>>> On Sat, May 18, 2019 at 10:53 PM Paul Gevers <[hidden email]> wrote:
>>> I'm sure most packages can't migrate to testing after binNMU in unstable.
>>> like etcd, nomad, snapd, umoci... which build-depends
>>> golang-golang-x-sys and/or golang-github-mattn-go-isatty(which are
>>> updated in unstable unexpectedly).
>>
>> Can those packages be reverted to have the content of buster, e.g. with
>> a +really version? Than we can schedule the binNMU's in unstable and
>> have them migrate with these changelog only change packages.
>>
>
> This could cause another mess. Since we are close to buster release, I
> hope this can be avoid...
I fear this is part of the problem of the static linking in combination
with the freeze. Without major changes to the Debian infrastructure, the
golang ecosystem really needs to refrain from updating in unstable
during the freeze. At this moment it's best to make unstable in line
with testing again.

> 1. I'm not sure how many golang dev packages started a "trasition" in unstable,
>
> ```
> zhsj@coccia:~$ cat golang-divarication.sql

[...]

> (And please take another note that some leaf/binary packages, like
> hugo, influxdb, mtail.. already uploaded new version in unstable...)

They will need to be reverted as well if I understand correctly.

> 2. more packages need rebuild

I don't understand why you say that. Only the reverse build dependencies
of golang-golang-x-net-dev need to be rebuild and migrate to testing,
no? That the packages in unstable aren't all build with the sources in
unstable is normal right, so no big deal.

Paul


signature.asc (499 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Bug#928026: Bug#928227: technical solutions enabling binNMUs in the security archive (support of golang packages)

Paul Gevers-4
In reply to this post by Ansgar Burchardt-8
Hi Ansgar,

On 20-05-2019 09:06, Ansgar wrote:
> I though about importing the full source to security-master already for
> a different reason: `Built-Using` leads to a similar problem as binNMUs
> in that uploads require source that is not already present in the
> archive.
>
> It is not necessary to push all sources to the public mirrors.

Does this mean you think it is feasible to do/fix this in the near future?

>> Another solution already raised by Shengjing is to merge the archives. I
>> *guess* that is undesirable due to the fact that the security archive
>> often has embargoed sources and binaries. Am I right there?
>
> That doesn't work as dak doesn't try to keep secrets.  There are various
> ways information would be leaked about embargoed issues (mails,
> database, web interface (rmadison), ...).
>
> I personally also don't find it too bad to have a fallback: if one of
> the hosts is broken at the same time we have to release a critical
> update, we can still do so by publishing via the "wrong" archive.
Regarding my other direction with wanna-build, I learned yesterday via
another bug (#894441 binNMUs should be replaced by easy no-change
uploads) that wanna-build is not in the place to fix this because
uploads need to be signed.

Paul


signature.asc (499 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Bug#928227: unblock: golang-golang-x-net-dev/1:0.0+git20181201.351d144+dfsg-3

Shengjing Zhu-3
In reply to this post by Paul Gevers-4
On Tue, May 21, 2019 at 2:53 AM Paul Gevers <[hidden email]> wrote:
[...]
> At this moment it's best to make unstable in line
> with testing again.

I could start to do this, but it would take some time.

> > 2. more packages need rebuild
>
> I don't understand why you say that. Only the reverse build dependencies
> of golang-golang-x-net-dev need to be rebuild and migrate to testing,
> no? That the packages in unstable aren't all build with the sources in
> unstable is normal right, so no big deal.

A Build-Depends golang-golang-x-net-dev and B;
B is divaricated in unstable and testing.

To make A migrate to testing after rebuilding in unstable, B needs
updating(revert), otherwise it is blocked by B.
Then B is updated and migrates to testing, now every package
Build-Depends B needs rebuilding, for out-dated Build-Using reason.

So eventually the rebuild set is {golang-golang-x-net-dev's reverse
Build-Depends, B's reverse Build-Depends}.
And this set is stable only when all Go related packages in unstable
and testing are aligned.

--
Shengjing Zhu

12