What to do when DD considers policy to be optional? [kubernetes]

classic Classic list List threaded Threaded
120 messages Options
1234 ... 6
Reply | Threaded
Open this post in threaded view
|

What to do when DD considers policy to be optional? [kubernetes]

Dmitry Smirnov
Something interesting just happened. An inexperienced DD adopted a very
complicated package (kubernetes) and uploaded it with changes that would have
never been accepted by ftp-masters.

What would be best to do in such situation?

The problem is not with DD's qualification (although this certainly plays a
role) but with intentional non-compliance with policies and packaging
practices.

As a person who originally introduced Kubernetes to Debian I can say that it
is indeed a lot of work to maintain this package and to reuse packaged
libraries. But I've demonstrated that it is possible at least to some extent.

New maintainer of kubernetes do not even make an attempt to build it properly
and blatantly states the following in README which demonstrates profound lack
of understanding of problems that were impairing maintenance of the package:

   "I kindly ask purist aspirations that effectively halted Kubernetes'
    release and updates in Debian for YEARS to be kept at bay."

I don't consider myself to be a purist. I have pragmatic approach to
packaging but I feel concerned when policies and packaging practices are
circumvented.

I don't recall a situation when policing of how a package is maintained would
be necessary long after package is accepted...
What do we do to ensure quality work in situation of technological hijack
when maintainer is unwilling to follow the practice or comply with policy?

This is not a technical disagreement so I think that involving technical
committee may not be the right way to handle the problem... Or is it?

--
Cheers,
 Dmitry Smirnov.

---

We do our friends no favors by pretending not to notice flaws in their
work, especially when those who are not their friends are bound to notice
these same flaws.
        -- Sam Harris

signature.asc (849 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: What to do when DD considers policy to be optional? [kubernetes]

Sean Whitton
Hello Dmitry, Janos, others,

On Mon 23 Mar 2020 at 05:32PM +11, Dmitry Smirnov wrote:

> Something interesting just happened. An inexperienced DD adopted a very
> complicated package (kubernetes) and uploaded it with changes that would have
> never been accepted by ftp-masters.

Specifically, as README.Debian states, the vendor/ subdirectory of the
source package contains more than two hundred Go libraries.

I can't speak for the whole FTP Team here, but if this had ended up in
NEW and I had been the FTP Team member to review it, I would indeed have
rejected it, on the grounds that accepting the upload renders Debian
less maintaintable in various ways.

> What would be best to do in such situation?

I think that I would start by filing an RC bug.

> The problem is not with DD's qualification (although this certainly plays a
> role) but with intentional non-compliance with policies and packaging
> practices.
>
> As a person who originally introduced Kubernetes to Debian I can say that it
> is indeed a lot of work to maintain this package and to reuse packaged
> libraries. But I've demonstrated that it is possible at least to some extent.
>
> New maintainer of kubernetes do not even make an attempt to build it properly
> and blatantly states the following in README which demonstrates profound lack
> of understanding of problems that were impairing maintenance of the package:
>
>    "I kindly ask purist aspirations that effectively halted Kubernetes'
>     release and updates in Debian for YEARS to be kept at bay."
The new maintainer presumably thinks that Policy should have an
exception for this sort of case -- let's assume good faith.

> I don't consider myself to be a purist. I have pragmatic approach to
> packaging but I feel concerned when policies and packaging practices are
> circumvented.
>
> I don't recall a situation when policing of how a package is maintained would
> be necessary long after package is accepted...
> What do we do to ensure quality work in situation of technological hijack
> when maintainer is unwilling to follow the practice or comply with policy?
>
> This is not a technical disagreement so I think that involving technical
> committee may not be the right way to handle the problem... Or is it?
I think it counts as a technical disagreement but surely there is room
for discussion in a bug report before involving the T.C.

--
Sean Whitton

signature.asc (847 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: What to do when DD considers policy to be optional? [kubernetes]

Paul Wise via nm
On Tue, Mar 24, 2020 at 1:47 AM Sean Whitton wrote:

> Specifically, as README.Debian states, the vendor/ subdirectory of the
> source package contains more than two hundred Go libraries.

There are a *lot* of embedded code/data copies in Debian already.
While it would be nice to remove them, sometimes it isn't possible.
Often the copies are forked, or upstream refuses to remove them,
sometimes even though they forgot why they were added in the first
place. In addition the developer culture in various communities
encourages embedded copies. I think the best action we can do is send
patches to upstream projects to switch from vendoring to using the
native dependency system of the package manager for the related
language community. ISTR reading that Go has one of those now. Where
language communities don't have a native package manager, we need to
invent one for them. Then we can use things like dh-make-perl to
package the dependencies for Debian. I have no data but I think this
approach is more likely to have success than ranting about embedded
copies, tempting though that is. Apart from trying to discourage their
use, unfortunately embedded copies are here and they will never go
away and we need to accept that fact and to deal with the
consequences; for example to ensure that all copies get fixed for
security issues, try to get them updated upstream after important
bug/performance fixes and so on.

https://wiki.debian.org/EmbeddedCodeCopies
https://salsa.debian.org/security-tracker-team/security-tracker/raw/master/data/embedded-code-copies
https://wiki.debian.org/AutomaticPackagingTools

--
bye,
pabs

https://wiki.debian.org/PaulWise

Reply | Threaded
Open this post in threaded view
|

Re: What to do when DD considers policy to be optional? [kubernetes]

Vincent Bernat-3
 ❦ 24 mars 2020 03:11 +00, Paul Wise:

>> Specifically, as README.Debian states, the vendor/ subdirectory of the
>> source package contains more than two hundred Go libraries.
>
> There are a *lot* of embedded code/data copies in Debian already.
> While it would be nice to remove them, sometimes it isn't possible.
> Often the copies are forked, or upstream refuses to remove them,
> sometimes even though they forgot why they were added in the first
> place. In addition the developer culture in various communities
> encourages embedded copies. I think the best action we can do is send
> patches to upstream projects to switch from vendoring to using the
> native dependency system of the package manager for the related
> language community. ISTR reading that Go has one of those now.
Kubernetes is already using Go modules. They happen to have decided to
keep shipping a `vendor/` directory but this is not uncommon. It is
often considered as a protection against disappearing modules. So, there
is nothing to be done upstream. And BTW, there are currently 616
dependencies, pinned to a specific version.
--
Don't just echo the code with comments - make every comment count.
            - The Elements of Programming Style (Kernighan & Plauger)

signature.asc (847 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: What to do when DD considers policy to be optional? [kubernetes]

Holger Levsen-2
In reply to this post by Dmitry Smirnov
On Mon, Mar 23, 2020 at 05:32:54PM +1100, Dmitry Smirnov wrote:
> Something interesting just happened. An inexperienced DD adopted a very
> complicated package (kubernetes) and uploaded it with changes that would have
> never been accepted by ftp-masters.

file bugs then. maybe that new maintainer will be very happy about them being
pointed out?!

> This is not a technical disagreement so I think that involving technical
> committee may not be the right way to handle the problem... Or is it?

involving the TC without disagreements documented in the BTS seems premature.


--
cheers,
        Holger

-------------------------------------------------------------------------------
               holger@(debian|reproducible-builds|layer-acht).org
       PGP fingerprint: B8BF 5413 7B09 D35C F026 FE9D 091A B856 069A AA1C

"... the premise [is] that privacy is about hiding a wrong. It's not.
 Privacy is an inherent human right, and a requirement for maintaining
 the human condition with dignity and respect." (Bruce Schneier)

signature.asc (849 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: What to do when DD considers policy to be optional? [kubernetes]

Paul Wise via nm
In reply to this post by Vincent Bernat-3
On Tue, Mar 24, 2020 at 6:17 AM Vincent Bernat wrote:

> Kubernetes is already using Go modules. They happen to have decided to
> keep shipping a `vendor/` directory but this is not uncommon. It is
> often considered as a protection against disappearing modules. So, there
> is nothing to be done upstream. And BTW, there are currently 616
> dependencies, pinned to a specific version.

I wonder if the existence of Software Heritage could convince them
disappearing modules aren't a problem, or if another service is
needed.

--
bye,
pabs

https://wiki.debian.org/PaulWise

Reply | Threaded
Open this post in threaded view
|

Re: What to do when DD considers policy to be optional? [kubernetes]

Michael Lustfield
In reply to this post by Sean Whitton
On Mon, 23 Mar 2020 18:47:18 -0700
Sean Whitton <[hidden email]> wrote:

> Hello Dmitry, Janos, others,
>
> On Mon 23 Mar 2020 at 05:32PM +11, Dmitry Smirnov wrote:
>
> > What would be best to do in such situation?  
>
> [...]
>
> I think that I would start by filing an RC bug.

+1

If you run into issues, then you'll want to contact the ftp-masters team.

It would be helpful if the bug mentioned the two solutions I'm aware of:
- Revert the offending changes
- Migrate from main to non-free

The latter would be much easier, but would destroy all the work you put in. :(

> > [...]
> > As a person who originally introduced Kubernetes to Debian I can say that it
> > is indeed a lot of work to maintain this package and to reuse packaged
> > libraries. But I've demonstrated that it is possible at least to some extent.

As a person who temporarily introduced gitea into Debian, I fully understand.
Unfortunately, I've found that such problems are often a result of poorly
written code where the approach tends to be, "I don't know how to do this
thing, so I'll copy a module that does this and 200 other things just as
poorly." </rant>

The lesson I learned was-
Just because something /can/ be packaged, does not mean it /should/ be packaged.
(pabs warned me about this hundreds of hours prior to me giving up)

> > I don't recall a situation when policing of how a package is maintained would
> > be necessary long after package is accepted...

It's rare, but it happens. My most recent experience was with bitlbee.
Unfortunately, our current auto-reject system is quite limited. Catching things
like this automatically is (currently) not possible and Debian relies of folks
like you. (btw- thanks for this report)

> > What do we do to ensure quality work in situation of technological hijack
> > when maintainer is unwilling to follow the practice or comply with policy?
> >
> > This is not a technical disagreement so I think that involving technical
> > committee may not be the right way to handle the problem... Or is it?  

TC is not needed. This is a clear policy violation that the new maintainer
appears to have known about, even before the upload.

It concerns me that they thought this package warranted an exception...
--
Michael Lustfield

Reply | Threaded
Open this post in threaded view
|

Re: What to do when DD considers policy to be optional? [kubernetes]

Michael Lustfield
In reply to this post by Paul Wise via nm
On Tue, 24 Mar 2020 10:14:08 +0000
Paul Wise <[hidden email]> wrote:

> On Tue, Mar 24, 2020 at 6:17 AM Vincent Bernat wrote:
>
> > Kubernetes is already using Go modules. They happen to have decided to
> > keep shipping a `vendor/` directory but this is not uncommon. It is
> > often considered as a protection against disappearing modules. So, there
> > is nothing to be done upstream. And BTW, there are currently 616
> > dependencies, pinned to a specific version.  
>
> I wonder if the existence of Software Heritage could convince them
> disappearing modules aren't a problem, or if another service is
> needed.

I think this is a symptom of the tools being used. Using 'go vendor' is a
documented step in nearly all golang-based "release tutorials." Most never even
get as far as considering that maybe their source should have a version,
because the toolset mentality is "download latest at build time."

The 'go vendor' approach is especially bad within the Debian context because it
will download any/all modules that are referenced. In some cases, 'go get [..]'
can go from downloading a single repository to downloading 200+ because one (1)
extra dependency was added for one (1) extra feature that almost nobody will
ever use.

It's nearly guaranteed that at least a large handful of those will have no
license at all and at least one is going to have large embedded non-dfsg blobs.

Or, to summarize my rant...

These lazy young whipper snappers don't know what good source looks like!

.. back in my day, we coded on paper, had real bugs, and that's just the way we
liked it.

--
Michael Lustfield

Reply | Threaded
Open this post in threaded view
|

Re: What to do when DD considers policy to be optional? [kubernetes]

Vincent Bernat-3
In reply to this post by Paul Wise via nm
 ❦ 24 mars 2020 10:14 +00, Paul Wise:

>> Kubernetes is already using Go modules. They happen to have decided to
>> keep shipping a `vendor/` directory but this is not uncommon. It is
>> often considered as a protection against disappearing modules. So, there
>> is nothing to be done upstream. And BTW, there are currently 616
>> dependencies, pinned to a specific version.
>
> I wonder if the existence of Software Heritage could convince them
> disappearing modules aren't a problem, or if another service is
> needed.

There are other reasons, notably that you speed up builds by having all
the source code ready. If the vendor/ directory wasn't there, the
presence of `go.sum` would ensure you get something reproducible by
downloading all the deps, but I think it implies a full checkout of all
deps, which can be lengthy. There is also a caching mechanism (local and
remote).
--
Make sure every module hides something.
            - The Elements of Programming Style (Kernighan & Plauger)

signature.asc (847 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: What to do when DD considers policy to be optional? [kubernetes]

Vincent Bernat-3
In reply to this post by Michael Lustfield
 ❦ 24 mars 2020 05:37 -05, Michael Lustfield:

>> > Kubernetes is already using Go modules. They happen to have decided to
>> > keep shipping a `vendor/` directory but this is not uncommon. It is
>> > often considered as a protection against disappearing modules. So, there
>> > is nothing to be done upstream. And BTW, there are currently 616
>> > dependencies, pinned to a specific version.  
>>
>> I wonder if the existence of Software Heritage could convince them
>> disappearing modules aren't a problem, or if another service is
>> needed.
>
> I think this is a symptom of the tools being used. Using 'go vendor' is a
> documented step in nearly all golang-based "release tutorials." Most never even
> get as far as considering that maybe their source should have a version,
> because the toolset mentality is "download latest at build time."
With Go modules, that's not true anymore. It will use the minimal
version satisfying the minimal versions specified in go.mod.
--
Follow each decision as closely as possible with its associated action.
            - The Elements of Programming Style (Kernighan & Plauger)

signature.asc (847 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: What to do when DD considers policy to be optional? [kubernetes]

Julien Puydt-3
In reply to this post by Vincent Bernat-3
Le mardi 24 mars 2020 à 14:03 +0100, Vincent Bernat a écrit :
>  
> There are other reasons, notably that you speed up builds by having
> all
> the source code ready.

Sorry, I don't know much about how go works, but : can't the developer
just have the deps ready -- and just not commit them to the repo and
not ship them?

>From what I have read in this thread, go developers seem to be about as
sloppy as javascript ones : put junk together and throw it to the
world...

J.Puydt

Reply | Threaded
Open this post in threaded view
|

Re: What to do when DD considers policy to be optional? [kubernetes]

Vincent Bernat-3
 ❦ 24 mars 2020 14:18 +01, Julien Puydt:

>> There are other reasons, notably that you speed up builds by having
>> all the source code ready.
>
> Sorry, I don't know much about how go works, but : can't the developer
> just have the deps ready -- and just not commit them to the repo and
> not ship them?

These projects heavily rely on automated builds. Depending on platform,
it may or may not be easy to have shared cache for these builds. If they
have, you have to debug them as well. From their point of view, it's
simpler to deliver the artifacts with the rest of the source code.
Fast, simple and more reliable.

Moreover, the vendor directory is absolutely not a problem for Debian
(except for licensing where you would have to do a repack for stuff not
distributable). Debian just has to ignore the vendor directory and have
all the dependencies ready. The tooling around Go in Debian already
handles that. The problem is the number of dependencies.

> From what I have read in this thread, go developers seem to be about as
> sloppy as javascript ones : put junk together and throw it to the
> world...

That's not comparable. Go is not using micro-dependencies. However, they
don't have optional dependencies, so anything cloud-based has a lot of
dependencies to manage.
--
Make it clear before you make it faster.
            - The Elements of Programming Style (Kernighan & Plauger)

signature.asc (847 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: What to do when DD considers policy to be optional? [kubernetes]

Sean Whitton
In reply to this post by Michael Lustfield
Hello,

On Tue 24 Mar 2020 at 05:37AM -05, Michael Lustfield wrote:

> The 'go vendor' approach is especially bad within the Debian context because it
> will download any/all modules that are referenced. In some cases, 'go get [..]'
> can go from downloading a single repository to downloading 200+ because one (1)
> extra dependency was added for one (1) extra feature that almost nobody will
> ever use.
>
> It's nearly guaranteed that at least a large handful of those will have no
> license at all and at least one is going to have large embedded non-dfsg blobs.
>
> Or, to summarize my rant...
>
> These lazy young whipper snappers don't know what good source looks like!
>
> .. back in my day, we coded on paper, had real bugs, and that's just the way we
> liked it.
If there are multiple DFSG issues then I would say that this is more
than a rant.

--
Sean Whitton

signature.asc (847 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: What to do when DD considers policy to be optional? [kubernetes]

Florian Weimer
In reply to this post by Paul Wise via nm
* Paul Wise:

> On Tue, Mar 24, 2020 at 6:17 AM Vincent Bernat wrote:
>
>> Kubernetes is already using Go modules. They happen to have decided to
>> keep shipping a `vendor/` directory but this is not uncommon. It is
>> often considered as a protection against disappearing modules. So, there
>> is nothing to be done upstream. And BTW, there are currently 616
>> dependencies, pinned to a specific version.
>
> I wonder if the existence of Software Heritage could convince them
> disappearing modules aren't a problem, or if another service is
> needed.

The required versions of the modules could disappear from Debian, though.
Services like Software Heritage will not help with that.

Reply | Threaded
Open this post in threaded view
|

Re: What to do when DD considers policy to be optional? [kubernetes]

LENART Janos
In reply to this post by Sean Whitton
Hi Dimitry, FTP masters and others,

I know Dimitry was fighting an uphill battle with kubernetes between 2016 and 2018 and he experienced first hand the problems posed by vendored code.

We see more and more software making excessive use of vendored code. Pretty much everything that is written in Go. Some of these are crucially important, like Docker or Kubernetes. So I understand the concern everyone has about how this fits with the Debian Policy.

Debian Policy, paragraph 4.13 states:
(for your convenience I include it below :) )

=================
4.13 Convenience copies of code

Some software packages include in their distribution convenience copies of code from other software packages, generally so that users compiling from source don’t have to download multiple packages. Debian packages should not make use of these convenience copies unless the included package is explicitly intended to be used in this way. [17] If the included code is already in the Debian archive in the form of a library, the Debian packaging should ensure that binary packages reference the libraries already in Debian and the convenience copy is not used. If the included code is not already in Debian, it should be packaged separately as a prerequisite if possible. [18]

[18] Having multiple copies of the same code in Debian is inefficient, often creates either static linking or shared library conflicts, and, most importantly, increases the difficulty of handling security vulnerabilities in the duplicated code.
=================

I think this is the part that has the most bearing on the vendored code problem, especially the footnote. I agree with this principle. But we should apply it to the state of affairs in 2020, and to this specific situation.

Keeping all that in mind, here are the reasons why I think it is acceptable for now to package Kubernetes with the vendored code, and even the best solution that is available currently:

1. OTHER EXAMPLES. If we take this paragraph completely literally and to the extreme then other packages are also in violation of it. True, the current packaging of kubernetes does this to a greater extent than its predecessor for example, but perhaps this shows that this section was always open for interpretation. Examples of some prominent packages in Debian that bundle and use the vendored code (in parentheses is the number of go packages bundled, estimate):
- docker.io (58, including some that are vendored more than once within the same source package, but not including the fact that docker.io itself is made up of 7 tarballs)
- kubernetes (20 for the previous version, 200 now)
- prometheus (4)
- golang (4)
None of these were REJECTed, and please don't sabotage these packages now :-D The idea was only to show that, at least for now, vendoring is a fact in Debian. There is an effort to improve the situation but in the meantime we just go on. Not great, not terrible..

2. MAINTAINABILITY. Having every single vendored repo available as a separate package in Debian is not feasible. It is true that some of them are already packaged. But the expectation that all of them are (with the exact version that is needed for Kubernetes), is not going to happen. Also, the golang-* packages have a number of different maintainers. Hundreds of such packages would be required to build Kubernetes. So one can be rest assured that every future release in Debian will be blocked on waiting for dozens of these packages to be updated. Dimitry and a few others worked hard on trying to pull this off but even they could not do it. Since 2016 a total of 3 Kubernetes releases made it into Debian/unstable, but there have been 17 major and countless minor upstream releases of Kubernetes. Thousands of issues were fixed upstream, including serious security flaws, these never made it into Debian. Exactly because the packaging was too difficult to maintain. So, how maintainable was that solution then, despite the huge amount of effort put in? In my opinion this shows that the reasoning on maintainability in DP does not apply here.

3. NO FORKS. Debian developers hacking Kubernetes source code, so it compiles with a lucky enough version of a dependency that made it into Debian, makes the Debian version of Kubernetes different from the standard one that everyone expects. This is totally unwelcome by almost every user. No sane cluster admin would dare to use this "fork", ever. There were some attempts to get the Kubernetes contributors to update dependencies to a specific version: https://github.com/kubernetes/kubernetes/issues/27543 . Reading the whole thread helps to put some perspective on this. The Kubernetes contributors were actually quite helpful throughout but they have made it clear that they will not update dependencies for update's sake. Maybe with some projects Debian would have the upper hand, but not with Kubernetes.

4. TESTING. The Kubernetes releases are meticulously tested, with far greater technical resources that Debian can collectively muster. The Kubernetes project runs e2e tests regularly on thousands of nodes (donated compute time). If we were to continue to have a fork we would be obliged to do the same. Even if we could run such extensive tests on our fork, and these e2e tests revealed a problem, who is going to interpret the results and fix our snowflake? The debian fork was never tested this way and it seems unlikely that it could ever be.

5. SECURITY. The strongest and most applicable point from the DP footnote is about security vulnerabilities in the duplicated code. This is completely valid. But again, with the maintainability issues (see 2) we won't be able to roll out security fixes in time. How did security in the Debian forks created by DDs worked in the past? https://www.explainxkcd.com/wiki/index.php/424:_Security_Holes . It is true that without listing the bundled dependencies in Built-Using, it is harder to find out if a vulnerability in one of them affects the binary. (Hint: it is hard anyway.) In the case of Kubernetes, and other Go programs in general, an automated tool could be made that extracts go.mod/go.sum for monitoring the dependencies for security vulnerability reports. Doing the whole dance of let's package and maintain hundreds of dependencies so we have a machine readable Built-Using instead of a machine readable go.mod/go.sum seems a lot more harm than good for security. Furthermore the current situation forces users to add third party repos to sources.list to get up to date Kubernetes releases and/or download who-knows-whats-in-it binaries. So this is not great, but the alternative is terrible.

6. EFFICIENCY. Go libraries, vendored or not, are essentially statically linked into the binary. This is still the case when the result is a "dynamic" Go binary, e.g. linked to libc. While there are some experiments for shared libraries in Go, there is no real world use. This means that vendoring has no effect on linking behavior so the whole point is beside the issue.

7. DFSG. I am not aware of any DFSG issues in the vendored packages. No funny licenses, blobs, network downloaded stuff, etc. If there are any, please point it out specifically, and it will be fixed with high priority. I have checked the licences of every dependency and have compiled it in a container with no network access, so what's in the .orig.tar.gz is exactly what was compiled, nothing more.

I do think there is a good case for Kubernetes to be an exception from 4.13 for now, just like other Go packages effectively are. It is a massively popular project topped only by the Linux kernel. We cannot afford not to have up to date versions in Debian, or have forks that no one can use.

So let's find a way to make this happen!

Regards,
-- 
LENART, János
<[hidden email]>
Reply | Threaded
Open this post in threaded view
|

Re: What to do when DD considers policy to be optional? [kubernetes]

Jeremy Stanley
On 2020-03-24 19:08:23 +0000 (+0000), Janos LENART wrote:
> I know Dimitry was fighting an uphill battle with kubernetes
> between 2016 and 2018 and he experienced first hand the problems
> posed by vendored code.
>
> We see more and more software making excessive use of vendored
> code. Pretty much everything that is written in Go. Some of these
> are crucially important, like Docker or Kubernetes. So I
> understand the concern everyone has about how this fits with the
> Debian Policy.
[snip rationale for packaging Kubernetes while giving up on policy]

If this represents the actual state of building Kubernetes, it's
unclear to me why Debian would package it at all. I don't see the
value to users in consuming Kubernetes from a Debian package if the
result is compromising on Debian's vision and values so that they
can get the exact same thing they'd have if they just used the
Kubernetes community's recommended tooling to install it instead.
I'm all for using the best tool for the job, and while I've been a
die-hard Debian user for more than two decades I also don't install
every last bit of software from its packages. Some software
ecosystems have chosen to focus on tools and workflows which are
incompatible with Debian's, but that doesn't mean that either one is
inherently bad nor that they need to be integrated at all costs.
--
Jeremy Stanley

signature.asc (981 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: What to do when DD considers policy to be optional? [kubernetes]

Russ Allbery-2
Jeremy Stanley <[hidden email]> writes:

> If this represents the actual state of building Kubernetes, it's
> unclear to me why Debian would package it at all. I don't see the
> value to users in consuming Kubernetes from a Debian package if the
> result is compromising on Debian's vision and values so that they
> can get the exact same thing they'd have if they just used the
> Kubernetes community's recommended tooling to install it instead.

I have been very grateful over the past few months to have Kubernetes
available in Debian (and have been quite annoyed at the irritating things
I have to do get and update Helm, which at least has a snap, and Argo CD,
which doesn't have anything useful).

I have no opinion about the best solution to the huge vendoring problem.
(The Rust team is trying the package everything approach with some success
but is uncovering other limitations in our processes and tools.)  But
having these tools in Debian is hugely valuable for Debian users who need
them, which is sort of the point of Debian at the end of the day.

> I'm all for using the best tool for the job, and while I've been a
> die-hard Debian user for more than two decades I also don't install
> every last bit of software from its packages.

I do when I can because I otherwise have to remember to wire together N
different update mechanisms, which is remarkably not-fun and takes time
away from the actual work I'm trying to do.

--
Russ Allbery ([hidden email])              <https://www.eyrie.org/~eagle/>

Reply | Threaded
Open this post in threaded view
|

Re: new kubernetes packaging

Sean Whitton
In reply to this post by LENART Janos
Hello Janos,

Thank you for your e-mail.  I agree with you that security support is
the most pressing reason to avoid piles of vendored code, and you make
an interesting argument regarding how it can be difficult to provide
security fixes if our refusal to use vendored code means we lag too far
behind upstream.

I am not sure, however, that your argument applies to security updates
to our stable releases.  These updates are almost always a matter of
backporting small fixes, rather than updating to new upstream releases.
And for backported fixes, vendoring makes things much harder.

You also write:

On Tue 24 Mar 2020 at 07:08PM +00, Janos LENART wrote:

> 1. OTHER EXAMPLES. If we take this paragraph completely literally and to
> the extreme then other packages are also in violation of it. True, the
> current packaging of kubernetes does this to a greater extent than its
> predecessor for example, but perhaps this shows that this section was
> always open for interpretation. Examples of some prominent packages in
> Debian that bundle and use the vendored code (in parentheses is the number
> of go packages bundled, estimate):
> - docker.io (58, including some that are vendored more than once within the
> same source package, but not including the fact that docker.io itself is
> made up of 7 tarballs)
> - kubernetes (20 for the previous version, 200 now)
> - prometheus (4)
> - golang (4)
but I am not sure this is relevant because the number of vendored copies
in the new kubernetes package is an order of magnitude larger than any
of these examples.

Finally, I would like to hear why you think it is valuable for us to
have a package like this in Debian as opposed to expecting people to
install it from upstream:

On Tue 24 Mar 2020 at 08:37PM +00, Jeremy Stanley wrote:

> If this represents the actual state of building Kubernetes, it's
> unclear to me why Debian would package it at all. I don't see the
> value to users in consuming Kubernetes from a Debian package if the
> result is compromising on Debian's vision and values so that they
> can get the exact same thing they'd have if they just used the
> Kubernetes community's recommended tooling to install it instead.
> I'm all for using the best tool for the job, and while I've been a
> die-hard Debian user for more than two decades I also don't install
> every last bit of software from its packages. Some software
> ecosystems have chosen to focus on tools and workflows which are
> incompatible with Debian's, but that doesn't mean that either one is
> inherently bad nor that they need to be integrated at all costs.
I find this persuasive.

--
Sean Whitton

signature.asc (847 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: new kubernetes packaging

Russ Allbery-2
Sean Whitton <[hidden email]> writes:

> Thank you for your e-mail.  I agree with you that security support is
> the most pressing reason to avoid piles of vendored code, and you make
> an interesting argument regarding how it can be difficult to provide
> security fixes if our refusal to use vendored code means we lag too far
> behind upstream.

> I am not sure, however, that your argument applies to security updates
> to our stable releases.  These updates are almost always a matter of
> backporting small fixes, rather than updating to new upstream releases.
> And for backported fixes, vendoring makes things much harder.

I think this calculus is not entirely obvious.

What you say is true if the library is used by multiple applications in
Debian (although it's still not as good of a story with Go as it is for
C).  We can backport a patch to that one library, and then rebuild the
applications that incorporate it.

However, if a library exists in Debian solely because it is a dependency
of some sprawling application and isn't used by other things, it may be
easier to do a security update if it's vendored.  There are, at the least,
fewer packages to rebuild, and the testing story is slightly more
straightforward.

Possibly more significantly is how the flow of security advisories work.
If the advisory is likely to come from Kubernetes and their security fix
release is a point release update to their package including the vendored
modules, we can potentially adopt the "sane upstream stable point release"
policy and just update stable to their point release.  (Kubernetes does
maintain long-lived stable branches, although I don't know how stringent
they are about what changes they're willing to take in the stable
branches.)  In this case, we create a bit more security work by separately
packaging the dependencies, since we now have to trace down the package
that corresponds to a Kubernetes security advisory and update it.

This of course doesn't apply if the individual libraries are releasing
their own security advisories.

--
Russ Allbery ([hidden email])              <https://www.eyrie.org/~eagle/>

Reply | Threaded
Open this post in threaded view
|

Re: new kubernetes packaging

Sean Whitton
Hello,

On Tue 24 Mar 2020 at 03:14PM -07, Russ Allbery wrote:

> What you say is true if the library is used by multiple applications in
> Debian (although it's still not as good of a story with Go as it is for
> C).  We can backport a patch to that one library, and then rebuild the
> applications that incorporate it.
>
> However, if a library exists in Debian solely because it is a dependency
> of some sprawling application and isn't used by other things, it may be
> easier to do a security update if it's vendored.  There are, at the least,
> fewer packages to rebuild, and the testing story is slightly more
> straightforward.
Right, if something is technically an independent language module but is
used only by kubernetes, and we don't expect that to change, there's no
need for it to be in its own source package just because it might seem
tidier to us.

I doubt that this is true of all the hundreds of dependencies presently
bundled with kubernetes, however.

> Possibly more significantly is how the flow of security advisories work.
> If the advisory is likely to come from Kubernetes and their security fix
> release is a point release update to their package including the vendored
> modules, we can potentially adopt the "sane upstream stable point release"
> policy and just update stable to their point release.  (Kubernetes does
> maintain long-lived stable branches, although I don't know how stringent
> they are about what changes they're willing to take in the stable
> branches.)  In this case, we create a bit more security work by separately
> packaging the dependencies, since we now have to trace down the package
> that corresponds to a Kubernetes security advisory and update it.
This is certainly a reasonable approach, if such releases aren't going
to violate the expectations of users of Debian stable.

I guess we'd have to see whether the Security Team are up for another
package which gets updated in this way.

--
Sean Whitton

signature.asc (847 bytes) Download Attachment
1234 ... 6