Proposal: Repository for fast-paced package backports

classic Classic list List threaded Threaded
68 messages Options
1234
Reply | Threaded
Open this post in threaded view
|

Proposal: Repository for fast-paced package backports

Dominik George-7
Heisann, alle sammen,

as announced in the recent thread about maintaining, I hereby propose a
repository that allows making “backports” of packages available to users
of the stable distribution, if those packages cannot be maintained in
testing and backported in the usual way. If you are interested in what
lead up to that, please see bug #915050. I will give a short summary of
it here.


Reasons for having a special place for some packages
====================================================

(You may want to skip this part if you are familiar with the situation.)

As all developers know (but passers-by may not), for software to enter
the Debian archive, it is always uploaded to the unstable distribution,
then migrates to testing (hopefully ;)), which is at some point snapshot
and made the new stable release. From there on, maintainers have two
obligations: Firstly, keep the package in stable good and secure, e.g.
by uploading security fixes for it once they become available upstream,
or even backport fixes themselves. Secondly, provide the package in
unstable with updates and ensure its migration, to keep it ready for the
next stable release.

Now, for some software packages, this process is problematic, because
upstream may have another idea about software lifecycles. Concerning the
GitLab example, upstream provides security fixes for three months for
their stable releases. Backporting fixes from newer versions is very
hard or impossible because the massive amounts of changes to the
software in every new versions. This is something that also affects
other packages, like Mozilla Firefox, which has a firefox package in
unstable, and a separate firefox-esr package, with the ESR version of
Firefox. Only the latter migrates to testing.

Users of Debian honour it for its stability, but as an agile software
lifecycle is adapted by more and more very popular software packages,
not being able to install these packages in the trusted, well-known
fashion through the official apt repositories is becoming more and more
of a drawback.

It can easily be assumed that the normal release and maintenance cycle
of Debian stable will not change, which is very good, so we should find
a way to still provide such software as described above to users.


Why backports is not enough
===========================

This also is well-known, but for completeness: Formal backports in
stable-backports are required to be direct backports from testing, and
are a stepping stone within the upgrade from stable to stable+1. Thus, a
version of a package that is not in testing can never be in
stable-backports.


Name of the new repository
==========================

In the past, the name “volatile” was used for a similar repository, but
with a different scope (limited to data packages for things like virus
scanners). I will thus use the working title volatile throughout this
proposal, although this may change.

Other ideas: fastlane, unsupported

(Please feel free to add other ideas.)


Requirements for a package to go into stable-volatile
=====================================================

The new volatile proposal is not intended to ease life for package
maintainers who want to bypass the migration and QA requirements of the
regular stable lifecycle, so special need must be taken to ensure only
packages that need it go into volatile. I want to summarise the
requirements like so:

 - The package must be maintained in unstable, like every other package.
 - The package must not be in testing, and care must be taken for the
   package not to migrate to testing.
 - Regular maintenance for the lifetime of stable must be impossible
   or unnecessarily hard, and this requirement should be assessed in
   a verifiable manner, e.g. referring to upstream’s lifecycle model.
 - There must be notable need for the package. Like for backports, user
   requests might be an indicator.
 - Should the package be removed from unstable, it must also be removed
   from volatile.
 - Should the package begin to migrate to testing again, it must
   be moved to stable-backports.

Before starting to maintain a volatile package, the maintainer shall
seek consent (or doubt) on debian-devel.


Building packages and package dependencies
==========================================

Packages for volatile are built the same way as formal backports, only
that the source is taken from unstable rather than testing. In
particular:

 - Changes shall be kept as small as possible.
 - The package is rebuilt against stable.
 - The package may depend on packages in stable, stable-backports or stable-volatile.

Dependencies on other packages in volatile should be avoided if
possible. Especially, dependencies of the package that also need
backporting must not be added to volatile just because they are
dependencies — every dependency that is needed to be backported to
support the volatile package must be considered on its own and in all
but unprobable edge cases be maintained as a formal backport. Obviously,
the unprobable edge case occurs when the package depends on another
package that also fully qualifies for volatile, as described above.


Versions of packages in volatile
================================

I am not yet certain about this. As stressed before, volatile should be
an extension of backports, so starting with the well-known backports
suffix ~bpoN seems reasonable. I’d even say this is enough, as a package
is never both in volatile and backports, and if maintenance changes to
the regular lifecycle, it can easily be moved to backports.


Responsibility and location of the repository
=============================================

I propose to add the volatile repository next to the backports
repository, and treat it as part of backports. This should not impose
new workload on the backports ftp-masters, so this needs people who
volunteer to do the extra work. It should, however, be not too much of a
workload anyway, as the number of packages qualifying for volatile is
quite limited. (I do volunteer for the backports team, not only for my
own proposal, but also in general.)

This implies that new binary uploads to volatile have to undergo the
same NEW queue as backports.


volatile repositories for other distributions
=============================================

You guessed it: Same as for backports, but in green ;).


Technical requirements
======================

Apart from the new section in the repository, care needs to be taken to
ensure removal from volatile if a package moves to -backports again. The
mechanisms used for decrufting experimental might apply.


Implications for the situation at hand (gitlab)
===============================================

As there were quite a few concerns raised (some of which I share, and
some I don’t): Of course, if a software intended for volatile has a ton
of dependencies (intended to go into backports), all backports rules and
powers of the ftp-masters apply. Repeating myself: volatile is not meant
to ease the life of maintainers.



I ask the community, the backports team and the release team for their opinions.

Cheers, ha det bra,
Nik

signature.asc (919 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Proposal: Repository for fast-paced package backports

Alexander Wirt
On Tue, 25 Dec 2018, Dominik George wrote:

> Heisann, alle sammen,
>
> as announced in the recent thread about maintaining, I hereby propose a
> repository that allows making “backports” of packages available to users
> of the stable distribution, if those packages cannot be maintained in
> testing and backported in the usual way. If you are interested in what
> lead up to that, please see bug #915050. I will give a short summary of
> it here.
>
>
> Reasons for having a special place for some packages
> ====================================================
>
> (You may want to skip this part if you are familiar with the situation.)
>
> As all developers know (but passers-by may not), for software to enter
> the Debian archive, it is always uploaded to the unstable distribution,
> then migrates to testing (hopefully ;)), which is at some point snapshot
> and made the new stable release. From there on, maintainers have two
> obligations: Firstly, keep the package in stable good and secure, e.g.
> by uploading security fixes for it once they become available upstream,
> or even backport fixes themselves. Secondly, provide the package in
> unstable with updates and ensure its migration, to keep it ready for the
> next stable release.
>
> Now, for some software packages, this process is problematic, because
> upstream may have another idea about software lifecycles. Concerning the
> GitLab example, upstream provides security fixes for three months for
> their stable releases. Backporting fixes from newer versions is very
> hard or impossible because the massive amounts of changes to the
> software in every new versions. This is something that also affects
> other packages, like Mozilla Firefox, which has a firefox package in
> unstable, and a separate firefox-esr package, with the ESR version of
> Firefox. Only the latter migrates to testing.
>
> Users of Debian honour it for its stability, but as an agile software
> lifecycle is adapted by more and more very popular software packages,
> not being able to install these packages in the trusted, well-known
> fashion through the official apt repositories is becoming more and more
> of a drawback.
>
> It can easily be assumed that the normal release and maintenance cycle
> of Debian stable will not change, which is very good, so we should find
> a way to still provide such software as described above to users.
>
>
> Why backports is not enough
> ===========================
>
> This also is well-known, but for completeness: Formal backports in
> stable-backports are required to be direct backports from testing, and
> are a stepping stone within the upgrade from stable to stable+1. Thus, a
> version of a package that is not in testing can never be in
> stable-backports.
>
>
> Name of the new repository
> ==========================
>
> In the past, the name “volatile” was used for a similar repository, but
> with a different scope (limited to data packages for things like virus
> scanners). I will thus use the working title volatile throughout this
> proposal, although this may change.
>
> Other ideas: fastlane, unsupported
>
> (Please feel free to add other ideas.)
>
>
> Requirements for a package to go into stable-volatile
> =====================================================
>
> The new volatile proposal is not intended to ease life for package
> maintainers who want to bypass the migration and QA requirements of the
> regular stable lifecycle, so special need must be taken to ensure only
> packages that need it go into volatile. I want to summarise the
> requirements like so:
>
>  - The package must be maintained in unstable, like every other package.
>  - The package must not be in testing, and care must be taken for the
>    package not to migrate to testing.
>  - Regular maintenance for the lifetime of stable must be impossible
>    or unnecessarily hard, and this requirement should be assessed in
>    a verifiable manner, e.g. referring to upstream’s lifecycle model.
>  - There must be notable need for the package. Like for backports, user
>    requests might be an indicator.
>  - Should the package be removed from unstable, it must also be removed
>    from volatile.
>  - Should the package begin to migrate to testing again, it must
>    be moved to stable-backports.
>
> Before starting to maintain a volatile package, the maintainer shall
> seek consent (or doubt) on debian-devel.
>
>
> Building packages and package dependencies
> ==========================================
>
> Packages for volatile are built the same way as formal backports, only
> that the source is taken from unstable rather than testing. In
> particular:
>
>  - Changes shall be kept as small as possible.
>  - The package is rebuilt against stable.
>  - The package may depend on packages in stable, stable-backports or stable-volatile.
>
> Dependencies on other packages in volatile should be avoided if
> possible. Especially, dependencies of the package that also need
> backporting must not be added to volatile just because they are
> dependencies — every dependency that is needed to be backported to
> support the volatile package must be considered on its own and in all
> but unprobable edge cases be maintained as a formal backport. Obviously,
> the unprobable edge case occurs when the package depends on another
> package that also fully qualifies for volatile, as described above.
>
>
> Versions of packages in volatile
> ================================
>
> I am not yet certain about this. As stressed before, volatile should be
> an extension of backports, so starting with the well-known backports
> suffix ~bpoN seems reasonable. I’d even say this is enough, as a package
> is never both in volatile and backports, and if maintenance changes to
> the regular lifecycle, it can easily be moved to backports.
>
>
> Responsibility and location of the repository
> =============================================
>
> I propose to add the volatile repository next to the backports
> repository, and treat it as part of backports. This should not impose
> new workload on the backports ftp-masters, so this needs people who
> volunteer to do the extra work. It should, however, be not too much of a
> workload anyway, as the number of packages qualifying for volatile is
> quite limited. (I do volunteer for the backports team, not only for my
> own proposal, but also in general.)
>
> This implies that new binary uploads to volatile have to undergo the
> same NEW queue as backports.
>
>
> volatile repositories for other distributions
> =============================================
>
> You guessed it: Same as for backports, but in green ;).
>
>
> Technical requirements
> ======================
>
> Apart from the new section in the repository, care needs to be taken to
> ensure removal from volatile if a package moves to -backports again. The
> mechanisms used for decrufting experimental might apply.
>
>
> Implications for the situation at hand (gitlab)
> ===============================================
>
> As there were quite a few concerns raised (some of which I share, and
> some I don’t): Of course, if a software intended for volatile has a ton
> of dependencies (intended to go into backports), all backports rules and
> powers of the ftp-masters apply. Repeating myself: volatile is not meant
> to ease the life of maintainers.
>
>
>
> I ask the community, the backports team and the release team for their opinions.
We already told you to build your own repo. Please don't mix that with
backports.

Imho you should start the same way backports started - outside of debian.
Prove that it works and integrate into Debian later.

Alex

signature.asc (849 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Proposal: Repository for fast-paced package backports

Dominik George-7
> We already told you to build your own repo.

You should probably start with identifying the senders of mail
correctly ☺. I am not the gitlab maintainer (and will never be).

> Imho you should start the same way backports started - outside of
> debian.
> Prove that it works and integrate into Debian later.

I would agree with you if it were a big change - however, the proposal
has a very low impact, if not none at all, on existing stuff. In
contrast to what you seem to believe (accuse people of…), this proposal
is about helping Debian as a whole, not forcing a certain package into
the distribution. gitlab only serves as an example of why it is useful.
The Debian infrastructure already supports everything that is needed to
implement this, and starting with parallel infrastructure would probably
mean that it will fail because this requires a single person spending
time and money to maintain the infrastructure (which is otherwise
already there), and to make it really work, this is a low (think of
buildds, etc.).

In any case, I do not see why you would fight the fact that someone
makes a detailed proposal. A proposal can be accepted or denied, of
course, but your tone implies you think noone should have made the
proposal i nthe first place.

Please don't fight people wanting to help based on your opinion about a
prior case around gitlab.

-nik

signature.asc (919 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Proposal: Repository for fast-paced package backports

Alexander Wirt
On Tue, 25 Dec 2018, Dominik George wrote:

> > We already told you to build your own repo.
>
> You should probably start with identifying the senders of mail
> correctly ☺. I am not the gitlab maintainer (and will never be).
https://lists.debian.org/debian-backports/2018/12/msg00028.html

This wasn't about gitlab.

>
> > Imho you should start the same way backports started - outside of
> > debian.
> > Prove that it works and integrate into Debian later.
>
> I would agree with you if it were a big change - however, the proposal
> has a very low impact, if not none at all, on existing stuff. In
> contrast to what you seem to believe (accuse people of…), this proposal
> is about helping Debian as a whole, not forcing a certain package into
> the distribution. gitlab only serves as an example of why it is useful.
> The Debian infrastructure already supports everything that is needed to
> implement this, and starting with parallel infrastructure would probably
> mean that it will fail because this requires a single person spending
> time and money to maintain the infrastructure (which is otherwise
> already there), and to make it really work, this is a low (think of
> buildds, etc.).
>
> In any case, I do not see why you would fight the fact that someone
> makes a detailed proposal. A proposal can be accepted or denied, of
> course, but your tone implies you think noone should have made the
> proposal i nthe first place.
>
> Please don't fight people wanting to help based on your opinion about a
> prior case around gitlab.
I don't fight against it. I just want to keep it away from backports, thats
not the same.

Alex


signature.asc (849 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Proposal: Repository for fast-paced package backports

Dominik George-7
On Tue, Dec 25, 2018 at 10:11:43PM +0100, Alexander Wirt wrote:
> https://lists.debian.org/debian-backports/2018/12/msg00028.html
>
> This wasn't about gitlab.

Oh. I must have misread the "gitlab" in the subject, along withthe mail
being sent to the gitlab maintainer, a gitlab bugreport in the BTS, and
concerning a request to accept gitlab into backports ;).

Still, there's a big difference:

 * The thread you refer to is about uploading to backports. This proposal
   ia about *not* uploading to backports. The newly-proposed section is
   only intended to co-exist with backports, and interact nicely with
   backports. (Mind the difference between backport as a general term
   for a package made available for an older distribution, and the name
   backports for a section in the Debian repository).
 * Your mail you are referring to talks about "backports" from unstable
   being a different workflow - this proposal proposes such a workflow.
 * Your mail refers to packages being indistinguishable in -backports -
   this proposal is all about having a new section in the repository to
   distinguish them.

In short: This proposal addresses the exact concerns you raised before
)although I am not the person you expressed them towards).

-nik

signature.asc (919 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Proposal: Repository for fast-paced package backports

Dominik George-7
> In short: This proposal addresses the exact concerns you raised before
> )although I am not the person you expressed them towards).

Well, sure, I was involved in that thread, but only in the way that I
announced a proposal (this one). Not in any of the stuff concerning
adding something to -backports.

-nik

signature.asc (919 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Proposal: Repository for fast-paced package backports

Micha Lenk-9
In reply to this post by Dominik George-7
Hi all,

having read the whole Gitlab discussion, I still don't get how/why the new repository depends or relates to backports. Instead it could be self-contained, except for stuff already available in stable. Couldn't you roll the new repository entirely independent of any backports? Even if you say there won't be any additional work for the backport policy owners, letting a new repo depend on backports will implicitly have an impact, which doesn't sound fully thought through yet.

I consider especially copying parts of the version scheme fairly confusing. This gives your concept a bad touch of just trying to work around established rules (i.e. backports rules). Instead of defining such minor facets I would recommend you to work on clarity about what rules you want to establish in the new repo instead.

Also, as Alex suggested, I would prefer if such experiments could be started outside the official Debian archive, like backports once successfully did. Given how much efforts it took to get backports integrated officially, I don't consider adding a new repo a minor change. Did you discuss your idea with ftp masters, dak maintainers, and buildd admins before?

I acknowledge that Debian needs a solution to support fast moving projects like Gitlab better than now. Yet, without a *proof* of concept how this could work out in the long run (i.e. across more than one Debian release cycle), I don't think it is the right time to ask for such a big change now. I consider Debian open enough to support such concepts outside the official archive first.

Kind regards,
Micha

Reply | Threaded
Open this post in threaded view
|

Re: Proposal: Repository for fast-paced package backports

Dominik George-7
Hi,

>having read the whole Gitlab discussion, I still don't get how/why the
>new repository depends or relates to backports. Instead it could be
>self-contained, except for stuff already available in stable. Couldn't
>you roll the new repository entirely independent of any backports? Even
>if you say there won't be any additional work for the backport policy
>owners, letting a new repo depend on backports will implicitly have an
>impact, which doesn't sound fully thought through yet.

This is answered in the proposal. The reason is to not have volatile abused to ease backporting, and to allow packages to easily move back to backports again.

>I consider especially copying parts of the version scheme fairly
>confusing. This gives your concept a bad touch of just trying to work
>around established rules (i.e. backports rules). Instead of defining
>such minor facets I would recommend you to work on clarity about what
>rules you want to establish in the new repo instead.

I am a bit disappointed that my efforts to emphasize good compatibility with established processes is interpreted that way.

As I already laid out several times during the last days, I am in fact disappointed that assuming bad or egoistic intentions seems to have become normal in Debian.

That said, the version numbering is a way to ensure work *with* established rules, not around.

>Also, as Alex suggested, I would prefer if such experiments could be
>started outside the official Debian archive, like backports once
>successfully did. Given how much efforts it took to get backports
>integrated officially, I don't consider adding a new repo a minor
>change. Did you discuss your idea with ftp masters, dak maintainers,
>and buildd admins before?

I did not discuss this proposal before discussing this proposal, no. That's why I am discussing this proposal :).

If you read it properly, you will find that does not add anything really new, but extends something existing - yet without interfering with it much.

>I acknowledge that Debian needs a solution to support fast moving
>projects like Gitlab better than now. Yet, without a *proof* of concept
>how this could work out in the long run (i.e. across more than one
>Debian release cycle), I don't think it is the right time to ask for
>such a big change now.

Again, the change is not new - it is an extension of backports, using the exact same concepts and rules, apart from the source distribution and the target directory. It is an extension designed to play very nicely with backports.

> I consider Debian open enough to support such
>concepts outside the official archive first.

I hope that e.g. official buildds will not grab code from my private machine and build it, for example.

-nik

Reply | Threaded
Open this post in threaded view
|

Re: Proposal: Repository for fast-paced package backports

Dominik George-7
In reply to this post by Dominik George-7
Hi,

I like the general direction, but there are some aspects of your
>proposal
>which should be improved.

Thanks!

>> Other ideas: fastlane, unsupported
>
>Or maybe something like "fastpaced", after all this repo would not be
>unsupported at all, the very point is to provide actual support after
>all.

I actually think volatile is a good name. After all, it's not so far from the previous volatile.

>>  - The package must be maintained in unstable, like every other
>package.
>
>Given the nature of the packages in "fastpaced", it's counterproductive
>to mandate the same standards as for the standard archive, it rather
>makes
>sense to relax some aspects.
>
>E.g. we usually try to avoid embedded code copies. But for a package
>like Gitlab that doesn't really add any value, if an embedded Ruby
>package is affected, Gitlab upstream fixes it in their weekly release
>anyway. And if not using the embedded code copies you'll end up with
>plenty of
>dependencies which can no longer be fulfilled from stable as upstream
>moves forward.

The intention is to keep the way open to have a real backport again should the situation change. I find that very important for compatibility and assuring upgrade paths.

>> I propose to add the volatile repository next to the backports
>> repository, and treat it as part of backports.
>
>I wouldn't tie this to backports at all, rather make it a separate
>section of the archive and have some ACL mechanism to allow the DDs
>maintaining a fastpaced package to grant access to it (similar to
>#817285).

I am open to this, as long as the goals to have full compatibility with backports stay the same.

-nik

Reply | Threaded
Open this post in threaded view
|

Re: Proposal: Repository for fast-paced package backports

Peter Pentchev
In reply to this post by Dominik George-7
On Tue, Dec 25, 2018 at 11:52:07PM +0100, Dominik George wrote:

> Hi,
>
> >having read the whole Gitlab discussion, I still don't get how/why the
> >new repository depends or relates to backports. Instead it could be
> >self-contained, except for stuff already available in stable. Couldn't
> >you roll the new repository entirely independent of any backports? Even
> >if you say there won't be any additional work for the backport policy
> >owners, letting a new repo depend on backports will implicitly have an
> >impact, which doesn't sound fully thought through yet.
>
> This is answered in the proposal. The reason is to not have volatile
> abused to ease backporting, and to allow packages to easily move back to
> backports again.
Just to make things a bit clearer for people who may not have followed
some of the discussions on d-bp-users lately: the point is to be able to
support fast-moving software with not-so-fast moving dependencies;
the dependencies may easily be backported without too large a burden
(their versions will not come too often, so they will be able to migrate
 to testing and thus fulfil the criteria for being in backports), while
the main piece of software moves too fast, including across major
versions and with incompatible changes, so that it is not suitable for
being included in a stable release (thus the part in the proposal about
blocking its migration to testing).

The maintainers of the stack will first package the dependencies, wait
for them to migrate to testing, then backport them, and then they will
upload the main piece of software first to unstable and then to the new
suite under discussion.

G'luck,
Peter

--
Peter Pentchev  roam@{ringlet.net,debian.org,FreeBSD.org} [hidden email]
PGP key:        http://people.FreeBSD.org/~roam/roam.key.asc
Key fingerprint 2EE7 A7A5 17FC 124C F115  C354 651E EFB0 2527 DF13

signature.asc (849 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Proposal: Repository for fast-paced package backports

Dominik George-7
>Just to make things a bit clearer for people who may not have followed
>some of the discussions on d-bp-users lately: the point is to be able
>to
>support fast-moving software with not-so-fast moving dependencies;
>the dependencies may easily be backported without too large a burden
>(their versions will not come too often, so they will be able to
>migrate
> to testing and thus fulfil the criteria for being in backports), while
>the main piece of software moves too fast, including across major
>versions and with incompatible changes, so that it is not suitable for
>being included in a stable release (thus the part in the proposal about
>blocking its migration to testing).
>
>The maintainers of the stack will first package the dependencies, wait
>for them to migrate to testing, then backport them, and then they will
>upload the main piece of software first to unstable and then to the new
>suite under discussion.

Exactly.

And the result shall still have the same quality as any package in -backports, technically, as far as it can. Thus the requirements for version, etc.

Volatile is not to become a place to dump packages to bypass -backports. On the contrary.

-nik

Reply | Threaded
Open this post in threaded view
|

Re: Proposal: Repository for fast-paced package backports

Martin-36
In reply to this post by Dominik George-7
Hi all,

I like the idea of having a volatile archive and I agree with
almost all what Dominik wrote about the motivation.

I would, however, completely separate it from backports. I.e.

 - separate NEW queue
 - different suffix
 - no need to keep a volatile package out of testing

Why?

 - volatile is a different beast from backports, this should be
   very clear to both package maintainers and our users
 - in volatile we can give less guarantees about future
   upgradability than backport provides
 - volatile must not put any burden on the backports team, which
   e.g. a common NEW queue would probably impose

Just my 6¢, Cheers

Reply | Threaded
Open this post in threaded view
|

Re: Proposal: Repository for fast-paced package backports

Dominik George-7
Hi,

>I would, however, completely separate it from backports. I.e.
>
> - separate NEW queue
> - different suffix
> - no need to keep a volatile package out of testing
>
>Why?
>
> - volatile is a different beast from backports, this should be
>   very clear to both package maintainers and our users

The idea is to have them separated, but fully interoperable.

I.e. the proposal ensures such things as:

- foo is not supportable for the buster release cycle. It goes to volatile.
- foo becomes supportable for buster+2.
- foo is backported (as in -backports) to buster+1

This will work properly, among other such scenari.

> - volatile must not put any burden on the backports team, which
>   e.g. a common NEW queue would probably impose

The whole point is that it is not new work or a new burden. This is one reason for the rules being almost the same and the clear decision path and movement between -backports and -volatile. A -volatile package is handled exactly the same, except it comes from unstable. The workload is the same as if the package had migrated to testing and was being uploaded to -backports. The defined preconditions ensure this is not abused for a ton of packages.

-nik

Reply | Threaded
Open this post in threaded view
|

Re: Proposal: Repository for fast-paced package backports

Dominik George-7
In reply to this post by Martin-36
> - no need to keep a volatile package out of testing

Oh, and yes. Having a package in testing means it will be supported for a stable lifecycle - a full contradiction to volatile!

-nik

Reply | Threaded
Open this post in threaded view
|

Re: Proposal: Repository for fast-paced package backports

Pirate Praveen-3
In reply to this post by Dominik George-7


On 2018, ഡിസംബർ 26 2:16:07 AM IST, Dominik George <[hidden email]> wrote:

>Heisann, alle sammen,
>
>as announced in the recent thread about maintaining, I hereby propose a
>repository that allows making “backports” of packages available to
>users
>of the stable distribution, if those packages cannot be maintained in
>testing and backported in the usual way. If you are interested in what
>lead up to that, please see bug #915050. I will give a short summary of
>it here.
>
>
>Reasons for having a special place for some packages
>====================================================
>
>(You may want to skip this part if you are familiar with the
>situation.)
>
>As all developers know (but passers-by may not), for software to enter
>the Debian archive, it is always uploaded to the unstable distribution,
>then migrates to testing (hopefully ;)), which is at some point
>snapshot
>and made the new stable release. From there on, maintainers have two
>obligations: Firstly, keep the package in stable good and secure, e.g.
>by uploading security fixes for it once they become available upstream,
>or even backport fixes themselves. Secondly, provide the package in
>unstable with updates and ensure its migration, to keep it ready for
>the
>next stable release.
>
>Now, for some software packages, this process is problematic, because
>upstream may have another idea about software lifecycles. Concerning
>the
>GitLab example, upstream provides security fixes for three months for
>their stable releases. Backporting fixes from newer versions is very
>hard or impossible because the massive amounts of changes to the
>software in every new versions. This is something that also affects
>other packages, like Mozilla Firefox, which has a firefox package in
>unstable, and a separate firefox-esr package, with the ESR version of
>Firefox. Only the latter migrates to testing.
>
>Users of Debian honour it for its stability, but as an agile software
>lifecycle is adapted by more and more very popular software packages,
>not being able to install these packages in the trusted, well-known
>fashion through the official apt repositories is becoming more and more
>of a drawback.
>
>It can easily be assumed that the normal release and maintenance cycle
>of Debian stable will not change, which is very good, so we should find
>a way to still provide such software as described above to users.
>
>
>Why backports is not enough
>===========================
>
>This also is well-known, but for completeness: Formal backports in
>stable-backports are required to be direct backports from testing, and
>are a stepping stone within the upgrade from stable to stable+1. Thus,
>a
>version of a package that is not in testing can never be in
>stable-backports.
>
>
>Name of the new repository
>==========================
>
>In the past, the name “volatile” was used for a similar repository, but
>with a different scope (limited to data packages for things like virus
>scanners). I will thus use the working title volatile throughout this
>proposal, although this may change.
>
>Other ideas: fastlane, unsupported
>
>(Please feel free to add other ideas.)
>
>
>Requirements for a package to go into stable-volatile
>=====================================================
>
>The new volatile proposal is not intended to ease life for package
>maintainers who want to bypass the migration and QA requirements of the
>regular stable lifecycle, so special need must be taken to ensure only
>packages that need it go into volatile. I want to summarise the
>requirements like so:
>
>- The package must be maintained in unstable, like every other package.
> - The package must not be in testing, and care must be taken for the
>   package not to migrate to testing.
> - Regular maintenance for the lifetime of stable must be impossible
>   or unnecessarily hard, and this requirement should be assessed in
>   a verifiable manner, e.g. referring to upstream’s lifecycle model.
> - There must be notable need for the package. Like for backports, user
>   requests might be an indicator.
> - Should the package be removed from unstable, it must also be removed
>   from volatile.
> - Should the package begin to migrate to testing again, it must
>   be moved to stable-backports.
>
>Before starting to maintain a volatile package, the maintainer shall
>seek consent (or doubt) on debian-devel.
>
>
>Building packages and package dependencies
>==========================================
>
>Packages for volatile are built the same way as formal backports, only
>that the source is taken from unstable rather than testing. In
>particular:
>
> - Changes shall be kept as small as possible.
> - The package is rebuilt against stable.
>- The package may depend on packages in stable, stable-backports or
>stable-volatile.
>
>Dependencies on other packages in volatile should be avoided if
>possible. Especially, dependencies of the package that also need
>backporting must not be added to volatile just because they are
>dependencies — every dependency that is needed to be backported to
>support the volatile package must be considered on its own and in all
>but unprobable edge cases be maintained as a formal backport.
>Obviously,
>the unprobable edge case occurs when the package depends on another
>package that also fully qualifies for volatile, as described above.
>
>
>Versions of packages in volatile
>================================
>
>I am not yet certain about this. As stressed before, volatile should be
>an extension of backports, so starting with the well-known backports
>suffix ~bpoN seems reasonable. I’d even say this is enough, as a
>package
>is never both in volatile and backports, and if maintenance changes to
>the regular lifecycle, it can easily be moved to backports.
>
>
>Responsibility and location of the repository
>=============================================
>
>I propose to add the volatile repository next to the backports
>repository, and treat it as part of backports. This should not impose
>new workload on the backports ftp-masters, so this needs people who
>volunteer to do the extra work. It should, however, be not too much of
>a
>workload anyway, as the number of packages qualifying for volatile is
>quite limited. (I do volunteer for the backports team, not only for my
>own proposal, but also in general.)
>
>This implies that new binary uploads to volatile have to undergo the
>same NEW queue as backports.
>
>
>volatile repositories for other distributions
>=============================================
>
>You guessed it: Same as for backports, but in green ;).
>
>
>Technical requirements
>======================
>
>Apart from the new section in the repository, care needs to be taken to
>ensure removal from volatile if a package moves to -backports again.
>The
>mechanisms used for decrufting experimental might apply.
>
>
>Implications for the situation at hand (gitlab)
>===============================================
>
>As there were quite a few concerns raised (some of which I share, and
>some I don’t): Of course, if a software intended for volatile has a ton
>of dependencies (intended to go into backports), all backports rules
>and
>powers of the ftp-masters apply. Repeating myself: volatile is not
>meant
>to ease the life of maintainers.
>
>
>
>I ask the community, the backports team and the release team for their
>opinions.
>
>Cheers, ha det bra,
>Nik

Hi Dominik,

Thanks for the detailed proposal. Some changes suggested already could be incorporated.

Just a note about it being an extension of backports. I will take example of gitlab here.

A large number of core packages in both JavaScript and ruby team is maintained by people who care about gitlab (a large part by me personally).

rails (its previous uploader  moved on to other stuff and we took a large portion of work required for rails 5 transition), rollup (28 reverse build deps), webpack (38 reverse dependencies), gulp (15 reverse build deps), grunt (25 reverse build deps), babel (32 reverse dependencies), npm (it was not updated for over 3 years), npm2deb (tool to create new node packages) to name a few. And the other libraries we keep up-to-date because we also need it for gitlab. Also the number of new contributors we bring to Debian because the work is large. If you are excluding these from properly maintained in normal release cycle and instead embed them inside gitlab, it will be a big loss to Debian as a whole as these packages are core components that many other packages depend on.

If JavaScript team and ruby team does not care about the work I do in the team, I can just bundle the whole dependencies inside gitlab and be done with that as proposed by Moritz. That will definitely make my life easier.

I was under the impression that as long as there are people willing to do the work and it meets dfsg, it can be part of Debian. It seems quite a lot of people prefer to keep these outside Debian as a solution.

Can't new volunteers to -backports team solve the extra burden problem? Dominik and I volunteered already.

If it has to be completely separate from -backports, it means some packages will need to be maintained twice, even when they meet the criteria for backports fully, just because a package in volatile declare a dependency on them.

Thanks
Praveen
--
Sent from my Android device with K-9 Mail. Please excuse my brevity.

Reply | Threaded
Open this post in threaded view
|

Re: Proposal: Repository for fast-paced package backports

Holger Levsen-2
In reply to this post by Dominik George-7
On Wed, Dec 26, 2018 at 12:07:42AM +0100, Dominik George wrote:
> I actually think volatile is a good name. After all, it's not so far from the previous volatile.

volatile is a very bad name for this because we've used it already for
something else.


--
cheers,
        Holger

-------------------------------------------------------------------------------
               holger@(debian|reproducible-builds|layer-acht).org
       PGP fingerprint: B8BF 5413 7B09 D35C F026 FE9D 091A B856 069A AA1C

signature.asc (849 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Proposal: Repository for fast-paced package backports

Dominik George-7
>> I actually think volatile is a good name. After all, it's not so far
>from the previous volatile.
>
>volatile is a very bad name for this because we've used it already for
>something else.

Well, I consider it more or less the same basic idea. The old and new ideas have more in common than not, with the only difference being that previously, volatile packages also had versions in stable.

-nik

Reply | Threaded
Open this post in threaded view
|

Re: Proposal: Repository for fast-paced package backports

Antonio Terceiro-3
In reply to this post by Pirate Praveen-3
On Wed, Dec 26, 2018 at 01:04:44PM +0530, Pirate Praveen wrote:
> If it has to be completely separate from -backports, it means some packages will need to be maintained twice, even when they meet the criteria for backports fully, just because a package in volatile declare a dependency on them.

There is nothing that stops you, or whoever wants to maintain this newn
repository from doing it in a way that 1) reuses what's already in
backports, even automatically and 2) adds the bits that are not deemed
appropriate for backports.

signature.asc (849 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Proposal: Repository for fast-paced package backports

gregor herrmann-3
In reply to this post by Holger Levsen-2
On Wed, 26 Dec 2018 13:07:07 +0000, Holger Levsen wrote:

(Can we keep this on one mailing list, please? /me restricts this to
-devel)

> On Wed, Dec 26, 2018 at 12:07:42AM +0100, Dominik George wrote:
> > I actually think volatile is a good name. After all, it's not so far from the previous volatile.
> volatile is a very bad name for this because we've used it already for
> something else.

Agreed.

And besides that, I think the more universal answer is
bikesheds/PPAs/you-name-it instead of yet-another-suite.

IIRC, there's even a draft specification … What I found now is
https://lists.debian.org/debian-devel/2013/05/msg00131.html [0]
https://lists.debian.org/debian-dak/2015/09/msg00023.html ff.

Since then bikesheds keep being mentioned every now and then for
handling fast-moving software; e.g.
https://lists.debian.org/debian-devel/2016/01/msg00696.html
https://lists.debian.org/debian-devel/2016/03/msg00363.html
https://lists.debian.org/debian-devel/2016/04/msg00059.html
https://lists.debian.org/debian-devel/2018/10/msg00072.html


Cheers,
gregor


[0]
"Aggressive Backport:
    This is the "dotdeb.org" scenario.  For whatever reason, people need
    whatever the latest version of php/mysql/apache/nginx/selectyourpoison
    is, compiled for (old)stable, in order to run on their otherwise
    stable servers."

--
 .''`.  https://info.comodo.priv.at -- Debian Developer https://www.debian.org
 : :' : OpenPGP fingerprint D1E1 316E 93A7 60A8 104D  85FA BB3A 6801 8649 AA06
 `. `'  Member VIBE!AT & SPI Inc. -- Supporter Free Software Foundation Europe
   `-   NP: Rolling Stones: Moonlight-mile-live

signature.asc (981 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Proposal: Repository for fast-paced package backports

Martin-36
On 2018-12-26 15:05, gregor herrmann wrote:
> (Can we keep this on one mailing list, please? /me restricts this to
> -devel)

Agreed.

> And besides that, I think the more universal answer is
> bikesheds/PPAs/you-name-it instead of yet-another-suite.

IMHO, this is not the same. Both "volatile/fastlane/whatever"
and "PPAs/bikeshed" have legitimate use cases.

1234