Best practices for Debian developers who are also upstreams?

classic Classic list List threaded Threaded
16 messages Options
Reply | Threaded
Open this post in threaded view
|

Best practices for Debian developers who are also upstreams?

Otto Kekäläinen-4
Hello!

I've ended up in being both the maintainer in Debian and an upstream
developer for a couple of packages and I have been fantasizing about
how to optimize my workflow so that I primarily fix all bugs and do QA
directly on the upstream development version (=upstream git master)
and then have only a very small overhead work then importing and
uploading new upstream releases in Debian.

Is somebody else already doing something similar like this?
Any tips, blogs, examples on the topic?


I find it annoying to carry Debian patches for bugs that could be
fixed globally at upstream, and it is also annoying when something
Debian QA catches is broken at upstream and I find it out only at the
time I am preparing an upload to Debian.

My goals would be:
- have debian/ in upstream repository, and a CI that does the bulk of
Debian QA on every upstream commit and immediately gives feedback to
upstream devs that "break" something from Debian QA point of view
- have the delta of Debian debian/ an upstream debian/ as small as possible
- fix all bugs detected in Debian directly at upstream when possible
- when not possible, fix them "locally" first in Debian and eventually
have it upstreamed
- have it easy to compare Debian and upstream debian/ contents
- bonus: import upstream releases as git operations without having to
export and import tar.gz packages

I have in rdiff-backup pretty much this setup already in place but I
have some challenges on how to handle the debian/changelog file and a
couple of other details. I would be very keen to learn from others
before inventing too much of new.


- Otto

Reply | Threaded
Open this post in threaded view
|

Re: Best practices for Debian developers who are also upstreams?

David Bremner
Otto Kekäläinen <[hidden email]> writes:

> Hello!
>
> I've ended up in being both the maintainer in Debian and an upstream
> developer for a couple of packages and I have been fantasizing about
> how to optimize my workflow so that I primarily fix all bugs and do QA
> directly on the upstream development version (=upstream git master)
> and then have only a very small overhead work then importing and
> uploading new upstream releases in Debian.
>
> Is somebody else already doing something similar like this?
> Any tips, blogs, examples on the topic?

I basically maintain notmuch that way.

[snip]

> My goals would be:
> - have debian/ in upstream repository, and a CI that does the bulk of
> Debian QA on every upstream commit and immediately gives feedback to
> upstream devs that "break" something from Debian QA point of view

I do have debian in the upstream repo. salsa CI is currently not working
due to an obscure bug involving emacs and docker. I do provide an
upstream build target to make snapshot packages to help people test. And
we use travis, but only to test the upstream builds.

> - have the delta of Debian debian/ an upstream debian/ as small as possible

This is generally zero. I guess it comes down to definition. There is no
seperate Debian debian/ dir as far as I'm concerned.

> - fix all bugs detected in Debian directly at upstream when possible

Sure. I typically make point/micro releases for non-debian specific
problems. Although the package is non-native for flexibility, I mostly
treat it as native.

> - when not possible, fix them "locally" first in Debian and eventually
> have it upstreamed

I'm not sure I follow you precisely, but in my case I make a branch in
the upstream repo for e.g. stable updates.

> - bonus: import upstream releases as git operations without having to
> export and import tar.gz packages

I have only one git repo, so there's no importing per se.

Reply | Threaded
Open this post in threaded view
|

Re: Best practices for Debian developers who are also upstreams?

Otto Kekäläinen-2
Hi!


But apparently "upstream" releases are decoupled from also bumping the Debian version?


This means debian/changelog shows "wrong" version at least for one commit and that would not work in the CI pipelines I use as build products would end up with inaccurate version strings. This is something I seek to find an elegant solution for..

- Otto


Reply | Threaded
Open this post in threaded view
|

Re: Best practices for Debian developers who are also upstreams?

David Bremner
Otto Kekäläinen <[hidden email]> writes:

> Hi!
>
> Thanks for the example. I see you even maintain the changelog in upstream:
> https://git.notmuchmail.org/git?p=notmuch;a=blob;f=debian/changelog;h=4f7457cd7f588e6d3a821bf1530cc1f8742c8d6f;hb=refs/heads/master

Yes.

>
> But apparently "upstream" releases are decoupled from also bumping the
> Debian version?
>
> https://git.notmuchmail.org/git?p=notmuch;a=commit;h=3efa2ad72c8ffd8183fab2cd6592f35e72fbb7d7
>
> This means debian/changelog shows "wrong" version at least for one commit
> and that would not work in the CI pipelines I use as build products would
> end up with inaccurate version strings. This is something I seek to find an
> elegant solution for..

We're obviously less focussed on CI than you are. The fact that it takes
a couple commits to make a release has never been problematic for us. I
guess if it was important, you could do all the version updates in one
commit.

d

Reply | Threaded
Open this post in threaded view
|

Re: Best practices for Debian developers who are also upstreams?

Simon Richter
In reply to this post by Otto Kekäläinen-4
Hi,

On 08.02.20 21:07, Otto Kekäläinen wrote:

> I've ended up in being both the maintainer in Debian and an upstream
> developer for a couple of packages and I have been fantasizing about
> how to optimize my workflow so that I primarily fix all bugs and do QA
> directly on the upstream development version (=upstream git master)
> and then have only a very small overhead work then importing and
> uploading new upstream releases in Debian.

In my experience, it was often a good idea to separate my upstream and
Debian hats.

If your packages are part of a stable release, this effectively forces
you to split off a stable branch from whatever is in the archive at
freeze time, so it might be a good idea to package a LTS branch instead
of the latest release.

At the same time, Debian packages have a well-defined installation
procedure, so it is often possible to build a quick fix for a bug that
will give users a working package, but where the patch is not suitable
for upstream inclusion, like changing an init script or unit file.

> - have debian/ in upstream repository, and a CI that does the bulk of
> Debian QA on every upstream commit and immediately gives feedback to
> upstream devs that "break" something from Debian QA point of view

Would that mean that upstream developers are expected to update the
packaging when they make a breaking change, or do you have a workflow
that permits leaving the packaging in a broken state until someone gets
around to fixing the packaging? Does that scale for multiple distributions?

> - have the delta of Debian debian/ an upstream debian/ as small as possible
> - have it easy to compare Debian and upstream debian/ contents

If you keep a debian/ directory in upstream sources, it should not be
allowed to diverge at all -- anything else just leads to confusion after
checkout, and people building broken packages. Since there are multiple
distributions using the Debian package format, this also means you can
only support one of them in the upstream repo, and all others need to
find a patch workflow to implement their local changes.

> - bonus: import upstream releases as git operations without having to
> export and import tar.gz packages

Not much of a bonus though -- the Debian archive still requires a proper
release archive anyway for offline use.

> I have in rdiff-backup pretty much this setup already in place but I
> have some challenges on how to handle the debian/changelog file and a
> couple of other details. I would be very keen to learn from others
> before inventing too much of new.

Maintaining debian/changelog inside a version control system is a bit of
a category error, because it is a form of version control system log,
although one that lists far fewer individual commits, and where commits
addressing multiple issues are encouraged. Stacking two version control
systems on top of each other behaves roughly as well as tunneling TCP/IP
inside TCP/IP: works most of the time, but hiccups amplify each other.

Last but not least: package uploads should be spun from releases that
"upstream" feels have the required quality level. We have very few
"rolling release" packages in the archive, because users only complain
about them being outdated and buggy at the same time, and updating to a
different snapshot would freeze them at a different point in time with
different bugs. Ideally, a release has had a bit more extensive testing
upstream than just "passes CI".

   Simon


signature.asc (499 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Best practices for Debian developers who are also upstreams?

Alf Gaida
In reply to this post by Otto Kekäläinen-4
Hi Otto,

i'm doing this for years now (LXQt)

To sum it up:
* there are no special rules, but i suggest to keep your hats strictly
separated
* if there are some doubts upstream - just think how you would feel with
a release with your downstream hat on
* if you release something upstream and have to patch it for debian
(nuff said :D)
* if still in doubt - just make an Arch Linux package - and think about
the differences

At all, we (LXQt) have learned a lot about licenses, abi and api things
and simple no-go's for an reliable upstream. Most of my packages are
without any patches, some packages have some ness. overrides for
architectures, nothing i could do against. All packages are boring
packagingwise - and that's a very good sign. I use the classical way of
release tarballs, the workflow is: download, import, pristine-tar, check
control, copyright and be done with. In my case up- and down-stream are
fully discoupled. If something special happend - new abi, need for
package splits, things we couldn't handle upstream - i know it before
and be prepared, even packagingwise. So the maintainance is just a
breeze most of the time.

If you ware both upstream and downstream hats you should ask yourself
some questions:
* have i really done all the things to make downstream comftable
* if not, what could i do to make downstream more comftable
* what can i do to make downstream life better in future releases

Guess what, other downstreams will love you for.

Cheers Alf

Reply | Threaded
Open this post in threaded view
|

Re: Best practices for Debian developers who are also upstreams?

Sean Whitton
In reply to this post by Otto Kekäläinen-4
Hello Otto,

On Sat 08 Feb 2020 at 10:07PM +02, Otto Kekäläinen wrote:

> I've ended up in being both the maintainer in Debian and an upstream
> developer for a couple of packages and I have been fantasizing about
> how to optimize my workflow so that I primarily fix all bugs and do QA
> directly on the upstream development version (=upstream git master)
> and then have only a very small overhead work then importing and
> uploading new upstream releases in Debian.
>
> Is somebody else already doing something similar like this?
> Any tips, blogs, examples on the topic?
>
>
> I find it annoying to carry Debian patches for bugs that could be
> fixed globally at upstream, and it is also annoying when something
> Debian QA catches is broken at upstream and I find it out only at the
> time I am preparing an upload to Debian.
>
> My goals would be:
> - have debian/ in upstream repository, and a CI that does the bulk of
> Debian QA on every upstream commit and immediately gives feedback to
> upstream devs that "break" something from Debian QA point of view
> - have the delta of Debian debian/ an upstream debian/ as small as possible
> - fix all bugs detected in Debian directly at upstream when possible
> - when not possible, fix them "locally" first in Debian and eventually
> have it upstreamed
> - have it easy to compare Debian and upstream debian/ contents
> - bonus: import upstream releases as git operations without having to
> export and import tar.gz packages
>
> I have in rdiff-backup pretty much this setup already in place but I
> have some challenges on how to handle the debian/changelog file and a
> couple of other details. I would be very keen to learn from others
> before inventing too much of new.
I do this with the package git-remote-gcrypt (and some others).  I don't
try very hard to separate my upstream and Debian hats.  It hasn't been a
problem so far.

I use a single branch for upstream releases and the -1 Debian uploads.
If there are commits to the upstream source, and -2, -3 etc. uploads to
Debian before there is an upstream release, I just represent the
divergence with two git branches, which can be merged back together for
the next upstream release and -1 upload.

In order to avoid the problem of discovering Debian problems only after
tagging a new upstream release, this sbuild rune can build the package
without an .orig.tar:

    alias sbuild-prerelease="sbuild --dpkg-source-opts='-Zgzip -z1 --format=1.0 -sn'"

You can add --run-autopkgtest and --run-piuparts to that.

So, you could have your CI running that on all commits.

(credits to Ian Jackson for the rune)

To do an upstream release and -1 upload, I do this:

- `git merge` any -2, -3 etc. uploads to Debian of the last upstream
  release; this updates debian/
  + relies on `dpkg-mergechanglogs` to be enabled
- finalise Debian -1 changelog and upstream changelog, and git commit
  both
  + for git-remote-gcrypt they're actually the same file but that's for
    historical reasons
- git tag -s 1.2
- git deborig
- sbuild and upload
- start a 1.3-1 debian changelog entry and a 1.3 upstream changelog
  entry, and commit those

--
Sean Whitton

signature.asc (847 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Best practices for Debian developers who are also upstreams?

Wouter Verhelst
In reply to this post by Otto Kekäläinen-4
On Sat, Feb 08, 2020 at 10:07:48PM +0200, Otto Kekäläinen wrote:
> Hello!
>
> I've ended up in being both the maintainer in Debian and an upstream
> developer for a couple of packages and I have been fantasizing about
> how to optimize my workflow so that I primarily fix all bugs and do QA
> directly on the upstream development version (=upstream git master)
> and then have only a very small overhead work then importing and
> uploading new upstream releases in Debian.

So, I have four packages that are in various ways similar to this:

- nbd, which I started maintaining in Debian before becoming upstream
  for it;
- SReview, which I started maintaining upstream before uploading it to
  Debian;
- fdpowermon, which I released first by uploading it to Debian;
- ola, which I maintain for Debian and do not really maintain upstream,
  but which upstream did give me commit rights to their repository for.

I have separate upstream and debian branches for nbd; however, if
someone reports a bug, I will fix it upstream first, and then (possibly,
if applicable and it's too soon to release an upstream version)
cherry-pick the patch to the debian branch. There is no debian/
directory in the upstream branch. Updating to a new upstream release for
Debian involves a simple "git merge" of the upstream branch, which will
always work if I never violated that policy. Needless to say, nbd is not
a native package.

I do not maintain separate branches for SReview or fdpowermon. The
difference between the two, however, is that SReview is uploaded to CPAN
as well as Debian. The CPAN upload does not contain the debian/
directory, and I do use the tarball of the CPAN upload as the
orig.tar.gz for the Debian upload. However, they're both built from the
same git checkout. Fdpowermon, on the other hand, I do not upload it to
CPAN (it's too simple for that), and is instead uploaded as a native
package to Debian.

For ola, upstream at one point committed a squash commit of all the
Debian patches I had committed to my debian branch at that point in
time. The next time I tried to merge their newest release by way of the
git tag, things went a bit haywire. I do not recommend this approach.

--
To the thief who stole my anti-depressants: I hope you're happy

  -- seen somewhere on the Internet on a photo of a billboard

Reply | Threaded
Open this post in threaded view
|

Re: Best practices for Debian developers who are also upstreams?

Florian Weimer
In reply to this post by Otto Kekäläinen-4
* Otto Kekäläinen:

> Is somebody else already doing something similar like this?

We are doing this with glibc in Fedora, which is not Debian, but kind
of similar.  We try to push all backportable fixes to the upstream
release branches (and master) and synthesize new pseudo-release
tarballs from those branches.  This means that (the equivalent of)
.orig.tar.xz reflects upstream contents, but necessarily an actual
release tarball.

This means that the set of downstream patches is reasonably small, and
grows only moderately due to ongoing bug-fixing.  In some cases, we
want to backport things into Fedora which do not seem appropriate for
upstream, and those end up as patch files.

The distribution packaging bits do not live in the upstream
repository, but we have special (glibc-specific) scripts which allow
us to merge in upstream changes (using git cherrypick and git rebase),
so this mostly a matter of training people to use these scripts.

I think this approach is only possible because we have stable release
branches upstream where we can backport the things we are interested
in (and other contributors are reasonably conservative), and we are
willing to respin the tarballs.  It simplifies matters for the
uploaders, but we cannot easily bisect individual changes/upstream
commits because they do not exist downstream, and the upstream
repository lacks the distribution customizations.

Reply | Threaded
Open this post in threaded view
|

Re: Best practices for Debian developers who are also upstreams?

Otto Kekäläinen-4
Hello!

Thanks for all the examples (LXQt [where I looked at the sources of
lxqt-panel], git-remote-gcrypt, nbd, review, glibc).

Most interesting was perhaps Seans git-remote-gcrypt. You even have a
rpm spec file included which helps illustrate this is a true upstream
and not a fully native Debian package.

There does not seem to be any automation around release numbering —
currently the repo tags, debian/changelog and redhat spec file all
bumped manually and independently.
https://github.com/spwhitton/git-remote-gcrypt/commit/069f6ea33149fbaa8bd83423b7a7b591fcfed43b

In the other cases I did not see a real feedback loop where upstream
developers would be exposed to Debian policy requirements or Debian
QA.

Seems many misunderstood my list of goals. My emphasis is not on the
git branch layout or patch handling. So let me rephrase my goals from
a quality assurance perspective:

- Build .deb directly at upstream for every upstream commit: allow
upstream developers to notice if they break Debian packaging to the
level that it stops building. Upstream PRs (or MRs) should not be
accepted if they make changes without considering if they break .deb
(or .rpm if that is also included in the upstream repository).
Effectively this requires that upstream git master branch has a
debian/ directory.

- Since there will be a debian/ directory, upstream might as well run
full Debian QA on every commit. This will help detect if upstream
developes make grave mistakes, like try to run curl/git fetches
duringthe build stage, introduce library symbols in a wrong way or
embed in the build something that stops from being a reproducible
build anymore. Lintian even complains about spelling mistakes. Such QA
items are not Debian specific – all code everywhere should be without
spelling mistakes, all programs should stay reproducible if they now
are so etc.

Now some ask, if it reasonable to require all upstream developers to
follow Debian QA? No, I don't think so. If the CI runs automatically
and the QA is automatic, then developers will be served with
in-context errors and warnings at the time they are changing some
code. For everybody to know the Debian Policy by heart and so on is
naturally unreasonable, but reading a Lintian warning about a spelling
error or broken multi-arch is not hard or unreasonable, and when those
things are addressed the upstream code base benefits and all
distributions benefit.

The contrary is in my opinion unreasonable: to expect the Debian
maintainer to fix tens of upstream issues at Debian import time that
can be anything ranging like fixing a failing test suite or tracking
down why the build is no longer reproducible. Yes, the maintainer
could file N upstream bugs, but upstreams are less likely to fix them
(as they might have switched context and don't want to revisit
something they coded several months back) and even if upstreams fix
them, the maintainer would need to wait some weeks, then manually
cherry-pick and patch the Debian version, and then do extra work
following up on all those patches and dropping or refreshing them on
next upstream release.

Having a debian/ in upstream, having deb building as part of the
upstream CI, and keeping the delta of Debian debian/ and upstream
debian/ as small as possible makes life much easier for the Debian
maintainer, and in general all distros and the whole upstream code
base benefits of the extra QA.

- Otto

Reply | Threaded
Open this post in threaded view
|

Re: Best practices for Debian developers who are also upstreams?

Otto Kekäläinen-4
Hello!

Currently the project where I am closest to this goal is in
rdiff-backup. The upstream debian/ is almost 1:1 to the downstream
debian/. I've made a dh_gencontrol override to ensure the upstream CI
always builds binaries with the correct version number inherited from
a git tag in upstream:
https://github.com/rdiff-backup/rdiff-backup/blob/master/debian/rules

What remains is to decide what to do to the debian/changelog. The
version there still leaks the source package version number that ends
up in *.changes and *.dsc..

Otherwise it is rolling pretty well. The upstream .travis-ci.yml
builds the .eb just like Debian will eventually do (apart from some
version and changelog differences) and all bugs I encounter in Debian
or elsewhere can be fixed directly on upstream for everybody's
benefit. My work on the quality of the software is directly beneficial
at upstream and "globally" and not just for Debian. I still have a
separate Debian repo on salsa.debian.org but in future I think I'll
have a debian/master branch directly in upstream repository and
configure to git-buildpackage the upstream branch as 'master'.

One of the best inspirations for this was
https://vincent.bernat.ch/en/blog/2019-pragmatic-debian-packaging and
examples at https://github.com/vincentbernat/pragmatic-debian-packages/tree/master
but I am still looking to optimize the process as a whole.

I don't feel there is any problem with "two hats". Pretty much
everything the Debian policy states about software quality applies
universally to any open source project. The things that are very
Debian specific will only be expressed in code under debian/ and not
affect other distros. If this works out well I'll expand to become
upstream in all projects I maintain in Debian if the upstreams trust
me enough to give write access. The closer a maintainer is to the
upstream, the better everything will be I reason.

- Otto

Reply | Threaded
Open this post in threaded view
|

Re: Best practices for Debian developers who are also upstreams?

Richard Laager
In reply to this post by Simon Richter
On 2/8/20 7:57 PM, Simon Richter wrote:
> In my experience, it was often a good idea to separate my upstream and
> Debian hats.

+1

My current situation is that I'm a Debian packager who has then over
time become more involved in upstream and eventually ended up with a
commit bit.

On a different package, I'm in the process of taking over packaging for
something where I am already an upstream developer (or perhaps should
say "was", as I haven't been active upstream in some time, though that
may change here).

That sounds like it may be different than your situation, but I hope I
can still offer some useful comments.

> At the same time, Debian packages have a well-defined installation
> procedure, so it is often possible to build a quick fix for a bug that
> will give users a working package, but where the patch is not suitable
> for upstream inclusion, like changing an init script or unit file.

+1. In such a case, you can clean that up into a more general solution
for upstream and later drop the Debian patch. I've had that exact thing
happen.

>> - have debian/ in upstream repository, and a CI that does the bulk of
>> Debian QA on every upstream commit and immediately gives feedback to
>> upstream devs that "break" something from Debian QA point of view

I'm in the process of packaging something where upstream (which is not
me) puts their debian directory (for CI purposes) as packaging/debian.
That allows them to do real Debian packaging for CI, and update that as
necessary, but keeps it out of the way of the real "debian" directory.
You might consider that option. Upstream's CI presumably has to
move/copy/link ./packaging/debian to ./debian before kicking off the
Debian build, but that's just one extra step.

>> - bonus: import upstream releases as git operations

Do this either way! :)

In the packages I maintain, my debian/README.source says:
The `master` branch is upstream's `master` branch.
...
The `upstream/latest` branch contains the unpacked contents of the orig
tarball.  It is branched off the `master` branch, so the delta from
`master` (at the branch point) to `upstream/latest` contains generated
files that exist in the tarball but not in the upstream repository.

That said, I prefer to keep separate git checkouts for the package and
upstream to avoid the potential for accidents like pushing upstream pull
request branches to Debian Salsa or, far more likely, accidentally
pushing the current master to Debian Salsa.

> Maintaining debian/changelog inside a version control system is a bit of
> a category error, because it is a form of version control system log,

I'm a big fan of using git-buildpackage and `gbp dch`. My current
workflow (somewhat "copied" from e.g. lintian) is that debian/changelog
contains a stub like this until it is time to tag a Debian package release:

ntpsec (1.1.8+dfsg1-4+git) UNRELEASED; urgency=medium

  * UNRELEASED

 -- Richard Laager <[hidden email]>  Sat, 11 Jan 2020 22:49:28 -0600

Note that it is critical that the version in Salsa NOT be the same as
the version in the archive if you are using Salsa CI to test changes.
That's why the +git is added. If they are the same, when the CI code
tries to install the package, you'll have problems because of the
conflicting version numbers.

--
Richard


signature.asc (849 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Best practices for Debian developers who are also upstreams?

Sean Whitton
In reply to this post by Otto Kekäläinen-4
Hello,

On Sun 09 Feb 2020 at 10:57PM +02, Otto Kekäläinen wrote:

> Most interesting was perhaps Seans git-remote-gcrypt. You even have a
> rpm spec file included which helps illustrate this is a true upstream
> and not a fully native Debian package.
>
> There does not seem to be any automation around release numbering —
> currently the repo tags, debian/changelog and redhat spec file all
> bumped manually and independently.
> https://github.com/spwhitton/git-remote-gcrypt/commit/069f6ea33149fbaa8bd83423b7a7b591fcfed43b

Indeed.  If the package was more active I would implement some script
which would bump all the versions, which could be run right after
releasing.

> - Build .deb directly at upstream for every upstream commit: allow
> upstream developers to notice if they break Debian packaging to the
> level that it stops building. Upstream PRs (or MRs) should not be
> accepted if they make changes without considering if they break .deb
> (or .rpm if that is also included in the upstream repository).
> Effectively this requires that upstream git master branch has a
> debian/ directory.
>
> - Since there will be a debian/ directory, upstream might as well run
> full Debian QA on every commit. This will help detect if upstream
> developes make grave mistakes, like try to run curl/git fetches
> duringthe build stage, introduce library symbols in a wrong way or
> embed in the build something that stops from being a reproducible
> build anymore. Lintian even complains about spelling mistakes. Such QA
> items are not Debian specific – all code everywhere should be without
> spelling mistakes, all programs should stay reproducible if they now
> are so etc.
Right.  My sbuild-prerelease alias could be used in your CI to do this.

--
Sean Whitton

signature.asc (847 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Best practices for Debian developers who are also upstreams?

Antoine Beaupré-2
In reply to this post by Otto Kekäläinen-4
On 2020-02-08 22:07:48, Otto Kekäläinen wrote:

> Hello!
>
> I've ended up in being both the maintainer in Debian and an upstream
> developer for a couple of packages and I have been fantasizing about
> how to optimize my workflow so that I primarily fix all bugs and do QA
> directly on the upstream development version (=upstream git master)
> and then have only a very small overhead work then importing and
> uploading new upstream releases in Debian.
>
> Is somebody else already doing something similar like this?
> Any tips, blogs, examples on the topic?

I maintain a few packages in Debian for which I am also the (mostly
solo) upstream:

 * feed2exec
 * gameclock
 * monkeysign
 * undertime

Those packages are all *native* packages, in that they *don't* have a -N
suffix in the version number. This is generally frowned upon, but in my
situation, it's been helpful because I don't need to worry about
"upstream" vs "debian": everything is "in Debian".

This implies that I have only one debian/ directory: the upstream. When
I need to make a release, I push it to unstable. If I need to make a
hotfix release for security or stable, I make a branch. I use semantic
versionning and basically maintain a "major" version per Debian release
(as necessary, which is "rarely").

> I find it annoying to carry Debian patches for bugs that could be
> fixed globally at upstream, and it is also annoying when something
> Debian QA catches is broken at upstream and I find it out only at the
> time I am preparing an upload to Debian.

I do both at once, all the time, as part of CI and the release
process. I do the Debian checklist before I even push tags upstream, but
sometimes things come up in Debian after I pushed the tag. Then I just
make another patch release, no big deal.

> My goals would be:
> - have debian/ in upstream repository, and a CI that does the bulk of
> Debian QA on every upstream commit and immediately gives feedback to
> upstream devs that "break" something from Debian QA point of view

The tricky part here is generating a new version without mangling
debian/changelog all the time. I haven't found a great story for that
that works with git, but maybe you can generate syntetic commits to fake
a new Debian package version in CI.

> - have the delta of Debian debian/ an upstream debian/ as small as
> possible

For me, there's no diff. It's the same branch, same code. I do, however,
generate debian/changelog only on release so I don't have to repeat
myself between commit logs (in git) and release notes (in
debian/changelog).

> - fix all bugs detected in Debian directly at upstream when possible

Not sure what that means, but when I fix a bug in Debian, it's fixed in
"upstream" first, ie. I push it to git, and I tag it as "pending" in the
BTS to make it clear it will be fixed in the next upload.

> - when not possible, fix them "locally" first in Debian and eventually
> have it upstreamed

I never have to do that. To fix it "first in Debian", I just make a new
upstream release. If it's on a previous "branch" (e.g. stable), I make a
release on that branch.

For example right now undertime is 1.7.0 in buster and 1.8.0 in
bullseye/sid. Next RC bugfix release is 1.8.1 in sid and 1.7.1 in
buster, and I would have a 1.7.x branch to follow buster (which I
haven't needed yet).

> - have it easy to compare Debian and upstream debian/ contents

Trivial. It's my different maintenance branches.

> - bonus: import upstream releases as git operations without having to
> export and import tar.gz packages

I export tarballs only as part of the release process and otherwise
don't bother with those artefacts at all.

> I have in rdiff-backup pretty much this setup already in place but I
> have some challenges on how to handle the debian/changelog file and a
> couple of other details. I would be very keen to learn from others
> before inventing too much of new.

The changelog is the hard part. I update it only on release which
simplifies things tremendously, but will make CI harder, as I mentioned
before.

Also, this is not for everyone: it makes it hard for Debian folks to
send updates to the git repository (because it might not be on
debian.org infra) and inversely it makes it hard for "upstream" to make
a release if they're not Debian developers. Fortunately, I am both and
those are mostly solo projects so that hasn't been an issue so far.

When it will be, the package will just go non-native and follow more
usual development practices. This happened to etckeeper when joeyh left
us, for example.

I hope that helps!

A.

--
Si les triangles avaient un Dieu, ils lui donneraient trois côtés.
                        - Montesquieu, Lettres persanes

Reply | Threaded
Open this post in threaded view
|

Re: Best practices for Debian developers who are also upstreams?

Simon McVittie-7
On Tue, 18 Feb 2020 at 15:47:17 -0500, Antoine Beaupré wrote:
> The tricky part here is generating a new version without mangling
> debian/changelog all the time. I haven't found a great story for that
> that works with git, but maybe you can generate syntetic commits to fake
> a new Debian package version in CI.

I think the way to do this is to have an uncommitted change to d/changelog
when you do your build, and choose your CI version numbers carefully so
that they interleave correctly with "real" version numbers.
https://gitlab.collabora.com/smcv/deb-build-snapshot/ (currently on my
employer's infrastructure, but I'll probably move it to Salsa at some
point) is designed for this workflow.

The version-number-generating part, deb-git-version-gen, implements the
careful choice of version numbers and has been split out into a separate
script which I might propose for inclusion in devscripts at some point -
it should be feasible to run as a git-buildpackage hook or similar,
and can either run dch itself, or output a simple version number or a
blob of JSON describing the package's versioning (the latter includes
some suggested arguments for dch).

> Also, this is not for everyone: it makes it hard for Debian folks to
> send updates to the git repository (because it might not be on
> debian.org infra)

This seems particularly weird for native packages, which are supposedly
"written specifically to be a Debian package" (Policy §5.6.12). When
maintaining upstream packages that include their own Debian-compatible
packaging, I sometimes make them non-native with versions like
1.2.3-0upstream1, which makes it trivial to repackage them as non-native
elsewhere.

> and inversely it makes it hard for "upstream" to make
> a release if they're not Debian developers.

This was a practical problem when Joey Hess left Debian: ikiwiki was a
native package, with no release procedure other than uploading it to
Debian, and I became a single point of failure for security releases
because I was suddenly the only person with both ikiwiki commit access
and Debian upload rights.

(I later disentangled it so that upstream releases are just git tags
pointing to a native package, and the Debian packaging is non-native.)

    smcv

Reply | Threaded
Open this post in threaded view
|

Re: Best practices for Debian developers who are also upstreams?

Antoine Beaupré-2
On 2020-02-18 23:14:14, Simon McVittie wrote:

> On Tue, 18 Feb 2020 at 15:47:17 -0500, Antoine Beaupré wrote:
>> The tricky part here is generating a new version without mangling
>> debian/changelog all the time. I haven't found a great story for that
>> that works with git, but maybe you can generate syntetic commits to fake
>> a new Debian package version in CI.
>
> I think the way to do this is to have an uncommitted change to d/changelog
> when you do your build, and choose your CI version numbers carefully so
> that they interleave correctly with "real" version numbers.
> https://gitlab.collabora.com/smcv/deb-build-snapshot/ (currently on my
> employer's infrastructure, but I'll probably move it to Salsa at some
> point) is designed for this workflow.

My trouble with doing that by hand is when I change the git-repo,
git-buildpackage and friends complain about uncommitted changes.. :) I'm
sure there are simple ways around that, but that was the blocker I
remembered back then.

> The version-number-generating part, deb-git-version-gen, implements the
> careful choice of version numbers and has been split out into a separate
> script which I might propose for inclusion in devscripts at some point -
> it should be feasible to run as a git-buildpackage hook or similar,
> and can either run dch itself, or output a simple version number or a
> blob of JSON describing the package's versioning (the latter includes
> some suggested arguments for dch).

That would be really nice.

>> Also, this is not for everyone: it makes it hard for Debian folks to
>> send updates to the git repository (because it might not be on
>> debian.org infra)
>
> This seems particularly weird for native packages, which are supposedly
> "written specifically to be a Debian package" (Policy §5.6.12).

I have always found the wording and context around native packages to be
really weird. Aren't all Debian *packages* written specifically to be
debian packages? :)

(I know the actual wording is "a piece of software was written
specifically to be a Debian package"... So yes, I'm being facetious
here.)

I think that restriction should be lifted, personnally. Using native
packages has made my job as an upstream and a Debian developer much
easier even if the tools weren't specifically written for inclusion in
Debian. There are many cases of such packages that don't respect policy
and I wonder if policy should adapt rather than changing that practice.

I also find it quite ironic to think about this provision in the context
where a lot of software specifically written for Debian is actually
*not* packaged in Debian itself. Instead it's being deployed through
other means on Debian.org infrastructure. So the prime use case of
native packages is not being really used in the first place. ;)

[...]

>> and inversely it makes it hard for "upstream" to make
>> a release if they're not Debian developers.
>
> This was a practical problem when Joey Hess left Debian: ikiwiki was a
> native package, with no release procedure other than uploading it to
> Debian, and I became a single point of failure for security releases
> because I was suddenly the only person with both ikiwiki commit access
> and Debian upload rights.
>
> (I later disentangled it so that upstream releases are just git tags
> pointing to a native package, and the Debian packaging is non-native.)

I have not found the switch between native and non-native to be that
problematic, personnally. It's something that needs to happen on
upstream or debian team changes, but it can be done reasonably
easily. Or at least it's easy enough that it's not a disincentive to use
native packages when you can.

A.

--
All governments are run by liars and nothing they say should be
believed.
                       - I. F. Stone