Thoughts on Debian quality, including automated testing

classic Classic list List threaded Threaded
86 messages Options
12345
Reply | Threaded
Open this post in threaded view
|

Thoughts on Debian quality, including automated testing

Lars Wirzenius-2
Subject: Thoughts on Debian quality, including automated testing

[ I'm subscribed to -devel, no Cc required. I apologize for the
  length, but it's only a bit over 3000 words. I hope the
  section titles help, if you want to skip parts. ]

For some time now I have been thinking about ways to make Debian
better from a technical point of view. Most of my actual efforts
have gone into writing piuparts[1], running it against packages
in the archive, and reporting any problems. I have also spent some
time to think about related issues.

    [1] http://packages.debian.org/unstable/devel/piuparts

This mail is primarily prompted by Ian Jackson's proposal[2] to
specify a framework for automated testing. I meant to write and send
it weeks ago, but for various reasons, I kept postponing finishing
it. Sorry about that. Part of the reason is that as I kept thinking
and writing about this, I kept expanding the scope. As a result, this
mail is quite long. Sorry about that, too. Further, I keep mentioning
piuparts in this mail. Sorry if it seems like I'm advertising it.

I'll start by saying that I fully support writing automated tests
for Debian packages. Automated tests can do very good things to
development of single programs. They can also do so to Debian
packages, and Debian as a whole.

    [2] http://lists.debian.org/debian-project/2005/11/msg00073.html
        and other threads

Before I get down to details, I'd like to be a bit philosophical
and preachy. You may want to skip a few paragraphs.

The quality of Debian is not bad at all. Debian works quite well for
a large number of people, and we get fairly few bug reports from
them relative to the number of programs we have packaged. That's
pretty much the only objective criterion we can currently use to
determine real quality.

Quality is sometimes hard to define. I claim that "package has
few bug reports in proportion to its user base" is one important
indicator of high quality.

Still, we could do much better. Our two best known quality assurance
tools, lintian and linda, are obviously not used by a lot of package
maintainers[4], given the number of packages that have problems.
Consider, for example, lintian's test for zero-byte files in the doc
directory[5]. There are a hundred packages that fail that test. Yet
the problem is really utterly simple to fix.

    [4] http://lintian.debian.org/reports/tags.html 
        [5]
        http://lintian.debian.org/reports/Tzero-byte-file-in-doc-directory.html

These zero-byte files are not a "real" problem: they use up an
inode, and make people spend a few seconds extra when looking
for information about the package, but they don't actually break
anything.

Yet the packages in question would be of higher quality if they
were non-empty or didn't exist at all. They may also indicate other
sloppiness, which may or may not be caught with automatic tools.
Sloppiness tends to result in real problems sooner or later.

I propose "respected automated tools find few problems" as the
second indicator of quality.

To improve the quality of Debian, we need to do several things:

    A) Prevent bugs from happening in the first place
    B) Find and report bugs
    C) Fix bugs that have been reported
    D) Prevent bugs from entering the archive

I will now discuss each of these things. After that I'll finally
get to discussing automated testing the way Ian Jackson proposed it.


    A) Prevent bugs from happening in the first place
    =================================================

In general, the way to prevent bugs from happening at all is by
reducing complexity. Simple things are easier to get right.

Most programmers find that using tools with higher abstraction
levels reduces complexity and the number of bugs they create for a
given task.  As an example, writing the shell command "cat *.txt >
all.dat" much more likely to work correctly than writing the same
program in C, where you would have to open and read and write files
yourself, checking for errors, etc.

In a Debian packaging context, this might mean using packaging
helpers to take care of the boring, repetitive chores that are the
same from one package to the next. For example, debhelper is pretty
good at reducing the debian/rules file to just a handful of simple
invocations of the individual debhelper tool programs.

Each invocation is very simple. Very simple bits of code are usually
correct. Result: fewer bugs in packages and in Debian. Debhelper
is a very good thing indeed.

I'm not saying that using debhelper, or another packaging helper,
should be mandatory. They are merely one way of reducing the
probability of bugs, and because of that, I am happy that most
packages in Debian do use them. On the whole, our quality is better
thanks to that. If you can make bug free packages, by all means don't
use a helper (I don't, my packages are simple enough as they are).

There are other ways of combating complexity. Picking sensible
defaults and not making the package configurable via debconf is
simple. I'd add more examples, but I can't think of any right now
(and people on IRC are discussing dressing pork in yellow underwear,
which is highly distracting).

Where we can, we should avoid complexity and make things simpler. If
you have ideas for how to do this, please tell.


    B) Find and report bugs
    =======================

We currently have about 82 thousand bugs open (counting from the
BTS summary page on packages; this includes all severities and
tags). That's a lot of bugs. There are, however, about 11 thousand
packages with bugs, so there are only about 7.3 bugs open per
package, on average.

Our new bug numbers are in the 340 thousand range, so we've closed
258 thousand reported bugs (plus a lot of unreported ones) over
the years.  That's a truly huge number of bugs.

A bug report is a wonderful thing. It means that there is no longer
any need to wonder whether there is something wrong in the package:
you know there is, and you know what it is. Someone, a nice person
using your package, has gone to the trouble of finding out what
the problem is, and also they decided to tell you. It is delightful
when people decide to be so helpful.

Sometimes it takes a long time for anyone to report a bug. It would
be more reassuring to be more proactive in finding bugs. Our two well
known tools for this are lintian and linda. They examine packages
for patterns that tend to indicate problems. On the whole, they are
very simple systems, but even so, they find a lot of problems. Most
problems probably never enter the Debian archive, because packagers
use the tools and fix anything that needs fixing before uploading.

The tool I wrote, piuparts, is similar: it should be used by the
package maintainer before uploading, so that a buggy package never
enters the archive. Piuparts is pretty new, so it's unsurprising
that most people don't use it yet.


    C) Fix bugs that have been reported
    ===================================

Not all of the bugs the automatic tools find are fixed, however. And
there's those 82 thousand other bugs that need fixing as well. While
many of them are wish list bugs, or it is questionable if they are
bugs at all, they do all require some attention.

What can we do about that? Can we get down to no (fixable) bugs in
a stable release? (Fixable bug here means a bug that can be fixed at
all with reasonable effort by the Debian package maintainer. Having
to rewrite the X server doesn't count as reasonable effort. Wish
list bugs should probably be excluded as well.)

Having no bugs is a good state to be in. When a bug is reported,
it is usually easier to fix if there aren't a bunch of other bugs
disturbing the process.

Our Bug Squashing Parties are useful, but they mostly concentrate
on release critical bugs. Other bugs get less attention, but needs
to be fixed too. I'm guilty of ignoring many of the bugs against
my own packages for extended periods of time. People like me are
part of the problem.

Several ideas have been floating around for years on how to improve
this situation, of which I'd like to mention three. While I've here
used the number of bugs as the measure of a package's quality,
the same ideas might help with other aspects, like getting new
upstream versions packaged soon after they're released.

    * Team maintenance. If a package is maintained by a team,
      there are more people sharing the work. When a team works
      well, more people look at the package, and finding and
      fixing problems is more effective. There is less work per
      person, so things don't lag as much. A well-working team
      is a good thing.

      As an example, the Debian GNOME team seems to work really
      well. Transitions to the next upstream version happen
      quite smoothly these days.

      Mandatory teams for packages seems ridiculous to me.
      Lots of packages are so small that having to arrange a
      team for them, even if it is only the effort to set up
      and subscribe to a team mailing list, is wasteful. Not
      everyone likes to work in a close team, either, and we
      shouldn't exclude them.

    * Less strong ownership of packages. The current state in
      Debian is that the package maintainer (or maintainer
      team) owns the package, and as long as they don't cause
      a lot of trouble, and don't have release critical bugs,
      everyone else is invited to keep their hands off.

      If the maintainer, for whatever reason, can't keep the
      quality of the package up, it will have to degrade a lot
      before anyone else dares to touch it. If this
      Non-Maintainer Upload threshold were lowered, it might
      be that quality could improve. There would probably
      be a number of mistakes made, but that also happens
      when people take on a new package.

      This is not the same as maintenance by a team. An NMU
      is done by someone interested in the package for
      whatever reason, but they only do the upload to fix a
      specific problem or problems, not to join the maintainer
      team for a long time.

      This idea hasn't been tested. It could be tested if
      some group of maintainers declared that some or all
      of their packages were part of the experiment, that
      anyone could NMU them for any reason whatsoever, as
      long as they take proper care not to mess the package
      up.

      (I'm willing to participate in such an experiment
      myself, but I haven't thought out the details yet.)

    * Abolishing package ownership completely. This is a more
      radical version of the previous one. I'm not going to
      argue for it until the milder form has been tested first.

The main theme here is the need for speed: bugs need to be closed
faster, and things that help this would be good.


    D) Prevent bugs from entering the archive
    =========================================

In program development, it is usually a good idea to not commit
anything until it passes all automatic tests. Similarly, I propose
that it would be good for Debian to use some of the automatic tools
before a package is accepted into the archive. For example, if
lintian finds the init.d-script-does-not-implement-required-option
error[6], is there any reason to accept the package, since it is
certainly buggy?

        [6]
        http://lintian.debian.org/reports/Tinit.d-script-does-not-implement-required-option.html

There are some practical issues with this, of course. Not all lintian
and linda warnings should prevent accepting a package, because
that might prevent fixing more important problems quickly. That's
fine-tuning, however, the general principle still applies: it's
better to prevent a buggy package from entering than fixing it later.

Some of the automatic checking might be too heavy, or too risky, or
otherwise impractical to run when processing incoming packages. In
these cases, it is better to accept the package into the archive
and then run tests later, and file bugs for any problems found.

Lintian has been run on all packages for many years. The results
are listed on a website[7], but many packages go for months, even
years without fixing the problems. When I started running piuparts
on all packages, I decided to report any problems as bugs, instead
of just publishing log files. This seems to work better: many of
the bugs have been fixed, some of them even in the same day.

    [7] http://lintian.debian.org/


    Automated testing of program functionality
    ==========================================

I'm speaking here about whether the programs in the package work,
not whether the packaging itself works. Lintian, linda, and piuparts
already test the packaging fairly well, I think. Also, I'm speaking
about active tests that require running programs in the package;
lintian and linda don't do that, and shouldn't.

In this section especially I'm partly rephrasing what Ian Jackson
and others said in the previous discussion (or what I think they
said), partly adding my own thoughts.

Having a way to automatically test that a package is at least
minimally functional is clearly a good thing. Speaking from the
point of view of someone who occasionally does NMUs to fix release
critical bugs in other people's packages, the easier it is to
check that a package still works after I've mucked about with it,
the easier things will be for everyone involved.

Automatic testing needs to happen in various contexts:

    * When the package is being built. Most of such tests should go
      into an upstream test module. Traditionally, this would
      correspond to "make check".

    * When the package has been built, but before it is uploaded.
      This is similar to testing with lintian, linda, and piuparts.
      The difference from build-time tests is that the tests are
      run when the package is installed onto a system (possibly a
      chroot or a virtual system).

    * Before an uploaded package is accepted into the archive. This
      would prevent buggy packages from entering the archive.

    * On specifically crafted test systems. This would check that
      packages still work even though other packages they depend
      on have changed.

    * On real systems, to verify that things still work. This would
      potentially be a big help to system administrators.

Some issues:

    * Test data. Some tests are going to require a large amount
      of test data and that is best kept out of the binary package.
      It is probably best to keep it in the source package only:
      the test program then needs to install (and maybe partially
      build) the source package.

    * Test dependencies. Many tests will require using tools
      that neither using nor building the package needs. Thus we
      probably need "Tests-Depends" (for the source package).

    * Generic tests. Since Debian has so many packages, as much
      as possible should be tested using generic tests that apply
      to many packages.  For example, checking that an executable
      can be run at all should be a generic test. Expecting ten
      thousand source packages to add tests for that is unwarranted
      optimism. Generic tests should require nothing from the
      package itself.

    * Specific tests. Obviously it is also necessary for each
      package to be able to provide tests for its particular
      peculiarities, such as instances of old bugs to avoid them
      re-appearing. The interface for this should allow various
      tools to be used for implementing the tests, so that there
      is space for evolution of good tools. Compare the situation
      with a raw debian/rules file and helper packages.

    * Non-burdening of buildds. Especially slow architectures
      might want to skip build-time tests to save time. There
      should be a way to build the package without running tests,
      and of course to run the tests only.

My concrete proposals:

    * Let's write a tool that can do at least simple generic tests
      (we'll expand it later).

    * Let's standardize on a way to invoke package specific tests:
      "debian/rules test-build" for build-time tests and
      "debian/rules test-install" for tests of the installed
      package. Neither must require the package to be built
      already. The rules targets can only be assumed to exist if
      debian/control contains a "Tests-Depends".  Whoever calls
      these must take care of installing test-dependencies.

    * Let's modify pbuilder to run test-build tests and (if
      possible) also the generic tool and test-install tests.
      These belong, I think, better into pbuilder then piuparts,
      but it might be that piuparts should run them also.

    * Let's also write a tool that a sysadmin (or tester) can use to
      run test-install for particular packages, or all installed
      packages.

    * Ian's proposed debian/tests/control interface sounds
      nifty. I'm not going to debate the exact details (at least
      here), they should probably be decided by the implementer
      ("those who, decide"). It should be implementable as a tool
      that can be run from "debian/rules test-install", I think.
      This will allow Debian packagers who like it to use it, but
      those who prefer something else can use that instead.  Some
      people might want to use all available tools, even.

    * After all this is done, let's start a campaign where every bug
      fix includes a patch to add a regression test for it.


    Let's take quality assurance seriously
    ======================================

Quality assurance is currently performed by a few people organized
around the debian-qa mailing list, and various other people
(including me). I see the need for a more aggressive, systematic
approach to quality assurance.  This might be implemented by
(re-)forming the debian-qa team with a modified agenda, something
like this:

    The task of the debian-qa team is to proactively find and fix
    (technical) problems in Debian packages, and to temporarily
    maintain orphaned packages.

Some of the things that it might make sense to organize better
include (if these already are organized well, I apologize):

    * Reporting serious problems found by lintian/linda as bugs
      against packages.

    * Reporting problems found by piuparts. I already do this,
      but it would be good to expand it to a couple more
      architectures (at least), and to have more people process
      logs of failed tests.

    * Testing that all packages that are of optional or higher
      priority actually can be installed at the same time. (This
      should be partly doable by analyzing Contents files, if it
      isn't already.)

    * Testing that all packages can still be re-built even when
      compilers, libraries, or other build dependencies have
      changed.

    * Checking that a system with as many packages as possible
      installed can be upgraded from stable via testing to sid.

Then, of course, there is the fixing of bugs, but I've discussed
that above.


PS. Sorry again, my foot note numbering got confused.


--
On a clear disk, you seek forever.


--
To UNSUBSCRIBE, email to [hidden email]
with a subject of "unsubscribe". Trouble? Contact [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Thoughts on Debian quality, including automated testing

Roger Leigh
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Lars Wirzenius <[hidden email]> writes:

>     Automated testing of program functionality
>     ==========================================
>
> Automatic testing needs to happen in various contexts:
>
>     * When the package has been built, but before it is uploaded.
>       This is similar to testing with lintian, linda, and piuparts.
>       The difference from build-time tests is that the tests are
>       run when the package is installed onto a system (possibly a
>       chroot or a virtual system).

For this task, you might find schroot(1) useful.  It's a means of
accessing chroot environments, but it supports LVM snapshots as one
method.  This is a very quick method to create and destroy a test
environment (on my system, 2 seconds to create and 5 to destroy).
This is quite a low overhead compared with the alternatives (untarring
a tarfile, or copying a filesystem, or running cdebootstrap), and the
low cleanup overhead is also advantageous.

I also hope to add support for Xen on top of LVM snapshots as well.
I'm just waiting for Xen to work on powerpc so I can actually use it.
If anyone is interested who would like this, please get in touch.

sbuild is also partially integrated with sbuild (the Debian package),
though I haven't added session handling for LVM snapshots yet.  Once
tests become standardised with a debian/rules target, it would be
fairly simple to add this to sbuild.

[schroot has received some minor criticism for being written in C
using GLib/GObject.  I have however spent the last week converting it
to C++, and I'm just finishing that up now.]


Regards,
Roger

- --
Roger Leigh
                Printing on GNU/Linux?  http://gimp-print.sourceforge.net/
                Debian GNU/Linux        http://www.debian.org/
                GPG Public Key: 0x25BFB848.  Please sign and encrypt your mail.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.2 (GNU/Linux)
Comment: Processed by Mailcrypt 3.5.8+ <http://mailcrypt.sourceforge.net/>

iD8DBQFDqS5QVcFcaSW/uEgRAvZUAKCcIgh8QL33CEJO24/A9669KkvF6ACgpYdC
2s5xPCXcfUkYNZZIRCFY2so=
=huA9
-----END PGP SIGNATURE-----


--
To UNSUBSCRIBE, email to [hidden email]
with a subject of "unsubscribe". Trouble? Contact [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Thoughts on Debian quality, including automated testing

Thomas Hood-3
In reply to this post by Lars Wirzenius-2
First, thanks to Lars for drawing our attention to an important topic
and for taking an initiative that is long overdue.

Lars, I agree fully with what you say.  When it comes to team
maintenance I would go even further than you do.  You say:

>     Mandatory teams for packages seems ridiculous to me.
>     Lots of packages are so small that having to arrange a
>     team for them, even if it is only the effort to set up
>     and subscribe to a team mailing list, is wasteful. Not
>     everyone likes to work in a close team, either, and we
>     shouldn't exclude them.

I don't think that it is ridiculous to require that every package have a
team behind it---i.e., at least two maintainers.  First, if someone can't
find ONE other person willing to be named as a co-maintainer of a given
package then I would seriously doubt that that package (or that person)
is an asset to Debian.  Second, putting packages in the custody of a
team makes it easy for a tired maintainer to relinquish control.  If the
team works via an alioth project then there are many benefits. Code is
kept under version control and thus backed up; the change history can be
easily viewed by anyone; the mailing list becomes an easily browsed
history of package development.  Team maintainership is working very
well for some other distributions.

I would support requiring team maintainership because TM will be
beneficial in almost all cases and making it a requirement it cuts off a
lot of useless discussion.  There are several packages in Debian that are
notoriously undermaintained and whose maintainers have mused from time
to time about getting help, but haven't bothered to do it.  They should
be forced to get help, or to give up maintaining those packages.

Consistent with this view, I have just created teams for all my packages
even though most of them are mature.  I am glad to have the help; having
new people to work with has given me some new ideas.

Combined with the principle of non-responsibility (constitution §2.1),
the institution of exclusive solitary package ownership has made some
Debian packages into bastions of untended bugs.
--
Thomas Hood


--
To UNSUBSCRIBE, email to [hidden email]
with a subject of "unsubscribe". Trouble? Contact [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Thoughts on Debian quality, including automated testing

Lars Wirzenius
In reply to this post by Roger Leigh
ke, 2005-12-21 kello 10:28 +0000, Roger Leigh kirjoitti:
> For this task, you might find schroot(1) useful.  It's a means of
> accessing chroot environments, but it supports LVM snapshots as one
> method.

Does this require the user to set up LVM somehow before using schroot?

>   This is a very quick method to create and destroy a test
> environment (on my system, 2 seconds to create and 5 to destroy).

For me, unpacking a tar file takes about 4 seconds (from a cold cache,
machine had just been rebooted) and afterwards less than a second to
remove (but then it was all in the cache).

This is a small part of the whole process, which for piuparts can take
several minutes, if it tests upgrades from stable via testing to
unstable.

--
Debian is a beast that speaks with many voices -- R.B.


--
To UNSUBSCRIBE, email to [hidden email]
with a subject of "unsubscribe". Trouble? Contact [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Thoughts on Debian quality, including automated testing

paddy-9
In reply to this post by Lars Wirzenius-2
On Wed, Dec 21, 2005 at 02:07:30AM +0200, Lars Wirzenius wrote:
> Sloppiness tends to result in real problems sooner or later.

possible slogan for volatile-sloppy ? :)

> Several ideas have been floating around for years on how to improve
> this situation, of which I'd like to mention three. While I've here
> used the number of bugs as the measure of a package's quality,
> the same ideas might help with other aspects, like getting new
> upstream versions packaged soon after they're released.
>
>     * Team maintenance.
<snip>
>     * Less strong ownership of packages.
<snip>
>     * Abolishing package ownership completely.
<snip>

It might be implied in one of the above, but what about:

      * competitive maintenance

Sorry, I couldn't resist.

Regards,
Paddy
--
Perl 6 will give you the big knob. -- Larry Wall


--
To UNSUBSCRIBE, email to [hidden email]
with a subject of "unsubscribe". Trouble? Contact [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Thoughts on Debian quality, including automated testing

Adrian von Bidder
In reply to this post by Thomas Hood-3
On Wednesday 21 December 2005 12.23, Thomas Hood wrote:

> I don't think that it is ridiculous to require that every package have a
> team behind it---i.e., at least two maintainers.  First, if someone can't
> find ONE other person willing to be named as a co-maintainer of a given
> package then I would seriously doubt that that package (or that person)
> is an asset to Debian.

The problem is: do you honestly want to force people who don't want to have
comaintainers on their packages to leave Debian?

Or do you want people who really don't want to have comaintainers for their
packages to put somebody in just so they are following the rules, while
they regard anything done by this comaintainer on his own like they would
regard an intrusive NMU?

Don't misunderstand me: team maintenance is great, and I think it makes
sense even for small and trivial packages.  But trying to force anybody to
do anything is no productive in Debian (and we'd have to modify the
constitution for this, anyway :-)

-- vbi

--
Every religion is about absolute belief in its own superiority and the
divine right to impose its version of truth upon others.
        -- Pervez Amir Ali Hoodbhoy, Prospect Magazine Feb 2002

attachment0 (398 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Thoughts on Debian quality, including automated testing

Roger Leigh
In reply to this post by Lars Wirzenius
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Lars Wirzenius <[hidden email]> writes:

> ke, 2005-12-21 kello 10:28 +0000, Roger Leigh kirjoitti:
>> For this task, you might find schroot(1) useful.  It's a means of
>> accessing chroot environments, but it supports LVM snapshots as one
>> method.
>
> Does this require the user to set up LVM somehow before using schroot?

Yes.  You would create a separate logical volume (LV) for each
distribution you want to support, set them up with debootstrap.  Once
done, you add a configuration stanza like this:

[sid]
type=lvm-snapshot
description=Debian sid snapshot
priority=3
groups=root,sbuild
root-groups=root,sbuild
device=/dev/hda_vg/sid_chroot
mount-options=-o atime,sync,user_xattr
lvm-snapshot-options=--size 2G
run-setup-scripts=true
run-session-scripts=true

I plan to add support for tar(.gz|.bz2) and zip files as well once
I've finished the C++ conversion (the other alternatives are currently
directories and any mountable block device), then when combined with
sbuild, you'll have a system almost but not quite entirely unlike
pbuilder.  It's all nicely modular, so adding a new chroot type is
trivial.

The other advantage is that it uses PAM in a similar manner to sudo,
so you can grant certain users access to root privs (root-groups) in
the chroots, which allows them to install/upgrade/remove packages in
the chroots.  Obviously this is quite simple to abuse if you know what
you're doing, so you would only grant it to folks you could trust.
When it supports Xen, you could also grant root privs to folks you
/don't/ trust, since they would be entirely self-contained.

>>   This is a very quick method to create and destroy a test
>> environment (on my system, 2 seconds to create and 5 to destroy).
>
> For me, unpacking a tar file takes about 4 seconds (from a cold cache,
> machine had just been rebooted) and afterwards less than a second to
> remove (but then it was all in the cache).

The difference for a minimal chroot is not too great.  The main
advantage of schroot LVM snapshotting is that the time is constant
irrespective of the size of the LV (it's copy-on-write), whereas for
tar it is linear.  For slow machines and older architectures, this is
an advantage.

> This is a small part of the whole process, which for piuparts can take
> several minutes, if it tests upgrades from stable via testing to
> unstable.

OK.


- --
Roger Leigh
                Printing on GNU/Linux?  http://gimp-print.sourceforge.net/
                Debian GNU/Linux        http://www.debian.org/
                GPG Public Key: 0x25BFB848.  Please sign and encrypt your mail.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.2 (GNU/Linux)
Comment: Processed by Mailcrypt 3.5.8+ <http://mailcrypt.sourceforge.net/>

iD8DBQFDqWRvVcFcaSW/uEgRAqbKAJ9Oy4S1TT8FaHq7aZVzX7CJhpsoNQCfRZo0
3kX9PCMU7X/FZf9a8tSbLkA=
=RcKK
-----END PGP SIGNATURE-----


--
To UNSUBSCRIBE, email to [hidden email]
with a subject of "unsubscribe". Trouble? Contact [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Thoughts on Debian quality, including automated testing

Matthew Garrett-2
In reply to this post by Lars Wirzenius-2
Lars Wirzenius <[hidden email]> wrote:

>     * Less strong ownership of packages.

(snip)

>       This idea hasn't been tested. It could be tested if
>       some group of maintainers declared that some or all
>       of their packages were part of the experiment, that
>       anyone could NMU them for any reason whatsoever, as
>       long as they take proper care not to mess the package
>       up.

It's not something that's been tested in Debian, but it's the standard
way of getting things done in Ubuntu. So far, it seems to have worked
fairly well. The two situations aren't directly analagous since Ubuntu
has a smaller and more cohesive group of people involved, but I think
it's possibly indicative that this proposal would work.

I think I've said this before, but I have no objections to anyone
uploading any of my packages. I'd be even happier if anyone who did so
was willing to enter into some sort of reciprocal agreement.
--
Matthew Garrett | [hidden email]


--
To UNSUBSCRIBE, email to [hidden email]
with a subject of "unsubscribe". Trouble? Contact [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Thoughts on Debian quality, including automated testing

Erinn Clark
In reply to this post by Thomas Hood-3
* Thomas Hood <[hidden email]> [2005:12:21 12:23 +0100]:
> I don't think that it is ridiculous to require that every package have a
> team behind it---i.e., at least two maintainers.  First, if someone can't
> find ONE other person willing to be named as a co-maintainer of a given
> package then I would seriously doubt that that package (or that person)
> is an asset to Debian.  

I disagree. There are plenty of people who are maintaining packages
alone that are doing an excellent job, they just don't happen to get
shout-outs on the -qa list. Quite a lot of people in Debian are
responsible enough to go it alone as well as to know when it's time to
pass the torch. Forcing team maintenance on people would result in very
few technical enhancements for such maintainers and would probably
engender quite a bit of resentment (nevermind the fact that they'd
likely resist altogether).

It just seems to me like telling responsible DDs who've done a stellar job
that they need a babysitter is a bit... insulting.

> Second, putting packages in the custody of a
> team makes it easy for a tired maintainer to relinquish control.  

Is that always true though? For example, I can see how that could
benefit some of the more isolated members of Debian who aren't in
constant communication with other developers, but the ones who are --
half of the time they just say "anyone want this package?" and that's
about all there is to it.

> Team maintainership is working very well for some other distributions.

That may be true, but it's not a good argument for forcing such a
situation in Debian.

--
off the chain like a rebellious guanine nucleotide


--
To UNSUBSCRIBE, email to [hidden email]
with a subject of "unsubscribe". Trouble? Contact [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Thoughts on Debian quality, including automated testing

Petter Reinholdtsen
In reply to this post by Adrian von Bidder

[Thomas Hood]
> I don't think that it is ridiculous to require that every package
> have a team behind it---i.e., at least two maintainers.  First, if
> someone can't find ONE other person willing to be named as a
> co-maintainer of a given package then I would seriously doubt that
> that package (or that person) is an asset to Debian.

[Adrian von Bidder]
> The problem is: do you honestly want to force people who don't want
> to have comaintainers on their packages to leave Debian?

I'm having a hard time trying to understand the thinking of people
refusing to co-maintain packages, but am aware that there are a few of
those.  These seem to be the same with issues regarding cooperating
with the rest of the project as well, so yes, I guess I am am willing
to force them to leave Debian.

But perhaps we should not do this.  Perhaps the co-maintainer
requirement should only be implemented on high-profile packages?
Perhaps something like all packages used by more then 10% of the
debian population (as measured using popularity-contest) should be
required to have co-maintainers?  This way the people with
co-maintainer issues can do the little used packages, and the more
used (and presumably more important) packages will have a team of
people working on them.

At the very minimum, I believe all base packages (those installed by
debootstrap by default) should have co-maintainers.

If the package is important for Debian, it is too important to leave
in the hand of only one person.  People go tired, get occupied with
real life or are run over by a bus.  The maintenence of important
packages in Debian should not stop when any of these things happen,
and because of this I seriously believe all packages in Debian should
have co-maintainers.

> Or do you want people who really don't want to have comaintainers
> for their packages to put somebody in just so they are following the
> rules, while they regard anything done by this comaintainer on his
> own like they would regard an intrusive NMU?

That would not be co-maintenence, but lying to the project about how
they follow the project rules.  Should we trust these people to
maintain packages in Debian?

And yes, we would probably need to change the constitution to make
this happen, so it will take a while.  :/


--
To UNSUBSCRIBE, email to [hidden email]
with a subject of "unsubscribe". Trouble? Contact [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Thoughts on Debian quality, including automated testing

Thijs Kinkhorst
On Wed, 2005-12-21 at 17:08 +0100, Petter Reinholdtsen wrote:
> At the very minimum, I believe all base packages (those installed by
> debootstrap by default) should have co-maintainers.

This sounds like a good compromise between the two sides of this
discussion.


Thijs

signature.asc (196 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Thoughts on Debian quality, including automated testing

Thomas Hood-6
In reply to this post by Erinn Clark
I wrote:
> I don't think that it is ridiculous to require that every package have
> a team behind it---i.e., at least two maintainers.  First, if someone
> can't find ONE other person willing to be named as a co-maintainer of
> a given package then I would seriously doubt that that package (or
> that person) is an asset to Debian.

Erinn Clark wrote:
> There are plenty of people who are maintaining packages alone
> that are doing an excellent job

True.  However, the issue in question is whether or not it would be
better if they maintained in teams.

> Forcing team maintenance on people would result in very few
> technical enhancements for such maintainers

How do you know?  I would expect most packages to benefit.  Every
person brings different expertise to the table.

> It just seems to me like telling responsible DDs who've done a
> stellar job that they need a babysitter is a bit... insulting.

This is not a fair characterization of what the introduction of
a two-maintainer rule would be doing.  No one should be insulted
by general rule changes designed to make Debian work better.
--
Thomas Hood


--
To UNSUBSCRIBE, email to [hidden email]
with a subject of "unsubscribe". Trouble? Contact [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Thoughts on Debian quality, including automated testing

David Nusinow
In reply to this post by Erinn Clark
On Wed, Dec 21, 2005 at 11:00:15AM -0500, Erinn Clark wrote:
> * Thomas Hood <[hidden email]> [2005:12:21 12:23 +0100]:
> > Team maintainership is working very well for some other distributions.
>
> That may be true, but it's not a good argument for forcing such a
> situation in Debian.

I agree that we shouldn't force teams on anyone, but I'd like to see more
large-scale teams encompassing loosely connected smaller packages[0]. If,
for no other reason, than for developers to claim ownership of (and by
extension responsibility for) the whole project rather than their little
package feifdom. It might help overcome the sense that key people aren't
communicating, as well as provide more easy avenues for NM's to get
involved that don't simply involve packaging some crap little script from
freshmeat.

 - David Nusinow

[0] A Debian Games team would be a perfect hypothetical example of this. So
    is the Debian libruby extras team I helped create.


--
To UNSUBSCRIBE, email to [hidden email]
with a subject of "unsubscribe". Trouble? Contact [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Thoughts on Debian quality, including automated testing

Clint Adams
In reply to this post by Thomas Hood-6
> True.  However, the issue in question is whether or not it would be
> better if they maintained in teams.

I imagine that it would not be better.


--
To UNSUBSCRIBE, email to [hidden email]
with a subject of "unsubscribe". Trouble? Contact [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Thoughts on Debian quality, including automated testing

Erinn Clark
In reply to this post by Thomas Hood-6
* Thomas Hood <[hidden email]> [2005:12:21 17:32 +0100]:

> Erinn Clark wrote:
> > There are plenty of people who are maintaining packages alone
> > that are doing an excellent job
>
> True.  However, the issue in question is whether or not it would be
> better if they maintained in teams.
>
> > Forcing team maintenance on people would result in very few
> > technical enhancements for such maintainers
>
> How do you know?  I would expect most packages to benefit.  Every
> person brings different expertise to the table.

For maintainers who are doing a lot of good work, there's simply not
enough to justify more people. Once there's already a certain level of
efficiency, adding another person is not going to increase it, and will
likely decrease it. I can't see the point of enforcing this as a rule,
which, luckily, was not proposed by Lars.

> > It just seems to me like telling responsible DDs who've done a
> > stellar job that they need a babysitter is a bit... insulting.
>
> This is not a fair characterization of what the introduction of
> a two-maintainer rule would be doing.  No one should be insulted
> by general rule changes designed to make Debian work better.

Bureaucracy is often designed to do lots of things "better" and it often
doesn't achieve them. It creates needless hassle, more 'paperwork', and
has very few benefits besides making people feel like they've done
something useful when they haven't.

Of course, we're both starting from entirely different premises (yours
that all packages are better maintained by more than one person, mine
that this is not universally true and can be worse in some cases) so
there's probably not a lot of wiggle room for agreement here.

--
off the chain like a rebellious guanine nucleotide


--
To UNSUBSCRIBE, email to [hidden email]
with a subject of "unsubscribe". Trouble? Contact [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Thoughts on Debian quality, including automated testing

David Nusinow
In reply to this post by Thomas Hood-6
On Wed, Dec 21, 2005 at 05:32:21PM +0100, Thomas Hood wrote:
> This is not a fair characterization of what the introduction of
> a two-maintainer rule would be doing.  No one should be insulted
> by general rule changes designed to make Debian work better.

I think a two-maintainer rule is a bit artificial and perhaps the wrong
solution. Taking a cue from teams that work well in Debian (Gnome, d-i,
etc) indicates that somewhat larger teams with a common goal and a fairly
large set of packages under their umbrella will work in practice. This
provides the opportunity for autonomy (someone can take ownership of a
package within the team) but also a cohesiveness and a known set of people
to turn to when you're in need.

It's pretty simple to found such a team too. All it takes is some
interested people and an alioth project.

 - David Nusinow


--
To UNSUBSCRIBE, email to [hidden email]
with a subject of "unsubscribe". Trouble? Contact [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Thoughts on Debian quality, including automated testing

Russ Allbery-2
In reply to this post by Thijs Kinkhorst
Thijs Kinkhorst <[hidden email]> writes:
> On Wed, 2005-12-21 at 17:08 +0100, Petter Reinholdtsen wrote:

>> At the very minimum, I believe all base packages (those installed by
>> debootstrap by default) should have co-maintainers.

> This sounds like a good compromise between the two sides of this
> discussion.

packages.qa.debian.org already bugs you to find a co-maintainer if the
package is priority standard or higher.  It's a good hint but not
mandatory, which feels about right to me.

I don't really understand what requiring a package to have a co-maintainer
looks like.  Do you remove the package from Debian if there's no
co-maintainer?  Surely not for base packages.  Do you force a
co-maintainer on the package somehow?  How?  What if someone is just
listed as a co-maintainer but never does anything?

Also, I think this is a little silly for small packages.  My experience
with this sort of volunteer work in other areas is that if one person does
nearly all the work on a regular basis, you're not gaining that much by
having a backup.  The person who is theoretically the backup isn't up to
speed on the package anyway and is going to be starting roughly as cold as
any other random person out there.  And there simply isn't enough work for
two people for a lot of packages.

I think that the energy used to define these sorts of procedures is
probably better used finding a package with a large bug count and
volunteering to work with the maintainer to try to get the bug count down.

--
Russ Allbery ([hidden email])               <http://www.eyrie.org/~eagle/>


--
To UNSUBSCRIBE, email to [hidden email]
with a subject of "unsubscribe". Trouble? Contact [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Thoughts on Debian quality, including automated testing

Steve Langasek
In reply to this post by Lars Wirzenius-2
On Wed, Dec 21, 2005 at 02:07:30AM +0200, Lars Wirzenius wrote:
> Several ideas have been floating around for years on how to improve
> this situation, of which I'd like to mention three. While I've here
> used the number of bugs as the measure of a package's quality,
> the same ideas might help with other aspects, like getting new
> upstream versions packaged soon after they're released.

>     * Team maintenance. If a package is maintained by a team,
>       there are more people sharing the work. When a team works
>       well, more people look at the package, and finding and
>       fixing problems is more effective. There is less work per
>       person, so things don't lag as much. A well-working team
>       is a good thing.

>       As an example, the Debian GNOME team seems to work really
>       well. Transitions to the next upstream version happen
>       quite smoothly these days.

OTOH, sometimes "team maintenance" means "no one takes responsibility for
the package."  I've dealt with release critical bugs on packages where the
Maintainer field pointed to a "team" mailing list, there were several
Uploaders listed for the package, and the bug sat unattended for well over a
month for no apparent reason.

I've also *been* part of maintainer teams where all the members of the team
were busy people, and bugs tended to linger because they were somebody
else's problem.

Anyway, I agree that team maintenance can be a force for good.

I also agree with the rest of your mail, including the call for a more
proactive QA team.

Cheers,
--
Steve Langasek                   Give me a lever long enough and a Free OS
Debian Developer                   to set it on, and I can move the world.
[hidden email]                                   http://www.debian.org/

signature.asc (196 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Thoughts on Debian quality, including automated testing

Daniel Ruoso
In reply to this post by Matthew Garrett-2
Em Qua, 2005-12-21 às 14:34 +0000, Matthew Garrett escreveu:
> I think I've said this before, but I have no objections to anyone
> uploading any of my packages. I'd be even happier if anyone who did so
> was willing to enter into some sort of reciprocal agreement.

So do I, but I would be really happy if the uploader knows how I
maintain the packages (I mean, using CVS, SVN or such) and cooperate
using such tools.

Maybe it would be interesting to have some information in the package
saying how the package is managed and the preferrable way of doing an
NMU (I actually, think that it's desirable to self-include in the
Uploaders field, to acquire responsability also)...

daniel


--
To UNSUBSCRIBE, email to [hidden email]
with a subject of "unsubscribe". Trouble? Contact [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Thoughts on Debian quality, including automated testing

Thomas Hood-6
In reply to this post by Erinn Clark
Erinn Clark wrote:
> For maintainers who are doing a lot of good work, there's simply not
> enough to justify more people. Once there's already a certain level of
> efficiency, adding another person is not going to increase it, and will
> likely decrease it. I can't see the point of enforcing this as a rule,


There is a point if it helps more than it hurts.  That is how rules have
to be judged.  Now, you might say that rules are stupid and those people
who need help should just go and get it.  But experience has shown that,
too often, they don't.

How much would this rule "hurt" those lone ranger maintainers you are
talking about, the ones who package everything perfectly and cannot
possibly do any better?

It turns out that there is no need for them to be hurt at all.  Lone
can carry on working as before and find a co-maintainer who won't get
in his way.  But when Lone falls off his horse he'll be glad that Tonto
is nearby.  

In other words, this rule can have only positive effects.  :)


> > This is not a fair characterization of what the introduction of
> > a two-maintainer rule would be doing.  No one should be insulted
> > by general rule changes designed to make Debian work better.
>
> Bureaucracy is often designed to do lots of things "better" and it often
> doesn't achieve them. It creates needless hassle, more 'paperwork', and
> has very few benefits besides making people feel like they've done
> something useful when they haven't.


You are saying that requiring people to find co-maintainers is
"bureaucracy"?  Someone I know well recently got co-maintainers for
three of his packages by posting a single message to debian-devel.
That's less of a burden than that imposed by many another Debian rule.

Fortunately for your position, it probably won't take arguments to kill
this idea.
--
Thomas Hood


--
To UNSUBSCRIBE, email to [hidden email]
with a subject of "unsubscribe". Trouble? Contact [hidden email]

12345