Introducting Debian Trends: historical graphs about Debian packaging practices, and "packages smells"

classic Classic list List threaded Threaded
37 messages Options
12
Reply | Threaded
Open this post in threaded view
|

Introducting Debian Trends: historical graphs about Debian packaging practices, and "packages smells"

Lucas Nussbaum-4
TL;DR: see https://trends.debian.net and
https://trends.debian.net/#smells

Hi,

Following this blog post[1] I did some work on setting up a proper
framework to graph historical trends about Debian packaging practices.
The result is now available at [2], and I'm confident that I will be
able to update this on a regular basis (every few months).

Additionally (and much more controversially I guess :-) ) I also added
an analysis of "package smells"[3], such as "not using dh", "not using a
recent debhelper compat level", "not using a 3.0 source format", etc. I
understand that in some cases there might be good reasons to keep those
"smells", but I find it valuable to have them presented in a more
actionable way to fix the cases that should be fixed. So there's a list
of smells, sorted by maintainer/uploader [4].

Given that Debian is currently frozen to prepare the buster release,
this is a bad time to start fixing those smells, but I will send a
reminder to debian-devel@ once buster is released. (It's interesting to
see how the number of smells plateaued during previous freezes).

- Lucas

[1] https://www.lucas-nussbaum.net/blog/?p=945
[2] https://trends.debian.net/
[3] https://trends.debian.net/#smells
[4] https://trends.debian.net/smells-dd-list.txt

signature.asc (849 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Introducting Debian Trends: historical graphs about Debian packaging practices, and "packages smells"

Bas Couwenberg-3
On 4/13/19 10:20 AM, Lucas Nussbaum wrote:
> Additionally (and much more controversially I guess :-) ) I also added
> an analysis of "package smells"[3], such as "not using dh", "not using a
> recent debhelper compat level", "not using a 3.0 source format", etc. I
> understand that in some cases there might be good reasons to keep those
> "smells", but I find it valuable to have them presented in a more
> actionable way to fix the cases that should be fixed. So there's a list
> of smells, sorted by maintainer/uploader [4].

You should check if those issues present in unstable are not already
fixed in experimental.

Kind Regards,

Bas

--
 GPG Key ID: 4096R/6750F10AE88D4AF1
Fingerprint: 8182 DE41 7056 408D 6146  50D1 6750 F10A E88D 4AF1


signature.asc (849 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Introducing Debian Trends: historical graphs about Debian packaging practices, and "packages smells"

Lucas Nussbaum-4
On 13/04/19 at 10:24 +0200, Sebastiaan Couwenberg wrote:

> On 4/13/19 10:20 AM, Lucas Nussbaum wrote:
> > Additionally (and much more controversially I guess :-) ) I also added
> > an analysis of "package smells"[3], such as "not using dh", "not using a
> > recent debhelper compat level", "not using a 3.0 source format", etc. I
> > understand that in some cases there might be good reasons to keep those
> > "smells", but I find it valuable to have them presented in a more
> > actionable way to fix the cases that should be fixed. So there's a list
> > of smells, sorted by maintainer/uploader [4].
>
> You should check if those issues present in unstable are not already
> fixed in experimental.
Yeah. Currently my code uses a partial mirror of snapshot.debian.org,
and I ran a custom lintian myself on 339491 source packages.

Ideally all the needed additional tags will be added to lintian, and
doing a more detailed and "real-time" analysis will be possible just by
looking at the lintian results on https://lintian.debian.org. (Once it's
available on lintian.d.o, it's easy to generate the DD list using UDD,
for example).

Lucas

signature.asc (849 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Introducting Debian Trends: historical graphs about Debian packaging practices, and "packages smells"

Holger Levsen-2
In reply to this post by Lucas Nussbaum-4
Hi Lucas,

On Sat, Apr 13, 2019 at 10:20:53AM +0200, Lucas Nussbaum wrote:
> TL;DR: see https://trends.debian.net and
> https://trends.debian.net/#smells

that's beautiful! thank you!

> [4] https://trends.debian.net/smells-dd-list.txt

for me there are two smelly packages, src:tuxtype should use source 3.0
indeed, but for src:piuparts I really don't see why. Though I would
quite probably agree that src:piuparts should not be native, and then
3.0 would make more sense. But you don't consider native packages as
smelly, which I think you maybe should.


--
tschau,
        Holger

-------------------------------------------------------------------------------
               holger@(debian|reproducible-builds|layer-acht).org
       PGP fingerprint: B8BF 5413 7B09 D35C F026 FE9D 091A B856 069A AA1C

Dance like no one's watching. Encrypt like everyone is.

signature.asc (849 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Introducting Debian Trends: historical graphs about Debian packaging practices, and "packages smells"

Lucas Nussbaum-4
On 13/04/19 at 09:28 +0000, Holger Levsen wrote:

> Hi Lucas,
>
> On Sat, Apr 13, 2019 at 10:20:53AM +0200, Lucas Nussbaum wrote:
> > TL;DR: see https://trends.debian.net and
> > https://trends.debian.net/#smells
>
> that's beautiful! thank you!
>
> > [4] https://trends.debian.net/smells-dd-list.txt
>
> for me there are two smelly packages, src:tuxtype should use source 3.0
> indeed, but for src:piuparts I really don't see why. Though I would
> quite probably agree that src:piuparts should not be native, and then
> 3.0 would make more sense.
Well you could switch to 3.0 (native). Are there good reasons to stay
with 1.0 for a native package ? Even if I agree that there's probably
not much to gain, there's also even less to lose.

> But you don't consider native packages as smelly, which I think you maybe should.

I'm not sure we have ever had a real discussion about that. But yes that
might be true. It would make it easier to expose changes in derivatives.

Lucas

signature.asc (849 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

native packages? (Re: Introducting Debian Trends: historical graphs about Debian packaging practices, and "packages smells")

Holger Levsen-2
On Sat, Apr 13, 2019 at 11:42:38AM +0200, Lucas Nussbaum wrote:
> Well you could switch to 3.0 (native).

I see no point whatsoever in 3.0 (native).

IMO 3.0 (quilt) is sensible and 1.0 too, whether native or not. *If*
native package in todays world are still sensible...

> > But you don't consider native packages as smelly, which I think you maybe should.
> I'm not sure we have ever had a real discussion about that. But yes that
> might be true. It would make it easier to expose changes in derivatives.

I also see no benefit of native packages. None. I've just not done the
change for src:piuparts as change is hard :)

Maybe I forgot some argument in favor of native packages, if so, I'd be
glad to be reminded. (Except change is hard, which isn't a good argument
forever.)


--
tschau,
        Holger

-------------------------------------------------------------------------------
               holger@(debian|reproducible-builds|layer-acht).org
       PGP fingerprint: B8BF 5413 7B09 D35C F026 FE9D 091A B856 069A AA1C

Dance like no one's watching. Encrypt like everyone is.

signature.asc (849 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Introducting Debian Trends: historical graphs about Debian packaging practices, and "packages smells"

Shengjing Zhu-3
In reply to this post by Lucas Nussbaum-4
On Sat, Apr 13, 2019 at 4:21 PM Lucas Nussbaum <[hidden email]> wrote:
>
> TL;DR: see https://trends.debian.net and
> https://trends.debian.net/#smells
>

These graphs look ambiguous... Shouldn't the x-axis be year?

--
Shengjing Zhu

Reply | Threaded
Open this post in threaded view
|

Re: Introducting Debian Trends: historical graphs about Debian packaging practices, and "packages smells"

Holger Levsen-2
In reply to this post by Lucas Nussbaum-4
On Sat, Apr 13, 2019 at 10:20:53AM +0200, Lucas Nussbaum wrote:
> https://trends.debian.net/#smells

there are two minor issues with the smells *graph*: not using salsa
should only be graphed since salsa exists (and not since 2005), same for
compat < 9.


--
tschau,
        Holger

-------------------------------------------------------------------------------
               holger@(debian|reproducible-builds|layer-acht).org
       PGP fingerprint: B8BF 5413 7B09 D35C F026 FE9D 091A B856 069A AA1C

signature.asc (849 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: native packages? (Re: Introducting Debian Trends: historical graphs about Debian packaging practices, and "packages smells")

Simon McVittie-7
In reply to this post by Holger Levsen-2
On Sat, 13 Apr 2019 at 10:04:10 +0000, Holger Levsen wrote:
> I see no point whatsoever in 3.0 (native).

The main advantage of 3.0 (native) is that it makes it explicit that
the package is deliberately native, whereas a 1.0 native package is
indistinguishable from a package that was intended to be 1.0 non-native
but ended up native because the maintainer forgot to have the orig.tar.gz
in the right place when building it.

(The presence or absence of a -revision in the version number should in
theory be well-correlated with non-native or native packaging, but this
doesn't always hold - for example python3-defaults is a native package
with a non-native-style version number. Perhaps Policy should require
packages like python3-defaults to use a native-style version number like
3.7.3+1 instead of 3.7.3-1?)

dpkg also compresses 3.0 (native) packages with xz by default.

    smcv

Reply | Threaded
Open this post in threaded view
|

Re: native packages? (Re: Introducting Debian Trends: historical graphs about Debian packaging practices, and "packages smells")

Holger Levsen-2
On Sat, Apr 13, 2019 at 01:48:01PM +0100, Simon McVittie wrote:
> On Sat, 13 Apr 2019 at 10:04:10 +0000, Holger Levsen wrote:
> > I see no point whatsoever in 3.0 (native).
> The main advantage of 3.0 (native) is that it makes it explicit that
> the package is deliberately native [...]

ok, sorry, I ment to say: I see no point whatsoever in native packages.
AFAICS there are no advantages, just downsides.

What's the point/advantage of native packages?


--
tschau,
        Holger

-------------------------------------------------------------------------------
               holger@(debian|reproducible-builds|layer-acht).org
       PGP fingerprint: B8BF 5413 7B09 D35C F026 FE9D 091A B856 069A AA1C

signature.asc (849 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: native packages? (Re: Introducting Debian Trends: historical graphs about Debian packaging practices, and "packages smells")

Andrey Rahmatullin-3
On Sat, Apr 13, 2019 at 12:59:19PM +0000, Holger Levsen wrote:
> > > I see no point whatsoever in 3.0 (native).
> > The main advantage of 3.0 (native) is that it makes it explicit that
> > the package is deliberately native [...]
>
> ok, sorry, I ment to say: I see no point whatsoever in native packages.
> AFAICS there are no advantages, just downsides.
>
> What's the point/advantage of native packages?
No need to make a separate orig tarball.

--
WBR, wRAR

signature.asc (911 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: native packages? (Re: Introducting Debian Trends: historical graphs about Debian packaging practices, and "packages smells")

Alf Gaida
On 13.04.19 15:07, Andrey Rahmatullin wrote:
> On Sat, Apr 13, 2019 at 12:59:19PM +0000, Holger Levsen wrote:
>>>> I see no point whatsoever in 3.0 (native).
>> What's the point/advantage of native packages?
> No need to make a separate orig tarball.
Can't agree more, there are places where 3.0 (quilt|git|anything esls)
don't make any sense, think about meta packages with no upstream tarball
and such things.

And i wouldn't consider 1.0 bad or smelly, i use it on a regular base
for git based builds in my experimental snapshots, it's simply the best
format to do so (just put the new sources in and be done with)

Cheers Alf

Reply | Threaded
Open this post in threaded view
|

Re: Introducting Debian Trends: historical graphs about Debian packaging practices, and "packages smells"

gregor herrmann-3
In reply to this post by Lucas Nussbaum-4
On Sat, 13 Apr 2019 10:20:53 +0200, Lucas Nussbaum wrote:

> TL;DR: see https://trends.debian.net and
> https://trends.debian.net/#smells

Very nice, thank you.
 
> [4] https://trends.debian.net/smells-dd-list.txt

This list is slightly unhelpful (for my case / the Debian Perl Group)
as it reports hundreds [0] of "git repository still hosted on alioth"
warnings which are not factually true.

(I know, the Vcs-* fields in the packages in the archive still point
to alioth for all the not-yet-uploaded packages but the repos are
already moved …)


Cheers,
gregor

[0] for the perl team: 2266

--
 .''`.  https://info.comodo.priv.at -- Debian Developer https://www.debian.org
 : :' : OpenPGP fingerprint D1E1 316E 93A7 60A8 104D  85FA BB3A 6801 8649 AA06
 `. `'  Member VIBE!AT & SPI Inc. -- Supporter Free Software Foundation Europe
   `-   NP: Leonard Cohen: Suzanne

signature.asc (981 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Introducting Debian Trends: historical graphs about Debian packaging practices, and "packages smells"

Marco d'Itri
In reply to this post by Lucas Nussbaum-4
On Apr 13, Lucas Nussbaum <[hidden email]> wrote:

> TL;DR: see https://trends.debian.net and
Nice.

I suggest to add a graph detailing:
- packages with at least one init script
- packages with at least one systemd unit
- packages with at least one init script and one systemd unit

Also, did I miss the memo about traditional debhelper being superseded
by dh? Why should I switch?

--
ciao,
Marco

signature.asc (673 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Introducting Debian Trends: historical graphs about Debian packaging practices, and "packages smells"

Andreas Tille-5
In reply to this post by gregor herrmann-3
On Sat, Apr 13, 2019 at 03:46:57PM +0200, gregor herrmann wrote:
> On Sat, 13 Apr 2019 10:20:53 +0200, Lucas Nussbaum wrote:
>
> > TL;DR: see https://trends.debian.net and
> > https://trends.debian.net/#smells
>
> Very nice, thank you.

+1
I like it a lot!

> > [4] https://trends.debian.net/smells-dd-list.txt

As far as I understood you are using lintian results.  However,
the real packaging does not (always) reflect this.  At least I've
found an example.  Link [4] lists:

   
    biococoa (U)         does not use Debhelper (no compat level found) (source version: 2.2.2-4)
    biococoa (U)         should switch to dh. Current build system: cdbs (source version: 2.2.2-4)


However:  

   $ apt-get source biococoa
   $ cd biococoa-2.2.2/
   $ cat debian/compat
   10
   $ grep debhelper debian/control
   debhelper (>= 11~),

Are you sure that you are not tricked by false positives from lintian?

Kind regards and thanks a lot for this great effort.

       Andreas.

--
http://fam-tille.de

Reply | Threaded
Open this post in threaded view
|

Re: Introducting Debian Trends: historical graphs about Debian packaging practices, and "packages smells"

Bastian Blank
On Mon, Apr 15, 2019 at 04:55:12PM +0200, Andreas Tille wrote:
>     biococoa (U)         does not use Debhelper (no compat level found) (source version: 2.2.2-4)
>     biococoa (U)         should switch to dh. Current build system: cdbs (source version: 2.2.2-4)

| % grep cdbs -r biococoa-2.2.2
| biococoa-2.2.2/debian/rules:include /usr/share/cdbs/1/rules/gnustep.mk
| biococoa-2.2.2/debian/rules:include /usr/share/cdbs/1/class/gnumakefile.mk
| biococoa-2.2.2/debian/control:Build-Depends: cdbs,
| biococoa-2.2.2/debian/changelog:     dh_installsystemd instead -> that's a cdbs issue)
| biococoa-2.2.2/debian/changelog:  * debian/control (Build-Depends): Add cdbs and quilt.  Drop gnustep-make

Regards,
Bastian

--
Military secrets are the most fleeting of all.
                -- Spock, "The Enterprise Incident", stardate 5027.4

Reply | Threaded
Open this post in threaded view
|

Re: Introducting Debian Trends: historical graphs about Debian packaging practices, and "packages smells"

Lucas Nussbaum-4
In reply to this post by Andreas Tille-5
On 15/04/19 at 16:55 +0200, Andreas Tille wrote:
> Are you sure that you are not tricked by false positives from lintian?

I might be, but if lintian reports something incorrectly about your
package, it's probably worth fixing in lintian.

Lucas

Reply | Threaded
Open this post in threaded view
|

Re: native packages? (Re: Introducting Debian Trends: historical graphs about Debian packaging practices, and "packages smells")

Sam Hartman-3
In reply to this post by Holger Levsen-2
>>>>> "Holger" == Holger Levsen <[hidden email]> writes:

    Holger> On Sat, Apr 13, 2019 at 01:48:01PM +0100, Simon McVittie wrote:
    >> On Sat, 13 Apr 2019 at 10:04:10 +0000, Holger Levsen wrote:
    >> > I see no point whatsoever in 3.0 (native).  The main advantage
    >> of 3.0 (native) is that it makes it explicit that the package is
    >> deliberately native [...]

    Holger> ok, sorry, I ment to say: I see no point whatsoever in
    Holger> native packages.  AFAICS there are no advantages, just
    Holger> downsides.

I work on a lot of packages that don't really produce upstream tarballs.
It's painful and makes the workflow less fun to have to go deal with
upstream tarballs myself and I don't think it adds anything to the
workflow.  Upstream tarballs are perhaps nice if upstream actually
produces them (although I think even that is a discussion we may want to
have long-term as we move everything to git).

However if my sources are in git, git is the definitive format for
thinking about things, and the dsc I'm producing is only for the
convenience of the archive, I don't want to deal with an upstream
tarball.
This is even more true if I happen to be using dgit.

--Sam

Reply | Threaded
Open this post in threaded view
|

Re: native packages? (Re: Introducting Debian Trends: historical graphs about Debian packaging practices, and "packages smells")

Marco d'Itri
On Apr 15, Sam Hartman <[hidden email]> wrote:

> However if my sources are in git, git is the definitive format for
> thinking about things, and the dsc I'm producing is only for the
> convenience of the archive, I don't want to deal with an upstream
> tarball.
Generating an upstream tarball in this case is still useful because this
way we do not need to upload and store forever the full source archive
every time that something changes only in the packaging.

--
ciao,
Marco

signature.asc (673 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: native packages? (Re: Introducting Debian Trends: historical graphs about Debian packaging practices, and "packages smells")

Simon Richter
Hi,

On 15.04.19 21:23, Marco d'Itri wrote:

> Generating an upstream tarball in this case is still useful because this
> way we do not need to upload and store forever the full source archive
> every time that something changes only in the packaging.

That, and upstream tarballs generated with "git archive" have a magic
comment tag that says which revision was used to build them, so we don't
even lose information.

$ git archive --format=tar HEAD | git get-tar-commit-id
973188bd3b4628646c53ca9ab9bf71cc95eadd43

   Simon


signature.asc (499 bytes) Download Attachment
12