What to do with the "Upstream info" links in the PTS and the packages-metadata branch in collab-qa ?

classic Classic list List threaded Threaded
12 messages Options
Reply | Threaded
Open this post in threaded view
|

What to do with the "Upstream info" links in the PTS and the packages-metadata branch in collab-qa ?

Charles Plessy-12
Hello everybody,

sorry for the longish title... In brief, some years ago I was aiming at
gathering debian/upstream/metadata files from the VCS in which each
source packages is stored, and mirror them in a central repository
easily accessible.

As a repository I chose the collab-qa Subversion repository on Alioth.
And for the gathering I was relying on a service that listened to the
emails posted on the debian-devel-changes, fetched the metadata file
from the package's VCS if available, and commited it in the collab-qa
repository.

There are two reasons why I would like to interrupt this experiment (if
not abandonning it).  1) Alioth is going down soon.  2) The gatherer I
wrote in Perl (umegaya) had a bug that I never managed to track down,
and my attempts to rewrite it in Haskell failed (I never found enough
time to excell in Haskell...).

At the moment, the PTS points to files in the collab-qa repository
via Alioth' viewvc browser.  Can somebody disable the links ?  The
repository is completely outdated anyway.

I plan to "svn rm" the packages-metadata directory in collab-qa.  Would
that cause any trouble to somebody ?

With our brand new GitLab forge and its continuous integration system,
there may be something to do in the future.  But I do not have the
time yet.  Anbody interested in gathering the debian/upstream/metadata
files somewhere is welcome !

Have a nice day,

Charles

--
Charles Plessy
Debian Med packaging team,
http://www.debian.org/devel/debian-med
Tsurumi, Kanagawa, Japan

Reply | Threaded
Open this post in threaded view
|

Re: What to do with the "Upstream info" links in the PTS and the packages-metadata branch in collab-qa ?

Paul Wise via nm
On Mon, May 7, 2018 at 8:37 PM, Charles Plessy wrote:

> At the moment, the PTS points to files in the collab-qa repository
> via Alioth' viewvc browser.  Can somebody disable the links ?  The
> repository is completely outdated anyway.

Done in git:

https://salsa.debian.org/qa/pts/commit/ceda19c25cfc146fbe6445e9264ef18cc44cc342

> The gatherer I wrote in Perl (umegaya)

Is the code for this somewhere? Looks like it got deleted from alioth.

> And for the gathering I was relying on a service that listened to the
> emails posted on the debian-devel-changes, fetched the metadata file
> from the package's VCS if available, and commited it in the collab-qa
> repository.
...
> With our brand new GitLab forge and its continuous integration system,
> there may be something to do in the future.

If someone revives this service, this approach missed packages that do
not use a VCS. This could be worked around by the addition of fetching
from source packages when there is no VCS. The VCS should be preferred
since source packages don't need to be updated immediately when
upstream metadata changes.

--
bye,
pabs

https://wiki.debian.org/PaulWise

Reply | Threaded
Open this post in threaded view
|

Re: What to do with the "Upstream info" links in the PTS and the packages-metadata branch in collab-qa ?

Stuart Prescott-11
Paul Wise wrote:
> If someone revives this service, this approach missed packages that do
> not use a VCS. This could be worked around by the addition of fetching
> from source packages when there is no VCS. The VCS should be preferred
> since source packages don't need to be updated immediately when
> upstream metadata changes.

or looking at Contents-source.gz and linking to sources.debian.net. (or
asking ftp-master to export these files like they do with changelogs,
copyright files etc.)

Are there any other consumers of these files?

If there are no longer any consumers, should we really be nagging
maintainers to include them?

$ lintian-info --list-tags | grep upstream-metadata
upstream-metadata-file-is-missing
upstream-metadata-is-not-a-file
upstream-metadata-yaml-invalid

While the last two still make sense (if the file is there, make sure it is
valid), does the first one? I've never been a fan of lintian suggesting that
maintainers add a file that for most(?) packages, is not going to contain
anything that is not duplicated information from elsewhere in the package.

(If there are other users of this information, great...)

Stuart

--
Stuart Prescott    http://www.nanonanonano.net/   [hidden email]
Debian Developer   http://www.debian.org/         [hidden email]
GPG fingerprint    90E2 D2C1 AD14 6A1B 7EBB 891D BBC1 7EBB 1396 F2F7

Reply | Threaded
Open this post in threaded view
|

Re: What to do with the "Upstream info" links in the PTS and the packages-metadata branch in collab-qa ?

Paul Wise via nm
On Tue, May 8, 2018 at 4:02 PM, Stuart Prescott wrote:

> or looking at Contents-source.gz and linking to sources.debian.net. (or
> asking ftp-master to export these files like they do with changelogs,
> copyright files etc.)

Good idea. vcswatch could take over the VCS scanning too.

> Are there any other consumers of these files?

No services, but potentially tracker.d.o and packages.d.o.

People interacting with the source package might use them.

> If there are no longer any consumers, should we really be nagging
> maintainers to include them?

I think they are useful even if no services touch them.

> While the last two still make sense (if the file is there, make sure it is
> valid), does the first one? I've never been a fan of lintian suggesting that
> maintainers add a file that for most(?) packages, is not going to contain
> anything that is not duplicated information from elsewhere in the package.

The main value is aggregating the information and standardising it,
within upstream codebases the info could be in DOAP files or READMEs
or README.foo or HACKING etc. Some things like fundraising might only
be on the upstream website.

--
bye,
pabs

https://wiki.debian.org/PaulWise

Reply | Threaded
Open this post in threaded view
|

Re: What to do with the "Upstream info" links in the PTS and the packages-metadata branch in collab-qa ?

gregor herrmann-3
On Tue, 08 May 2018 17:22:40 +0800, Paul Wise wrote:

> On Tue, May 8, 2018 at 4:02 PM, Stuart Prescott wrote:

> > Are there any other consumers of these files?
> No services, but potentially tracker.d.o and packages.d.o.
> People interacting with the source package might use them.

Right, in the perl-team we (and some of our dpt-* tools) use them in
the git repos of our source packages (in practice mostly the
Repository line to add the upstream repo, and sometimes the
Bug-Database field for forwarding bugs/patches).
 

Cheers,
gregor

--
 .''`.  https://info.comodo.priv.at -- Debian Developer https://www.debian.org
 : :' : OpenPGP fingerprint D1E1 316E 93A7 60A8 104D  85FA BB3A 6801 8649 AA06
 `. `'  Member VIBE!AT & SPI Inc. -- Supporter Free Software Foundation Europe
   `-   BOFH excuse #211:  Lightning strikes.

Reply | Threaded
Open this post in threaded view
|

Re: What to do with the "Upstream info" links in the PTS and the packages-metadata branch in collab-qa ?

Charles Plessy-12
Hi Paul, and Stuart,

Le Tue, May 08, 2018 at 09:47:05AM +0800, Paul Wise a écrit :
>
> Is the code for this somewhere? Looks like it got deleted from alioth.

I was a bit over-zealous and cleaned the Alioth repository recently, but
the source is still on <https://umegaya.branchable.com/> and I can
transfer it to Salsa if there is interest.

Regarding the bug that I could not track down, here is the link to the
call for help that I sent in 2014:

https://lists.debian.org/debian-qa/2014/06/msg00022.html

Le Tue, May 08, 2018 at 06:02:36PM +1000, Stuart Prescott a écrit :
>
> Are there any other consumers of these files?

The Debian Med team includes upstream information from these files
in its "tasks" pages (see <https://blends.debian.org/med/tasks/bio>
for example), using the UDD as a gatherer.

Le Tue, May 08, 2018 at 05:22:40PM +0800, Paul Wise a écrit :
>
> The main value is aggregating the information and standardising it,
> within upstream codebases the info could be in DOAP files or READMEs
> or README.foo or HACKING etc. Some things like fundraising might only
> be on the upstream website.

Indeed, it would be great if we could signal to our users where to
gratify the upstream authors, and I have been hoping that the
upstream-metadata could be helpful for this.  But maybe this is more the
scope of AppStream <https://appstream.debian.org/> ?

Have a nice day,

Charles

--
Charles Plessy
Debian Med packaging team,
http://www.debian.org/devel/debian-med
Tsurumi, Kanagawa, Japan

Reply | Threaded
Open this post in threaded view
|

Re: What to do with the "Upstream info" links in the PTS and the packages-metadata branch in collab-qa ?

Paul Wise via nm
On Thu, May 10, 2018 at 1:21 PM, Charles Plessy wrote:

> I was a bit over-zealous and cleaned the Alioth repository recently, but
> the source is still on <https://umegaya.branchable.com/> and I can
> transfer it to Salsa if there is interest.

Branchable is missing a few of the commits from alioth.

> The Debian Med team includes upstream information from these files
> in its "tasks" pages (see <https://blends.debian.org/med/tasks/bio>
> for example), using the UDD as a gatherer.

This is using upstream-metadata.debian.net, seems like it will need
updating if a replacement shows up.

> Indeed, it would be great if we could signal to our users where to
> gratify the upstream authors, and I have been hoping that the
> upstream-metadata could be helpful for this

I filed a bug about that here:

https://bugs.debian.org/777306

> But maybe this is more the scope of AppStream

AppStream includes donation URLs:

https://www.freedesktop.org/software/appstream/docs/chap-Metadata.html#tag-url

I don't think AppStream is aimed at everything, for example libraries
or command-line tools aren't going to have them.

> <https://appstream.debian.org/> ?

This is more developer-oriented than user-oriented, I think we would
need apps.debian.org for a user-oriented view of Debian's AppStream
meta-data.

This would be my proposal (I won't volunteer) for replacing
upstream-metadata.debian.net:

Update vcswatch to scan each repo for upstream metadata files and
export those. Update the UDD gatherer to use the Contents-source files
from the local mirror and extract upstream metadata files. Update the
UDD gatherer to import the vcswatch results too.

--
bye,
pabs

https://wiki.debian.org/PaulWise

Reply | Threaded
Open this post in threaded view
|

Re: What to do with the "Upstream info" links in the PTS and the packages-metadata branch in collab-qa ?

Charles Plessy-12
Le Thu, May 10, 2018 at 01:46:46PM +0800, Paul Wise a écrit :
> On Thu, May 10, 2018 at 1:21 PM, Charles Plessy wrote:
>
> > I was a bit over-zealous and cleaned the Alioth repository recently, but
> > the source is still on <https://umegaya.branchable.com/> and I can
> > transfer it to Salsa if there is interest.
>
> Branchable is missing a few of the commits from alioth.

Good catch.  I do not have push/admin access util I come back home this
evening.  The only missing commit that may matter is:

        commit 2bfe95c1cd14f6e0d69b1dc3ab708730dec015dc
        Author: Charles Plessy <[hidden email]>
        Date:   Wed Jun 4 21:31:09 2014 +0900

            Build-depend on apache2-dev instead of dh-apache2.

        diff --git a/debian/control b/debian/control
        index b22ad25..ecae592 100644
        --- a/debian/control
        +++ b/debian/control
        @@ -3,7 +3,7 @@ Maintainer: Charles Plessy <[hidden email]>
         Section: database
         Priority: optional
         Build-Depends: debhelper (>= 9),
        -               dh-apache2,
        +               apache2-dev,
                        pandoc,
                        perl,
                        perl-doc

Cheers,

--
Charles

Reply | Threaded
Open this post in threaded view
|

Re: What to do with the "Upstream info" links in the PTS and the packages-metadata branch in collab-qa ?

Andreas Tille-5
In reply to this post by Stuart Prescott-11
On Tue, May 08, 2018 at 06:02:36PM +1000, Stuart Prescott wrote:
> Are there any other consumers of these files?

Definitely.
 

> If there are no longer any consumers, should we really be nagging
> maintainers to include them?
>
> $ lintian-info --list-tags | grep upstream-metadata
> upstream-metadata-file-is-missing
> upstream-metadata-is-not-a-file
> upstream-metadata-yaml-invalid
>
> While the last two still make sense (if the file is there, make sure it is
> valid), does the first one?

I agree that the first one is not needed.

> I've never been a fan of lintian suggesting that
> maintainers add a file that for most(?) packages, is not going to contain
> anything that is not duplicated information from elsewhere in the package.
>
> (If there are other users of this information, great...)

The file debian/upstream/metadata is described in wiki[1].  The
definition was cleaned up to not contain redundant information
any more.  Several of its data (Reference, Registry and others)
are read and injected into UDD.

Kind regards

      Andreas.

[1] https://wiki.debian.org/UpstreamMetadata

--
http://fam-tille.de

Reply | Threaded
Open this post in threaded view
|

Re: What to do with the "Upstream info" links in the PTS and the packages-metadata branch in collab-qa ?

Andreas Tille-5
In reply to this post by Paul Wise via nm
Hi Paul,

On Thu, May 10, 2018 at 01:46:46PM +0800, Paul Wise wrote:
>
> > The Debian Med team includes upstream information from these files
> > in its "tasks" pages (see <https://blends.debian.org/med/tasks/bio>
> > for example), using the UDD as a gatherer.
>
> This is using upstream-metadata.debian.net, seems like it will need
> updating if a replacement shows up.

I can not parse this?  The UDD gatherer is not using
upstream-metadata.debian.net.  It is rather reading Git repositories of
selected Blends teams (formerly on Alioth now rewritten for Salsa[1]).
The UDD gatherer consumes more data than only debian/upstream/metadata
but also about prospective packages.  It does not depend from
umegaya nor upstream-metadata.debian.net.
 
> Update vcswatch to scan each repo for upstream metadata files and
> export those.

May be vcswatch and fetch-machine-readable_salsa[1] should be merged.
I'm pretty sure that there is a lot room for enhancement at least for
the latter.

> Update the UDD gatherer to use the Contents-source files
> from the local mirror and extract upstream metadata files. Update the
> UDD gatherer to import the vcswatch results too.

For the Blends purpose Contents-source files are not sufficient since
also Vcs of non-released packages is read and we consume always latest
changes from Git and not from possibly way outdated uploaded packages.

Kind regards

      Andreas.

[1] https://salsa.debian.org/blends-team/website/blob/master/misc/machine_readable/fetch-machine-readable_salsa.py

--
http://fam-tille.de

Reply | Threaded
Open this post in threaded view
|

Re: What to do with the "Upstream info" links in the PTS and the packages-metadata branch in collab-qa ?

Paul Wise via nm
On Wed, May 16, 2018 at 3:01 AM, Andreas Tille wrote:

> On Thu, May 10, 2018 at 01:46:46PM +0800, Paul Wise wrote:
>> This is using upstream-metadata.debian.net, seems like it will need
>> updating if a replacement shows up.
>
> I can not parse this?  The UDD gatherer is not using
> upstream-metadata.debian.net.  It is rather reading Git repositories of
> selected Blends teams (formerly on Alioth now rewritten for Salsa[1]).
> The UDD gatherer consumes more data than only debian/upstream/metadata
> but also about prospective packages.  It does not depend from
> umegaya nor upstream-metadata.debian.net.

I was mislead by udd/upstream_reader.py from UDD saying it uses
upstream-metadata.debian.net, probably that comment needs to be
removed.

> May be vcswatch and fetch-machine-readable_salsa[1] should be merged.
> I'm pretty sure that there is a lot room for enhancement at least for
> the latter.

Possibly, but you mentioned the latter also scans packages not yet in
Debian so the vcswatch maintainer would have to be convinced that
feature is a good idea.

> For the Blends purpose Contents-source files are not sufficient since
> also Vcs of non-released packages is read and we consume always latest
> changes from Git and not from possibly way outdated uploaded packages.

Not every package has a VCS (and some never will) and some of those
have upstream metadata files, so Contents-source scanning is needed
anyway.

--
bye,
pabs

https://wiki.debian.org/PaulWise

Reply | Threaded
Open this post in threaded view
|

Re: What to do with the "Upstream info" links in the PTS and the packages-metadata branch in collab-qa ?

Andreas Tille-5
On Wed, May 16, 2018 at 09:56:11AM +0800, Paul Wise wrote:

> > I can not parse this?  The UDD gatherer is not using
> > upstream-metadata.debian.net.  It is rather reading Git repositories of
> > selected Blends teams (formerly on Alioth now rewritten for Salsa[1]).
> > The UDD gatherer consumes more data than only debian/upstream/metadata
> > but also about prospective packages.  It does not depend from
> > umegaya nor upstream-metadata.debian.net.
>
> I was mislead by udd/upstream_reader.py from UDD saying it uses
> upstream-metadata.debian.net, probably that comment needs to be
> removed.

I've updated the comment in Git[1].  Hope this avoids wrong assumptions
in the future and sorry for being that sloppy with docs. :-(
 
> > May be vcswatch and fetch-machine-readable_salsa[1] should be merged.
> > I'm pretty sure that there is a lot room for enhancement at least for
> > the latter.
>
> Possibly, but you mentioned the latter also scans packages not yet in
> Debian so the vcswatch maintainer would have to be convinced that
> feature is a good idea.

May be we should at least start talking about it.  DebConf 18 could be
a good opportunity for this.
 
> > For the Blends purpose Contents-source files are not sufficient since
> > also Vcs of non-released packages is read and we consume always latest
> > changes from Git and not from possibly way outdated uploaded packages.
>
> Not every package has a VCS (and some never will) and some of those
> have upstream metadata files, so Contents-source scanning is needed
> anyway.

I'm very picky about having those Blends related packages in VCS to have
at least those upstream files where the concept was invented for
(bibliographic references) in VCS.  But you are definitely correct that
there are packages with upstream files outside VCS (and if you ask me
its a shame ;-) ).

Kind regards

     Andreas.

[1] https://salsa.debian.org/qa/udd/commit/a6a9c3bebbba4ce9f08307aeea00b433466f5070 

--
http://fam-tille.de