[UDD] Is there any information about failed autopkgtest in UDD?

classic Classic list List threaded Threaded
26 messages Options
12
Reply | Threaded
Open this post in threaded view
|

[UDD] Is there any information about failed autopkgtest in UDD?

Andreas Tille-5
Hi,

I intend to enhance the Blends framework by a QA page which also should
include information about failed autopkgtests.  Unfortunately I can not
find any information about this in UDD.  While the source table contains
a field testsuite which tells whether the package has a test or not.
But where can I find whether the test succeeds (and may be a link to the
failed log).

If this information is really missing (and I did not just missed it
inside the current UDD) where is the best source to parse metadata to
fetch this information into UDD?  I have written some UDD importers and
could consider writing an importer.

Kind regards

       Andreas.

[1] my personal agenda item one at
    https://salsa.debian.org/med-team/community/2020-covid19-hackathon/-/wikis/Covid-19-hackathon

--
http://fam-tille.de

Reply | Threaded
Open this post in threaded view
|

Re: [UDD] Is there any information about failed autopkgtest in UDD?

Paul Gevers-4
Hi Andreas,

On 01-04-2020 15:23, Andreas Tille wrote:
> If this information is really missing (and I did not just missed it
> inside the current UDD) where is the best source to parse metadata to
> fetch this information into UDD?

It depends what you look for. If you're concerned about failures that
will impact migration, the canonical place is:
https://release.debian.org/britney/excuses.yaml.gz (or the non-zipped
version) or directly (around 20 minutes earlier) on respighi. This yaml
includes the links you are looking for.

If you're more interested in regressions is pure suites (like DDPO is
showing), than the results are available from
https://ci.debian.net/data/status/ e.g.
https://ci.debian.net/data/status/testing/amd64/packages.json for
testing/amd64 The URL needs to be constructed:
https://ci.debian.net/data/autopkgtest/testing/amd64/<package_letter>/<package>/<run_id>/log.gz
where package_letter is the first character for all packages except
packages that start with lib, where <package_letter> are the first four
characters.

Paul


signature.asc (499 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: [UDD] Is there any information about failed autopkgtest in UDD?

Andreas Tille-2
Hi Paul,

On Thu, Apr 02, 2020 at 05:20:54PM +0200, Paul Gevers wrote:
> It depends what you look for. If you're concerned about failures that
> will impact migration, the canonical place is:
> https://release.debian.org/britney/excuses.yaml.gz (or the non-zipped
> version) or directly (around 20 minutes earlier) on respighi. This yaml
> includes the links you are looking for.

I've checked this and found some missings for our purpose.  For example
that file contains some instances of deepnano but only as a rdepends of
theano:

$ grep -B11 -A2 deepnano excuses.yaml | grep -v -e ' - null' -e '[wd]-version' -e '^  m' | sed -e '/^--/,/arm64:/d' -e '/policy_info/,/autopkgtest:/d'
  item-name: theano
      deepnano:
        amd64:
        - RUNNING-ALWAYSFAIL
        - https://ci.debian.net/status/pending
        - https://ci.debian.net/packages/d/deepnano/testing/amd64
        arm64:
        - RUNNING-ALWAYSFAIL
        - https://ci.debian.net/status/pending
        - https://ci.debian.net/packages/d/deepnano/testing/arm64
        - PASS
        - https://ci.debian.net/data/autopkgtest/testing/arm64/d/debconf-kde/4796849/log.gz
        - https://ci.debian.net/packages/d/debconf-kde/testing/arm64
      deepnano:
        amd64:
        - RUNNING-ALWAYSFAIL
        - https://ci.debian.net/status/pending
        - https://ci.debian.net/packages/d/deepnano/testing/amd64
        arm64:
        - RUNNING-ALWAYSFAIL
        - https://ci.debian.net/status/pending
        - https://ci.debian.net/packages/d/deepnano/testing/arm64
        - PASS
        - https://ci.debian.net/data/autopkgtest/testing/arm64/d/dask/4786421/log.gz
        - https://ci.debian.net/packages/d/dask/testing/arm64
      deepnano:
        amd64:
        - RUNNING-ALWAYSFAIL
        - https://ci.debian.net/status/pending
        - https://ci.debian.net/packages/d/deepnano/testing/amd64
        arm64:
        - RUNNING-ALWAYSFAIL
        - https://ci.debian.net/status/pending
        - https://ci.debian.net/packages/d/deepnano/testing/arm64

Please excuse my rough parsing of that yaml file.

Now since I became suspicious about deepnano itself when checking tracker

    https://tracker.debian.org/pkg/deepnano

it does not even have any debci entry despite deepnano has an autopkgtest.
I admit I consider this a bug in tracker.
 
> If you're more interested in regressions is pure suites (like DDPO is
> showing), than the results are available from
> https://ci.debian.net/data/status/ e.g.
> https://ci.debian.net/data/status/testing/amd64/packages.json for

When greping this file for deepnano I get no hit at all.

> testing/amd64 The URL needs to be constructed:
> https://ci.debian.net/data/autopkgtest/testing/amd64/<package_letter>/<package>/<run_id>/log.gz
> where package_letter is the first character for all packages except
> packages that start with lib, where <package_letter> are the first four
> characters.

Hmmm, it seems I really need to parse these URLs

    https://ci.debian.net/packages/d/deepnano/  [1]

for all source packages (in the same way as for deepnano) which seems to
be no straightforward way.  I wonder whether you could drop some easily
parsable file containing

        source architecture pass version version_that_has_passed_before

or something like this - probably I've missed some important field.  This
could be importet into UDD and tracker could base on some UDD table that
imports this information.

Do you think it is feasible to put those data somewhere for easy UDD
import?

OK, when reading [1] it says:

    This package is currently blacklisted and will not have any new test runs.

but why? My goal is to assemble information about all Blends packages
featuring an autopkgtest that fails to give people intending to do some
QA work a good place to look for tasks.  I have no idea how I can easily
get this information in a structured way.

Kind regards

       Andreas.

--
http://fam-tille.de

Reply | Threaded
Open this post in threaded view
|

Re: [UDD] Is there any information about failed autopkgtest in UDD?

Paul Gevers-4
Hi Andreas,

On 05-04-2020 16:34, Andreas Tille wrote:

> Hi Paul,
>
> On Thu, Apr 02, 2020 at 05:20:54PM +0200, Paul Gevers wrote:
>> It depends what you look for. If you're concerned about failures that
>> will impact migration, the canonical place is:
>> https://release.debian.org/britney/excuses.yaml.gz (or the non-zipped
>> version) or directly (around 20 minutes earlier) on respighi. This yaml
>> includes the links you are looking for.
>
> I've checked this and found some missings for our purpose.
>
> For example
> that file contains some instances of deepnano but only as a rdepends of
> theano:
deepnano is indeed a bad example as it is blacklisted in all suites and
all architectures: https://ci.debian.net/status/blacklist/ because of
https://bugs.debian.org/921566 (parse-able source:
https://salsa.debian.org/ci-team/debian-ci-config/-/blob/master/cookbooks/debci/files/default/blacklist)

> $ grep -B11 -A2 deepnano excuses.yaml | grep -v -e ' - null' -e '[wd]-version' -e '^  m' | sed -e '/^--/,/arm64:/d' -e '/policy_info/,/autopkgtest:/d'
>   item-name: theano
>       deepnano:
>         amd64:
>         - RUNNING-ALWAYSFAIL
>         - https://ci.debian.net/status/pending
>         - https://ci.debian.net/packages/d/deepnano/testing/amd64
>         arm64:
>         - RUNNING-ALWAYSFAIL
>         - https://ci.debian.net/status/pending
>         - https://ci.debian.net/packages/d/deepnano/testing/arm64
>         - PASS
>         - https://ci.debian.net/data/autopkgtest/testing/arm64/d/debconf-kde/4796849/log.gz
>         - https://ci.debian.net/packages/d/debconf-kde/testing/arm64
>       deepnano:
>         amd64:
>         - RUNNING-ALWAYSFAIL
>         - https://ci.debian.net/status/pending
>         - https://ci.debian.net/packages/d/deepnano/testing/amd64
>         arm64:
>         - RUNNING-ALWAYSFAIL
>         - https://ci.debian.net/status/pending
>         - https://ci.debian.net/packages/d/deepnano/testing/arm64
>         - PASS
>         - https://ci.debian.net/data/autopkgtest/testing/arm64/d/dask/4786421/log.gz
>         - https://ci.debian.net/packages/d/dask/testing/arm64
>       deepnano:
>         amd64:
>         - RUNNING-ALWAYSFAIL
>         - https://ci.debian.net/status/pending
>         - https://ci.debian.net/packages/d/deepnano/testing/amd64
>         arm64:
>         - RUNNING-ALWAYSFAIL
>         - https://ci.debian.net/status/pending
>         - https://ci.debian.net/packages/d/deepnano/testing/arm64
>
> Please excuse my rough parsing of that yaml file.
>
> Now since I became suspicious about deepnano itself when checking tracker
>
>     https://tracker.debian.org/pkg/deepnano
>
> it does not even have any debci entry despite deepnano has an autopkgtest.
> I admit I consider this a bug in tracker.
I don't think so because it is blacklisted.

>> If you're more interested in regressions is pure suites (like DDPO is
>> showing), than the results are available from
>> https://ci.debian.net/data/status/ e.g.
>> https://ci.debian.net/data/status/testing/amd64/packages.json for
>
> When greping this file for deepnano I get no hit at all.

Because it is blacklisted.

>> testing/amd64 The URL needs to be constructed:
>> https://ci.debian.net/data/autopkgtest/testing/amd64/<package_letter>/<package>/<run_id>/log.gz
>> where package_letter is the first character for all packages except
>> packages that start with lib, where <package_letter> are the first four
>> characters.
>
> Hmmm, it seems I really need to parse these URLs
>
>     https://ci.debian.net/packages/d/deepnano/  [1]
>
> for all source packages (in the same way as for deepnano) which seems to
> be no straightforward way.  I wonder whether you could drop some easily
> parsable file containing
>
> source architecture pass version version_that_has_passed_before
You're missing the suite here. So, we already have that info, you just
need to loop over suite/arch. It's really in the json I already
mentioned: packages.json. Please use that.

> or something like this - probably I've missed some important field.  This
> could be importet into UDD and tracker could base on some UDD table that
> imports this information.

If you're interested in specific packages, there is the RSS feed (let's
pick a package that's not blacklisted):
https://ci.debian.net/data/feeds/c/cacti.xml

> OK, when reading [1] it says:
>
>     This package is currently blacklisted and will not have any new test runs.
>
> but why?

Follow the link under the word "blacklisted" and you would have found
the answer I gave above.

Paul


signature.asc (499 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: [UDD] Is there any information about failed autopkgtest in UDD?

Andreas Tille-2
Hi Paul,

On Sun, Apr 05, 2020 at 09:20:21PM +0200, Paul Gevers wrote:

> > For example
> > that file contains some instances of deepnano but only as a rdepends of
> > theano:
>
> deepnano is indeed a bad example as it is blacklisted in all suites and
> all architectures: https://ci.debian.net/status/blacklist/ because of
> https://bugs.debian.org/921566 (parse-able source:
> https://salsa.debian.org/ci-team/debian-ci-config/-/blob/master/cookbooks/debci/files/default/blacklist)
> ...
> I don't think so because it is blacklisted.
>
> > for all source packages (in the same way as for deepnano) which seems to
> > be no straightforward way.  I wonder whether you could drop some easily
> > parsable file containing
> >
> > source architecture pass version version_that_has_passed_before
>
> You're missing the suite here. So, we already have that info, you just
> need to loop over suite/arch. It's really in the json I already
> mentioned: packages.json. Please use that.

OK, I'm using that.  Its way more close to what I need.  I simply
started with your first hint. :-)

Do you see any chance to also mention blacklisted packages in
packages.json?
 
The json file contains the following values:

        'run_id',
        'created_at',
        'updated_at',
        'suite',
        'arch',
        'package',    ## that field will be named 'source' in UDD!
        'version',
        'trigger',
        'status',
        'requestor',
        'pin_packages',
        'worker',
        'date',
        'duration_seconds',
        'last_pass_date',
        'last_pass_version',
        'message',
        'previous_status',
        'duration_human',
        'blame',

I would consider to simply take over all these values as columns.
Is there any documentation for these fields?

I would parse these fields from all files I get via


#!/bin/sh

RELEASES="stable
          testing
          unstable
         "

ARCHS="amd64
       arm64
       ppc64el
      "

for release in $RELEASES ; do
    for arch in $ARCHS ; do
       wget https://ci.debian.net/data/status/${release}/${arch}/packages.json -O packages_${release}_${arch}.json
    done
done


Would you consider this a sensible approach for an UDD gatherer?
May be you consider some fields as really restricted to some
special applications and nobody would ever consider querying
UDD for it?

Kind regards

    Andreas.

--
http://fam-tille.de

Reply | Threaded
Open this post in threaded view
|

Re: [UDD] Is there any information about failed autopkgtest in UDD?

Paul Gevers-4
Hi Andreas,

On 08-04-2020 22:17, Andreas Tille wrote:

>>> for all source packages (in the same way as for deepnano) which seems to
>>> be no straightforward way.  I wonder whether you could drop some easily
>>> parsable file containing
>>>
>>> source architecture pass version version_that_has_passed_before
>>
>> You're missing the suite here. So, we already have that info, you just
>> need to loop over suite/arch. It's really in the json I already
>> mentioned: packages.json. Please use that.
>
> OK, I'm using that.  Its way more close to what I need.  I simply
> started with your first hint. :-)
Ok.

> Do you see any chance to also mention blacklisted packages in
> packages.json?

Antonio, what do you think? If we expose the blacklist, I can also have
britney consume it.

> The json file contains the following values:
>
> 'run_id',
> 'created_at',
> 'updated_at',
> 'suite',
> 'arch',
> 'package',    ## that field will be named 'source' in UDD!
> 'version',
> 'trigger',
> 'status',
> 'requestor',
> 'pin_packages',
> 'worker',
> 'date',
> 'duration_seconds',
> 'last_pass_date',
> 'last_pass_version',
> 'message',
> 'previous_status',
> 'duration_human',
> 'blame',
>
> I would consider to simply take over all these values as columns.
> Is there any documentation for these fields?
Yes, but not entirely in one place (at least I couldn't easily spot it):
https://ci.debian.net/doc/Debci/

> Would you consider this a sensible approach for an UDD gatherer?

Yes.

> May be you consider some fields as really restricted to some
> special applications and nobody would ever consider querying
> UDD for it?

Yes, I wouldn't add blame (I don't think we add those anymore, it's
legacy, I only found two occurrences on amd64), created_at and
updated_at. I also don't think that worker and message are very useful,
but one never knows. duration_seconds and duration_human feel double and
the former is leaner for in a database.

Paul


signature.asc (499 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: [UDD] Is there any information about failed autopkgtest in UDD?

Antonio Terceiro-3
On Thu, Apr 09, 2020 at 10:44:18AM +0200, Paul Gevers wrote:
> Hi Andreas,
>
> On 08-04-2020 22:17, Andreas Tille wrote:
> > Do you see any chance to also mention blacklisted packages in
> > packages.json?
>
> Antonio, what do you think? If we expose the blacklist, I can also have
> britney consume it.

I think it could be exposed, yes. I'm currrently working on optimizing
the generation of all of that data, so I added to my TODO list and will
sneak that in.

signature.asc (849 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: [UDD] Is there any information about failed autopkgtest in UDD?

Andreas Tille-5
Hi Antonio,

greetings to Curitiba. ;-)
Hope you are fine!

On Thu, Apr 09, 2020 at 09:37:41AM -0300, Antonio Terceiro wrote:
> On Thu, Apr 09, 2020 at 10:44:18AM +0200, Paul Gevers wrote:
> > Antonio, what do you think? If we expose the blacklist, I can also have
> > britney consume it.
>
> I think it could be exposed, yes. I'm currrently working on optimizing
> the generation of all of that data, so I added to my TODO list and will
> sneak that in.

Do you expect new important fields in the json file or do you intend to
change its structure in principle?  It would be stupid for me to write a
gatherer for UDD and once finished I learn that basic things might have
changed.

Kind regards

     Andreas.

--
http://fam-tille.de

Reply | Threaded
Open this post in threaded view
|

Re: [UDD] Is there any information about failed autopkgtest in UDD?

Antonio Terceiro-3
On Thu, Apr 09, 2020 at 04:55:30PM +0200, Andreas Tille wrote:

> Hi Antonio,
>
> greetings to Curitiba. ;-)
> Hope you are fine!
>
> On Thu, Apr 09, 2020 at 09:37:41AM -0300, Antonio Terceiro wrote:
> > On Thu, Apr 09, 2020 at 10:44:18AM +0200, Paul Gevers wrote:
> > > Antonio, what do you think? If we expose the blacklist, I can also have
> > > britney consume it.
> >
> > I think it could be exposed, yes. I'm currrently working on optimizing
> > the generation of all of that data, so I added to my TODO list and will
> > sneak that in.
>
> Do you expect new important fields in the json file or do you intend to
> change its structure in principle?
No, the format will not change.

FWIW I concur with what Paul said regarding which fields from
packages.json you should not consider.

signature.asc (849 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: [UDD] Is there any information about failed autopkgtest in UDD?

Andreas Tille-2
In reply to this post by Paul Gevers-4
Hi Paul,

On Thu, Apr 09, 2020 at 10:44:18AM +0200, Paul Gevers wrote:

>
> > May be you consider some fields as really restricted to some
> > special applications and nobody would ever consider querying
> > UDD for it?
>
> Yes, I wouldn't add blame (I don't think we add those anymore, it's
> legacy, I only found two occurrences on amd64), created_at and
> updated_at. I also don't think that worker and message are very useful,
> but one never knows. duration_seconds and duration_human feel double and
> the former is leaner for in a database.

So I probably go with the following keys (in Python syntax):

valid_keys = ( 'run_id',
    #           'created_at',           # Paul Gevers: should be ignored
    #           'updated_at',           # Paul Gevers: should be ignored
               'suite',
               'arch',
               'package', # ----> should be renamed to 'source'
               'version',
               'trigger',               # usually package.*version
               'status',
               'requestor',             # 'britney', 'debci' or e-mail
               'pin_packages',          # []
    #           'worker',               # Paul Gevers: should be ignored (is 'null' anyway)
               'date',
               'duration_seconds',
               'last_pass_date',
               'last_pass_version',
               'message',               # see below
               'previous_status',
    #           'duration_human',       # Paul Gevers: duration_seconds and duration_human feel double and the former is leaner for in a database
    #           'blame',                # Paul Gevers: should be ignored
             )

# message can be
#  $ grep '"message"' packages*.json | sed 's/^.*\.json: *//'  | sort | uniq
#  "message": "All tests passed"                                        -> "status": "pass"
#  "message": "Could not run tests due to a temporary testbed failure"  -> "status": "tmpfail"
#  "message": "elbrus"                                                  -> "status": "tmpfail"
#  "message": "Erroneous package"                                       -> "status": "fail"
#  "message": null                                                      -> "status": "fail"
#  "message": "No tests in this package or all skipped"                 -> "status": "neutral"
#  "message": "Tests failed",                                           -> "status": "fail"
#  "message": "Tests failed, and at least one test skipped"             -> "status": "fail"
#  "message": "Tests passed, but at least one test skipped"             -> "status": "pass"
#  "message": "Unexpected autopkgtest exit code 20"                     -> "status": "tmpfail"

I agree that leaving out worker which is really always null makes sense
but I tend to leave message since leaving this out looks like loosing
information.  I tried to find a relation to status but it seems the same
status can result in different messages.  I think just a field in addition
will not blow up UDD way more than it recently is - may be I consider
a normalised form, but usually UDD is not very normalised at all.

I wonder what might be the meaning of pin_packages which is always
equal [].

Kind regards

     Andreas.

Reply | Threaded
Open this post in threaded view
|

Re: [UDD] Is there any information about failed autopkgtest in UDD?

Paul Gevers-4
Hi Andreas,

On 09-04-2020 22:53, Andreas Tille wrote:
> valid_keys = ( 'run_id',
>     #           'created_at',           # Paul Gevers: should be ignored
>     #           'updated_at',           # Paul Gevers: should be ignored
>                'suite',
>                'arch',
>                'package', # ----> should be renamed to 'source'
>                'version',
>                'trigger',               # usually package.*version
I expected you to mostly see "" or "migration-reference/0" here, with
some hand crafted text from random DD's.

>                'status',
>                'requestor',             # 'britney', 'debci' or e-mail

Debian login to be precise, not e-mail.

>                'pin_packages',          # []

Since a couple of months this json and other pages only show "pure"
suite runs, to pin_packages is always empty. pin_packages contains which
packages are taken from another suite than the base suite.

>     #           'worker',               # Paul Gevers: should be ignored (is 'null' anyway)

Oh, bug somewhere I guess.

>                'date',
>                'duration_seconds',
>                'last_pass_date',
>                'last_pass_version',
>                'message',               # see below
>                'previous_status',
>     #           'duration_human',       # Paul Gevers: duration_seconds and duration_human feel double and the former is leaner for in a database
>     #           'blame',                # Paul Gevers: should be ignored
>              )
>
> # message can be
> #  $ grep '"message"' packages*.json | sed 's/^.*\.json: *//'  | sort | uniq
> #  "message": "All tests passed"                                        -> "status": "pass"
> #  "message": "Could not run tests due to a temporary testbed failure"  -> "status": "tmpfail"
> #  "message": "elbrus"                                                  -> "status": "tmpfail"
> #  "message": "Erroneous package"                                       -> "status": "fail"
> #  "message": null                                                      -> "status": "fail"
> #  "message": "No tests in this package or all skipped"                 -> "status": "neutral"
> #  "message": "Tests failed",                                           -> "status": "fail"
> #  "message": "Tests failed, and at least one test skipped"             -> "status": "fail"
> #  "message": "Tests passed, but at least one test skipped"             -> "status": "pass"
> #  "message": "Unexpected autopkgtest exit code 20"                     -> "status": "tmpfail"
>
> I agree that leaving out worker which is really always null makes sense
> but I tend to leave message since leaving this out looks like loosing
> information.  I tried to find a relation to status but it seems the same
> status can result in different messages.  I think just a field in addition
> will not blow up UDD way more than it recently is - may be I consider
> a normalised form, but usually UDD is not very normalised at all.
In general this is the final output from autopkgtest. But, as you see my
name there, I had to clean up several times and to be able to find those
back, I added my nick to the message. The list thus may change over time.

Paul


signature.asc (499 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: [UDD] Is there any information about failed autopkgtest in UDD?

Andreas Tille-2
Hi Paul,

thanks for the clarification.  This commit

   https://salsa.debian.org/qa/udd/-/commit/6a874a89365671dd37a14a9bca25290dc55a1fc9

imports the current data.  I will tests this a bit more and than activate
in a cron job as importer.

Thanks a lot for your contribution

      Andreas.

On Fri, Apr 10, 2020 at 09:05:31PM +0200, Paul Gevers wrote:

> Hi Andreas,
>
> On 09-04-2020 22:53, Andreas Tille wrote:
> > valid_keys = ( 'run_id',
> >     #           'created_at',           # Paul Gevers: should be ignored
> >     #           'updated_at',           # Paul Gevers: should be ignored
> >                'suite',
> >                'arch',
> >                'package', # ----> should be renamed to 'source'
> >                'version',
> >                'trigger',               # usually package.*version
> I expected you to mostly see "" or "migration-reference/0" here, with
> some hand crafted text from random DD's.
>
> >                'status',
> >                'requestor',             # 'britney', 'debci' or e-mail
>
> Debian login to be precise, not e-mail.
>
> >                'pin_packages',          # []
>
> Since a couple of months this json and other pages only show "pure"
> suite runs, to pin_packages is always empty. pin_packages contains which
> packages are taken from another suite than the base suite.
>
> >     #           'worker',               # Paul Gevers: should be ignored (is 'null' anyway)
>
> Oh, bug somewhere I guess.
>
> >                'date',
> >                'duration_seconds',
> >                'last_pass_date',
> >                'last_pass_version',
> >                'message',               # see below
> >                'previous_status',
> >     #           'duration_human',       # Paul Gevers: duration_seconds and duration_human feel double and the former is leaner for in a database
> >     #           'blame',                # Paul Gevers: should be ignored
> >              )
> >
> > # message can be
> > #  $ grep '"message"' packages*.json | sed 's/^.*\.json: *//'  | sort | uniq
> > #  "message": "All tests passed"                                        -> "status": "pass"
> > #  "message": "Could not run tests due to a temporary testbed failure"  -> "status": "tmpfail"
> > #  "message": "elbrus"                                                  -> "status": "tmpfail"
> > #  "message": "Erroneous package"                                       -> "status": "fail"
> > #  "message": null                                                      -> "status": "fail"
> > #  "message": "No tests in this package or all skipped"                 -> "status": "neutral"
> > #  "message": "Tests failed",                                           -> "status": "fail"
> > #  "message": "Tests failed, and at least one test skipped"             -> "status": "fail"
> > #  "message": "Tests passed, but at least one test skipped"             -> "status": "pass"
> > #  "message": "Unexpected autopkgtest exit code 20"                     -> "status": "tmpfail"
> >
> > I agree that leaving out worker which is really always null makes sense
> > but I tend to leave message since leaving this out looks like loosing
> > information.  I tried to find a relation to status but it seems the same
> > status can result in different messages.  I think just a field in addition
> > will not blow up UDD way more than it recently is - may be I consider
> > a normalised form, but usually UDD is not very normalised at all.
>
> In general this is the final output from autopkgtest. But, as you see my
> name there, I had to clean up several times and to be able to find those
> back, I added my nick to the message. The list thus may change over time.
>
> Paul
>




--
http://fam-tille.de

Reply | Threaded
Open this post in threaded view
|

Re: [UDD] Is there any information about failed autopkgtest in UDD?

Lucas Nussbaum-4
Hi Andreas,

I'm sorry if I haven't paid enough attention. But what is the difference
with the 'ci' importer?

https://salsa.debian.org/qa/udd/-/blob/master/rimporters/ci.rb

Thanks

Lucas



On 11/04/20 at 07:12 +0200, Andreas Tille wrote:

> Hi Paul,
>
> thanks for the clarification.  This commit
>
>    https://salsa.debian.org/qa/udd/-/commit/6a874a89365671dd37a14a9bca25290dc55a1fc9
>
> imports the current data.  I will tests this a bit more and than activate
> in a cron job as importer.
>
> Thanks a lot for your contribution
>
>       Andreas.
>
> On Fri, Apr 10, 2020 at 09:05:31PM +0200, Paul Gevers wrote:
> > Hi Andreas,
> >
> > On 09-04-2020 22:53, Andreas Tille wrote:
> > > valid_keys = ( 'run_id',
> > >     #           'created_at',           # Paul Gevers: should be ignored
> > >     #           'updated_at',           # Paul Gevers: should be ignored
> > >                'suite',
> > >                'arch',
> > >                'package', # ----> should be renamed to 'source'
> > >                'version',
> > >                'trigger',               # usually package.*version
> > I expected you to mostly see "" or "migration-reference/0" here, with
> > some hand crafted text from random DD's.
> >
> > >                'status',
> > >                'requestor',             # 'britney', 'debci' or e-mail
> >
> > Debian login to be precise, not e-mail.
> >
> > >                'pin_packages',          # []
> >
> > Since a couple of months this json and other pages only show "pure"
> > suite runs, to pin_packages is always empty. pin_packages contains which
> > packages are taken from another suite than the base suite.
> >
> > >     #           'worker',               # Paul Gevers: should be ignored (is 'null' anyway)
> >
> > Oh, bug somewhere I guess.
> >
> > >                'date',
> > >                'duration_seconds',
> > >                'last_pass_date',
> > >                'last_pass_version',
> > >                'message',               # see below
> > >                'previous_status',
> > >     #           'duration_human',       # Paul Gevers: duration_seconds and duration_human feel double and the former is leaner for in a database
> > >     #           'blame',                # Paul Gevers: should be ignored
> > >              )
> > >
> > > # message can be
> > > #  $ grep '"message"' packages*.json | sed 's/^.*\.json: *//'  | sort | uniq
> > > #  "message": "All tests passed"                                        -> "status": "pass"
> > > #  "message": "Could not run tests due to a temporary testbed failure"  -> "status": "tmpfail"
> > > #  "message": "elbrus"                                                  -> "status": "tmpfail"
> > > #  "message": "Erroneous package"                                       -> "status": "fail"
> > > #  "message": null                                                      -> "status": "fail"
> > > #  "message": "No tests in this package or all skipped"                 -> "status": "neutral"
> > > #  "message": "Tests failed",                                           -> "status": "fail"
> > > #  "message": "Tests failed, and at least one test skipped"             -> "status": "fail"
> > > #  "message": "Tests passed, but at least one test skipped"             -> "status": "pass"
> > > #  "message": "Unexpected autopkgtest exit code 20"                     -> "status": "tmpfail"
> > >
> > > I agree that leaving out worker which is really always null makes sense
> > > but I tend to leave message since leaving this out looks like loosing
> > > information.  I tried to find a relation to status but it seems the same
> > > status can result in different messages.  I think just a field in addition
> > > will not blow up UDD way more than it recently is - may be I consider
> > > a normalised form, but usually UDD is not very normalised at all.
> >
> > In general this is the final output from autopkgtest. But, as you see my
> > name there, I had to clean up several times and to be able to find those
> > back, I added my nick to the message. The list thus may change over time.
> >
> > Paul
> >
>
>
>
>
> --
> http://fam-tille.de
>
>

Reply | Threaded
Open this post in threaded view
|

Re: [UDD] Is there any information about failed autopkgtest in UDD?

Andreas Tille-2
Hi Lucas,

On Mon, Apr 13, 2020 at 10:37:57PM +0200, Lucas Nussbaum wrote:
> I'm sorry if I haven't paid enough attention. But what is the difference
> with the 'ci' importer?
>
> https://salsa.debian.org/qa/udd/-/blob/master/rimporters/ci.rb

I think the problem is that UDD is not documented and I simply did not
know about the ci table. :-(

However, when looking at it there is a difference:

udd=# select status, arch, count(*) from ci group by status, arch order by status, arch;
 status  | arch  | count
---------+-------+-------
 fail    | amd64 |   925
 neutral | amd64 |  1593
 pass    | amd64 | 10805
 tmpfail | amd64 |    14
(4 Zeilen)


udd=# select status, architecture, count(*) from autopkgtest group by status, architecture order by status, architecture;
 status  | architecture | count
---------+--------------+-------
 fail    | amd64        |  1561
 fail    | arm64        |  1471
 fail    | ppc64el      |   711
 neutral | amd64        |  2879
 neutral | arm64        |  1322
 neutral | ppc64el      |   298
 pass    | amd64        | 21373
 pass    | arm64        | 11114
 pass    | ppc64el      |  2458
 tmpfail | amd64        |    11
 tmpfail | arm64        |    85
 tmpfail | ppc64el      |     7
(12 Zeilen)

I guess we should merge both and make sure that all data is imported.
I would never have written a new importer if I would have been aware
of the existing one - but I do not speak Ruby to fix the existing one.

Kind regards
      Andreas.

> On 11/04/20 at 07:12 +0200, Andreas Tille wrote:
> > Hi Paul,
> >
> > thanks for the clarification.  This commit
> >
> >    https://salsa.debian.org/qa/udd/-/commit/6a874a89365671dd37a14a9bca25290dc55a1fc9
> >
> > imports the current data.  I will tests this a bit more and than activate
> > in a cron job as importer.
> >
> > Thanks a lot for your contribution
> >
> >       Andreas.
> >
> > On Fri, Apr 10, 2020 at 09:05:31PM +0200, Paul Gevers wrote:
> > > Hi Andreas,
> > >
> > > On 09-04-2020 22:53, Andreas Tille wrote:
> > > > valid_keys = ( 'run_id',
> > > >     #           'created_at',           # Paul Gevers: should be ignored
> > > >     #           'updated_at',           # Paul Gevers: should be ignored
> > > >                'suite',
> > > >                'arch',
> > > >                'package', # ----> should be renamed to 'source'
> > > >                'version',
> > > >                'trigger',               # usually package.*version
> > > I expected you to mostly see "" or "migration-reference/0" here, with
> > > some hand crafted text from random DD's.
> > >
> > > >                'status',
> > > >                'requestor',             # 'britney', 'debci' or e-mail
> > >
> > > Debian login to be precise, not e-mail.
> > >
> > > >                'pin_packages',          # []
> > >
> > > Since a couple of months this json and other pages only show "pure"
> > > suite runs, to pin_packages is always empty. pin_packages contains which
> > > packages are taken from another suite than the base suite.
> > >
> > > >     #           'worker',               # Paul Gevers: should be ignored (is 'null' anyway)
> > >
> > > Oh, bug somewhere I guess.
> > >
> > > >                'date',
> > > >                'duration_seconds',
> > > >                'last_pass_date',
> > > >                'last_pass_version',
> > > >                'message',               # see below
> > > >                'previous_status',
> > > >     #           'duration_human',       # Paul Gevers: duration_seconds and duration_human feel double and the former is leaner for in a database
> > > >     #           'blame',                # Paul Gevers: should be ignored
> > > >              )
> > > >
> > > > # message can be
> > > > #  $ grep '"message"' packages*.json | sed 's/^.*\.json: *//'  | sort | uniq
> > > > #  "message": "All tests passed"                                        -> "status": "pass"
> > > > #  "message": "Could not run tests due to a temporary testbed failure"  -> "status": "tmpfail"
> > > > #  "message": "elbrus"                                                  -> "status": "tmpfail"
> > > > #  "message": "Erroneous package"                                       -> "status": "fail"
> > > > #  "message": null                                                      -> "status": "fail"
> > > > #  "message": "No tests in this package or all skipped"                 -> "status": "neutral"
> > > > #  "message": "Tests failed",                                           -> "status": "fail"
> > > > #  "message": "Tests failed, and at least one test skipped"             -> "status": "fail"
> > > > #  "message": "Tests passed, but at least one test skipped"             -> "status": "pass"
> > > > #  "message": "Unexpected autopkgtest exit code 20"                     -> "status": "tmpfail"
> > > >
> > > > I agree that leaving out worker which is really always null makes sense
> > > > but I tend to leave message since leaving this out looks like loosing
> > > > information.  I tried to find a relation to status but it seems the same
> > > > status can result in different messages.  I think just a field in addition
> > > > will not blow up UDD way more than it recently is - may be I consider
> > > > a normalised form, but usually UDD is not very normalised at all.
> > >
> > > In general this is the final output from autopkgtest. But, as you see my
> > > name there, I had to clean up several times and to be able to find those
> > > back, I added my nick to the message. The list thus may change over time.
> > >
> > > Paul
> > >
> >
> >
> >
> >
> > --
> > http://fam-tille.de
> >
> >
>

--
http://fam-tille.de

Reply | Threaded
Open this post in threaded view
|

Re: [UDD] Is there any information about failed autopkgtest in UDD?

Andreas Tille-5
In reply to this post by Antonio Terceiro-3
Hi,

I think I've found some inconsistency of the autopkgtest data:
packages_unstable_amd64.json contains:

{
  "run_id": 4963349,
  "created_at": "2020-04-13 12:01:58 UTC",
  "updated_at": "2020-04-14 20:25:31 UTC",
  "suite": "unstable",
  "arch": "amd64",
  "package": "dnapi",
  "version": "1.1-1",
  "trigger": null,
  "status": "neutral",
  "requestor": "debci",
  "pin_packages": [

  ],
  "worker": null,
  "date": "2020-04-14 19:48:49 UTC",
  "duration_seconds": 32,
  "last_pass_date": null,
  "last_pass_version": "",
  "message": "No tests in this package or all skipped",
  "previous_status": "neutral",
  "duration_human": "32s"
}


But there is a test available:


udd=# SELECT source, release, testsuite FROM sources WHERE source = 'dnapi' ;
 source | release |       testsuite        
--------+---------+------------------------
 dnapi  | sid     | autopkgtest-pkg-python


and it seems to be run:

   https://ci.debian.net/data/autopkgtest/unstable/amd64/d/dnapi/4963349/log.gz

says:

autopkgtest [19:48:46]: test autodep8-python3: set -e ; for py in $(py3versions -r 2>/dev/null) ; do cd "$AUTOPKGTEST_TMP" ; echo "Testing with $py:" ; $py -c "import dnapilib; print(dnapilib)" ; done
autopkgtest [19:48:46]: test autodep8-python3: [-----------------------
Testing with python3.8:
<module 'dnapilib' from '/usr/lib/python3/dist-packages/dnapilib/__init__.py'>
autopkgtest [19:48:47]: test autodep8-python3: -----------------------]
autopkgtest [19:48:47]: test autodep8-python3:  - - - - - - - - - - results - - - - - - - - - -
autodep8-python3     PASS (superficial)
autopkgtest [19:48:47]: @@@@@@@@@@@@@@@@@@@@ summary
autodep8-python3     PASS (superficial)


So shouldn't this be rather "pass" than "neutral"?

Kind regards

      Andreas.

--
http://fam-tille.de

Reply | Threaded
Open this post in threaded view
|

Re: [UDD] Is there any information about failed autopkgtest in UDD?

Paul Gevers-4
Hi Andreas,

On 15-04-2020 22:45, Andreas Tille wrote:
> autodep8-python3     PASS (superficial)

superficial is translated to neutral. As is FAIL (flaky).

Paul


signature.asc (499 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: [UDD] Is there any information about failed autopkgtest in UDD?

Andreas Tille-2
On Thu, Apr 16, 2020 at 09:32:43AM +0200, Paul Gevers wrote:
> Hi Andreas,
>
> On 15-04-2020 22:45, Andreas Tille wrote:
> > autodep8-python3     PASS (superficial)
>
> superficial is translated to neutral. As is FAIL (flaky).

Hmmmm, what exactly means "superficial".  Are all those

   Testsuite: autopkgtest-pkg-*

superficial?  Do they qualify for early testing migration or not?
Wouldn't it be more informative to have a fourth category

   pass
   superficial
   neutral
   fail

My intention was to have a list with packages of our team where either a
test is missing or failing.  My idea was that some autopkgtest-pkg-* is
"test is not missing".  What is your opinion as debian-ci team about my
idea?

Kind regards

     Andreas.

--
http://fam-tille.de

Reply | Threaded
Open this post in threaded view
|

Re: [UDD] Is there any information about failed autopkgtest in UDD?

Paul Gevers-4
Hi Andreas,

On 16-04-2020 10:09, Andreas Tille wrote:
>> superficial is translated to neutral. As is FAIL (flaky).
>
> Hmmmm, what exactly means "superficial".

Please read the documentation:
https://salsa.debian.org/ci-team/autopkgtest/raw/master/doc/README.package-tests.rst

> Are all those
>    Testsuite: autopkgtest-pkg-*
>
> superficial?

No. Only those that only have superficial tests. E.g. ruby runs the full
upstream test-suite automatically.

> Do they qualify for early testing migration or not?

Superficial tests *are* neutral, so no.

> Wouldn't it be more informative to have a fourth category
>
>    pass
>    superficial
>    neutral
>    fail

No, because there are only three states. I.e. superficial and flaky and
skipped tests all end up meaning the same.

> My intention was to have a list with packages of our team where either a
> test is missing or failing.  My idea was that some autopkgtest-pkg-* is
> "test is not missing".  What is your opinion as debian-ci team about my
> idea?

It seems you want to be processing the message field then, but honestly
if there is an entry, "test is not missing". If there is no entry, "test
is missing". Failing is "fail". flaky tests are also "test is not
missing", skipped tests are also "test is not missing". Why would you
need all those states? You have the "message" field in your UDD schema
to check why you get the neutral state.

Paul


signature.asc (499 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: [UDD] Is there any information about failed autopkgtest in UDD?

Simon McVittie-7
In reply to this post by Andreas Tille-2
On Thu, 16 Apr 2020 at 10:09:36 +0200, Andreas Tille wrote:
> Hmmmm, what exactly means "superficial".

Originally proposed in:
https://salsa.debian.org/ci-team/autopkgtest/-/merge_requests/19

Implemented in:
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=904979

The design was originally called "trivial" but was renamed to
"superficial" during implementation.

> Are all those
>
>    Testsuite: autopkgtest-pkg-*
>
> superficial?

Most of them yes, because if they fail, that's Very Bad, but if they
succeed, they don't give us a whole lot of confidence that the package
works as intended.

For example, if you have a package python3-dbus, the most that can be
done with knowledge of Python but no knowledge of dbus is to "import dbus"
and see what happens. If that fails, obviously the dbus module is
unusable (or missing a dependency or something); but if that succeeds,
that fact doesn't tell us whether it can actually do D-Bus successfully,
which is its real purpose. If I replaced all the code in python3-dbus
with print("hello"), obviously it wouldn't be implementing its intended
API any more, but it would still pass the "import" test. That's why the
"import" test is considered to be superficial.

If a family of packages have a convention for how to discover and
run a real test suite with non-trivial coverage (for example GNOME-style
installed-tests in /usr/share/installed-tests/**/*.test), then an
autopkgtest-* helper that detected and used *that* convention would usually
*not* be superficial.

> Do they qualify for early testing migration or not?

If *all* the tests are superficial, passing them doesn't speed up testing
migration.

Packages get faster testing migration if they have at least one test that
results in a pass status (passing and not superficial), *and* all of their
tests result in either a pass or neutral status.

Normal tests: pass -> pass, fail -> fail

"superficial" tests: pass -> neutral, fail -> fail
(the test passing doesn't really give us much confidence that it works)

"flaky" tests: pass -> pass, fail -> neutral
(the fail doesn't give us much confidence that it *doesn't* work, because
maybe it just failed randomly)

"flaky, superficial" tests: pass -> neutral, fail -> neutral
(no machine-readable effect, should normally only be used when monitoring
whether a new test is reliable on Debian infrastructure or not)

"skippable" tests: pass -> as above, exit 77 -> neutral, fail -> as above

Tests that can't be run due to restrictions: neutral

    smcv

Reply | Threaded
Open this post in threaded view
|

Re: [UDD] Is there any information about failed autopkgtest in UDD?

Andreas Tille-2
In reply to this post by Paul Gevers-4
Hi Paul,

On Thu, Apr 16, 2020 at 10:30:17AM +0200, Paul Gevers wrote:
> > Hmmmm, what exactly means "superficial".
>
> Please read the documentation:
> https://salsa.debian.org/ci-team/autopkgtest/raw/master/doc/README.package-tests.rst

Thanks for this pointer.
 
> > Are all those
> >    Testsuite: autopkgtest-pkg-*
> >
> > superficial?
>
> No. Only those that only have superficial tests. E.g. ruby runs the full
> upstream test-suite automatically.

Ahhh!

> > Do they qualify for early testing migration or not?
>
> Superficial tests *are* neutral, so no.
>
> > Wouldn't it be more informative to have a fourth category
> >
> >    pass
> >    superficial
> >    neutral
> >    fail
>
> No, because there are only three states. I.e. superficial and flaky and
> skipped tests all end up meaning the same.
>
> > My intention was to have a list with packages of our team where either a
> > test is missing or failing.  My idea was that some autopkgtest-pkg-* is
> > "test is not missing".  What is your opinion as debian-ci team about my
> > idea?
>
> It seems you want to be processing the message field then, but honestly
> if there is an entry, "test is not missing". If there is no entry, "test
> is missing". Failing is "fail". flaky tests are also "test is not
> missing", skipped tests are also "test is not missing". Why would you
> need all those states? You have the "message" field in your UDD schema
> to check why you get the neutral state.

I definitely want to expose the full message (that's why I insisted to
put it into the table).  My intention is to expose only those packages
where some work needs to be done.  Otherwise the list will be to long.
I will think about the plan after receiving your information.

Kind regards

      Andreas.

--
http://fam-tille.de

12