running CI tests before package sponsorship

classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

running CI tests before package sponsorship

Stephen Sinclair
Dear Debian Science,

Is it possible to do CI testing on a package build in a salsa
repository that has not yet been uploaded?
I am getting ready to propose a package that features a lot of
numerical tests in its test suite and I am afraid of hitting many
unknown numerical errors on the variety of architectures that Debian
tests on.  I would like to fully test it in an environment as close to
ci.debian.net as possible before wasting a sponsor's time with a bunch
of test failures.

I see that I could set up Gitlab CI, perhaps this would be a good
approach.  But am I right to understand that this is not the same
testing environment as ci.debian.net?  Are there already
science/numerical packages using Salsa's CI for testing prior to
upload?

regards,
Steve

Reply | Threaded
Open this post in threaded view
|

Re: running CI tests before package sponsorship

Stuart Prescott-11
Hi Steve,

You can run the test suite yourself on your local machine with either
autopkgtest (using a chroot/schroot/pbuilder/vm/etc) or without isolation
using sadt from devscripts.

Running the tests yourself only tests against the hardware you have available
and so isn't probing different architectures, but then ci.debian.net doesn't
either.

It is possible to build and run tests from gitlab CI. You can write your own
code for that or see an existing project for it:

        https://salsa.debian.org/salsa-ci-team/pipeline

(I'm not sure I like the way it has been implemented in that project but it's
an option for you)

Gitlab CI is still not probing many architectures.

Established maintainers can get access to porterboxes with various
architectures to do their own testing and debugging on different hardware. I
don't know whether that's going to be an option for you, however.

To the source of your problem -- what sort of comparisons are you having
trouble with? It's generally best that numerical tests use some sort of
"almost equals" test probing only an acceptable number of significant figures;
that should then pass on all architectures. Would it help to point to the
specific code at this stage?

Cheers
Stuart


--
Stuart Prescott    http://www.nanonanonano.net/   [hidden email]
Debian Developer   http://www.debian.org/         [hidden email]
GPG fingerprint    90E2 D2C1 AD14 6A1B 7EBB 891D BBC1 7EBB 1396 F2F7

Reply | Threaded
Open this post in threaded view
|

Re: running CI tests before package sponsorship

Drew Parsons
On 2018-10-25 06:44, Stuart Prescott wrote:
> Hi Steve,
>
> You can run the test suite yourself on your local machine with either
> autopkgtest (using a chroot/schroot/pbuilder/vm/etc)

autopkgtest can also run without a chroot,

    autopkgtest -B -- null


Reply | Threaded
Open this post in threaded view
|

Re: running CI tests before package sponsorship

Stephen Sinclair
In reply to this post by Stuart Prescott-11
Hello,

Thanks for the response.  Indeed unit tests are already running during
package build on my local machine and I use autopkgtest to test the
installed packages for proper functioning.
However this only allows me to test on x86_64.  My concern was more
that I was under the impression that Debian packages get automatically
tested on various architectures, sometimes a bit esoteric, like Sparc,
etc.
I am already trying to determine which tests in upstream are sensitive
to different versions of BLAS etc and to skip them or make them
non-fatal.  There are about a thousand tests that are run and while
they all generally pass, between devs on the project there have been
varying reports depending on operating systems for example, so I am
expecting surprises ;)

(To be clear, I would like to _catch_ all failed tests, for reporting
to upstream, but I also don't want to block the package just because
the output of one particular solver is slightly different than
expected when run e.g. on ARM or something.)

But you are saying then that if it builds and tests just fine locally
then I should not expect different conditions under ci.debian.net, is
this correct?

Currently I use cowbuilder with `gbp buildpackage` for package
building and schroot for autopkgtest, I don't know if this is optimal
but it seems to work well.

thanks,
Steve

On Thu, Oct 25, 2018 at 12:44 AM Stuart Prescott <[hidden email]> wrote:

>
> Hi Steve,
>
> You can run the test suite yourself on your local machine with either
> autopkgtest (using a chroot/schroot/pbuilder/vm/etc) or without isolation
> using sadt from devscripts.
>
> Running the tests yourself only tests against the hardware you have available
> and so isn't probing different architectures, but then ci.debian.net doesn't
> either.
>
> It is possible to build and run tests from gitlab CI. You can write your own
> code for that or see an existing project for it:
>
>         https://salsa.debian.org/salsa-ci-team/pipeline
>
> (I'm not sure I like the way it has been implemented in that project but it's
> an option for you)
>
> Gitlab CI is still not probing many architectures.
>
> Established maintainers can get access to porterboxes with various
> architectures to do their own testing and debugging on different hardware. I
> don't know whether that's going to be an option for you, however.
>
> To the source of your problem -- what sort of comparisons are you having
> trouble with? It's generally best that numerical tests use some sort of
> "almost equals" test probing only an acceptable number of significant figures;
> that should then pass on all architectures. Would it help to point to the
> specific code at this stage?
>
> Cheers
> Stuart
>
>
> --
> Stuart Prescott    http://www.nanonanonano.net/   [hidden email]
> Debian Developer   http://www.debian.org/         [hidden email]
> GPG fingerprint    90E2 D2C1 AD14 6A1B 7EBB 891D BBC1 7EBB 1396 F2F7