Detecting (upcoming) problems using automatic tests

classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

Detecting (upcoming) problems using automatic tests

Julian Andres Klode-4
Dear fellow developers,

I was just thinking that adding deprecation warnings and stuff
to software is "nice", but the problem with warnings is that they
tend to not break tests.

I feel like it would be nice to come up with a standard environment
variable to turn warnings into errors, so we can ensure issues are
fixed and the warnings are actually useful.

For example, we could set such a variable for autopkgtests on unstable,
and then see deprecation errors there without breaking testing migration
because a thing was deprecated.

What do you think?

(I'm not subscribed to devel, so you probably want to CC me as I
 read devel only occasionally via nntp otherwise)
--
debian developer - deb.li/jak | jak-linux.org - free software dev
ubuntu core developer                              i speak de, en

Reply | Threaded
Open this post in threaded view
|

Re: Detecting (upcoming) problems using automatic tests

Chris Lamb -2
[adding jak@ to CC as requested; no need to CC me, however...]

Hi Julian,

> I was just thinking that adding deprecation warnings and stuff
> to software is "nice", but the problem with warnings is that they
> tend to not break tests.

I'm guessing you have a particular package or use-case in mind that
sparked this idea — could you share? That might help make this abstract
concept a little more concrete.

I'm also assuming that you meant for this to be wider than just GCC
so, for example, making -Werror global wouldn't be sufficient as it
wouldn't catch, say, warnings from pure-Python packages.

> I feel like it would be nice to come up with a standard environment
> variable to turn warnings into errors, so we can ensure issues are
> fixed and the warnings are actually useful.

Hm, although perhaps DEB_BUILD_OPTIONS is the prefered place for this
kind of toggle rather than an environment variable?


Regards,

--
      ,''`.
     : :'  :     Chris Lamb
     `. `'`      [hidden email] 🍥 chris-lamb.co.uk
       `-

Reply | Threaded
Open this post in threaded view
|

Re: Detecting (upcoming) problems using automatic tests

Simon Richter
Hi,

On Thu, Jul 11, 2019 at 10:23:59PM -0300, Chris Lamb wrote:

> > I feel like it would be nice to come up with a standard environment
> > variable to turn warnings into errors, so we can ensure issues are
> > fixed and the warnings are actually useful.

> Hm, although perhaps DEB_BUILD_OPTIONS is the prefered place for this
> kind of toggle rather than an environment variable?

We already have "nocheck" to disable tests, and IMO the default should be
to run tests if we can.

   Simon

Reply | Threaded
Open this post in threaded view
|

Re: Detecting (upcoming) problems using automatic tests

Chris Lamb -2
Hi Simon,

> > Hm, although perhaps DEB_BUILD_OPTIONS is the prefered place for this
> > kind of toggle rather than an environment variable?
>
> We already have "nocheck" to disable tests, and IMO the default should be
> to run tests if we can.

Unless I am grossly misunderstanding him, Julian's suggestion was for
an optional and orthogonal boolean of some description that turns
warnings into errors, rather than a proposal to cease execution of
tests by default.


Regards,

--
      ,''`.
     : :'  :     Chris Lamb
     `. `'`      [hidden email] 🍥 chris-lamb.co.uk
       `-

Reply | Threaded
Open this post in threaded view
|

Re: Detecting (upcoming) problems using automatic tests

Julian Andres Klode-4
In reply to this post by Chris Lamb -2
On Thu, Jul 11, 2019 at 10:23:59PM -0300, Chris Lamb wrote:

> Hi Julian,
>
> > I was just thinking that adding deprecation warnings and stuff
> > to software is "nice", but the problem with warnings is that they
> > tend to not break tests.
>
> I'm guessing you have a particular package or use-case in mind that
> sparked this idea — could you share? That might help make this abstract
> concept a little more concrete.
>
> I'm also assuming that you meant for this to be wider than just GCC
> so, for example, making -Werror global wouldn't be sufficient as it
> wouldn't catch, say, warnings from pure-Python packages.

I was thinking that if we add a run-time warning to APT because we
want to remove or change something next release cycle that that probably
won't break your autopkgtest if it just looks for success.

This means that there is not really much advantage to having deprecation
warnings, as everything will "surprisingly" break the next release cycle.

>
> > I feel like it would be nice to come up with a standard environment
> > variable to turn warnings into errors, so we can ensure issues are
> > fixed and the warnings are actually useful.
>
> Hm, although perhaps DEB_BUILD_OPTIONS is the prefered place for this
> kind of toggle rather than an environment variable?

For builds, yes; for autopkgtest I guess not.

--
debian developer - deb.li/jak | jak-linux.org - free software dev
ubuntu core developer                              i speak de, en