Thoughts on Debian quality, including automated testing

classic Classic list List threaded Threaded
86 messages Options
12345
Reply | Threaded
Open this post in threaded view
|

Re: A bit of experience after having updated some packages to use pbuilder-test testsuite engine.

Petter Reinholdtsen

[Junichi Uekawa]
> 3. support for X. Some of my packages are command-line console tools,
>    but many are actually graphical apps. It would be a plus to have
>    some kind of interactive/noninteractive X-based testing.

Would xnee do the trick?


--
To UNSUBSCRIBE, email to [hidden email]
with a subject of "unsubscribe". Trouble? Contact [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: A bit of experience after having updated some packages to use pbuilder-test testsuite engine.

Junichi Uekawa
Hi,

> [Junichi Uekawa]
> > 3. support for X. Some of my packages are command-line console tools,
> >    but many are actually graphical apps. It would be a plus to have
> >    some kind of interactive/noninteractive X-based testing.
>
> Would xnee do the trick?

Actually, I wasn't aware of xnee.

Does anyone have experience with using it?  stability/repeatability of
test is important if it were to be automated.

regards,
        junichi
--
dancer@{debian.org,netfort.gr.jp}   Debian Project


--
To UNSUBSCRIBE, email to [hidden email]
with a subject of "unsubscribe". Trouble? Contact [hidden email]

Reply | Threaded
Open this post in threaded view
|

Bug#315736: [Pbuilder-maint] Re: A bit of experience after having updated some packages to use pbuilder-test testsuite engine.

Junichi Uekawa
Hi,

> > [Junichi Uekawa]
> > > 3. support for X. Some of my packages are command-line console tools,
> > >    but many are actually graphical apps. It would be a plus to have
> > >    some kind of interactive/noninteractive X-based testing.
> >
> > Would xnee do the trick?
>
> Actually, I wasn't aware of xnee.
>
> Does anyone have experience with using it?  stability/repeatability of
> test is important if it were to be automated.

I've checked out xnee, I've noticed that the info documentation was
not properly installed, and provided a patch. Looked over the BTS,
tried running, found the same error as bug 315736.

My conclusion so far: completely unusable.

regards,
        junichi
--
dancer@{debian.org,netfort.gr.jp}   Debian Project


--
To UNSUBSCRIBE, email to [hidden email]
with a subject of "unsubscribe". Trouble? Contact [hidden email]

Reply | Threaded
Open this post in threaded view
|

Bug#315736: [Pbuilder-maint] Re: A bit of experience after having updated some packages to use pbuilder-test testsuite engine.

Loïc Minier-6
        Hi,

On Tue, Jan 24, 2006, Junichi Uekawa wrote:
> I've checked out xnee, I've noticed that the info documentation was
> not properly installed, and provided a patch. Looked over the BTS,
> tried running, found the same error as bug 315736.

 Curious of xnee, I installed it to try out too, I was hit by the same
 error, launching in -v suggested the program was stopping after
 "closing down while loop in async loop", I grepped for that in the
 source and found it would exit after a variable (presumably the number
 of events to record) would reach 0, this var being initialized to 100.

 Try running:
    xnee --record --mouse --keyboard --future-clients --out rec.xnl \
        --100k-log
 you should have room for more events.

 In general, keyboard and mouse record and replay worked fine here, but
 I did not particularly configure xnee for special cases like compose
 keys or meta keys, the biggest problem being that the documentation is
 a bit confuse to me.

 I think that this program is a bit low-level for the tests you might
 want to achieve, and it exposes various problems to the test suite that
 are not very interesting: it has to follow strict timings, and might
 not be able to replay what you did as fast as you want it to.
   I've found dogtail[1] to be quite interesting, perhaps it is the good
 layer for the integration of functional checks in your packages?

   Cheers,

[1] <http://people.redhat.com/zcerza/dogtail/>
--
Loïc Minier <[hidden email]>
Current Earth status:   NOT DESTROYED

Reply | Threaded
Open this post in threaded view
|

Re: [Pbuilder-maint] Re: A bit of experience after having updated some packages to use pbuilder-test testsuite engine.

Junichi Uekawa
Hi,

> On Tue, Jan 24, 2006, Junichi Uekawa wrote:
> > I've checked out xnee, I've noticed that the info documentation was
> > not properly installed, and provided a patch. Looked over the BTS,
> > tried running, found the same error as bug 315736.
>
>  Curious of xnee, I installed it to try out too, I was hit by the same
>  error, launching in -v suggested the program was stopping after
>  "closing down while loop in async loop", I grepped for that in the
>  source and found it would exit after a variable (presumably the number
>  of events to record) would reach 0, this var being initialized to 100.
>
>  Try running:
>     xnee --record --mouse --keyboard --future-clients --out rec.xnl \
>         --100k-log
>  you should have room for more events.

Ok, thanks I tried.

I tried creating a test for ecawave boot/close sequence, but it is
difficult to create the test case and run the test case.  A testtool
really needs to react to UI updates rather than depend on a
predetermined timing. Also, it requires real-time trial-and-error to
create testsuites, which is bad.
 
>  I think that this program is a bit low-level for the tests you might
>  want to achieve, and it exposes various problems to the test suite that
>  are not very interesting: it has to follow strict timings, and might
>  not be able to replay what you did as fast as you want it to.
>    I've found dogtail[1] to be quite interesting, perhaps it is the good
>  layer for the integration of functional checks in your packages?

dogtail sounds more like the way to go, although it seems to require
the GUI apps to cooperate.


regards,
        junichi
--
dancer@{debian.org,netfort.gr.jp}   Debian Project


--
To UNSUBSCRIBE, email to [hidden email]
with a subject of "unsubscribe". Trouble? Contact [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: [Pbuilder-maint] A bit of experience after having updated some packages to use pbuilder-test testsuite engine.

Junichi Uekawa
In reply to this post by Junichi Uekawa
Hi,

I've blogged about it, but I'll re-post to the list for easier commenting.

http://www.netfort.gr.jp/~dancer/diary/daily/2006-Mar-4.html.en#2006-Mar-4-23:12:11

While I was reviewing Debian Weekly News translations, I noticed there
was autopkgtest. Nice to see something that emerging to be
available. I've already written quite a few tests for pbuilder
testsuite, so it's unwelcome in terms of having a incompatible test
interface. Browsing autopkgtest, it's non-obvious how to write the
tests, or how to use it, so some documentation is probably
required. pbuilder testsuite has been deployed on quite a few of my
packages, so I have some insights. It's pbuilder, so it's slow, but if
it's included in the automated workflow (kick off build --
automatically-test -- upload), it's not too bad. The basic tests that
can be done are console based tests like running ldd, and giving it
some input to get some output. The tests are in source-code
debian/pbuilder-test/, and are just shell scripts run through
run-parts. The following is a rough list of things that I've done:

    * LADSPA plugins (.so files): run analyseplugin to see that it can
      be detected: see mcp-plugins

    * library package: create and build and run a test program from
      source-code and see that it does compile and run: see ecasound
      package, dancer-xml, dlisp, libdshconfig

    * emacs lisp: run the lisp code non-interactively using -eval, and
      grep for output, see yc-el.

I've not yet done much GUI testing, since I am yet to find a usable GUI automatic testing tool.


> > >     * Let's modify pbuilder to run test-build tests and (if
> > >       possible) also the generic tool and test-install tests.
> > >       These belong, I think, better into pbuilder then piuparts,
> > >       but it might be that piuparts should run them also.
> >
> > pbuilder hook is available for those who want to test this framework.
>
> I've been working with this, with a very simplistic set-up of running
> multiple shell scripts. There are things that you know beforehand (if
> you know the gory details of pbuilder as it is documented).
>
> 1. your source code is available in /tmp/buildd/package-version/,
> which you can probably match with the wildcard '/tmp/buildd/*/debian/../'
>
> 2. your testsuite resides in /tmp/buildd/*/debian/pbuilder-testsuite/*,
> which will be executed with run-parts.
>
> 3. previous version of your package will be installed by B92test-pkg
> by default, from your default apt sources, and if it fails, test won't progress.
>
> 4. within the chroot, B92test-pkg by default does the following:
> your previous version is installed from apt sources
> your previous version is removed
> your new package is installed with dpkg -i
> this sequence should be improved, and should probably allow failures
>
>
>
> Things that I really want to have after testing a few packages.
>
>
> 1. support for common operations, maybe a shell library is in order?
>    For example, I usually want to compile/link/execute a test code to
>    test a -dev package. That requires some amount of idiom in plain
>    shell code, and I shouldn't have to copy-and-paste code all over
>    the place.
>
> 2. support for providing source data. Most applications eat data and
>    output data. I really want some way to provide data and to say
>    what's the expected output.
>
> 3. support for X. Some of my packages are command-line console tools,
>    but many are actually graphical apps. It would be a plus to have
>    some kind of interactive/noninteractive X-based testing.


regards,
        junichi
--
dancer@{debian.org,netfort.gr.jp}   Debian Project


--
To UNSUBSCRIBE, email to [hidden email]
with a subject of "unsubscribe". Trouble? Contact [hidden email]

12345