Packaging libsis-jhdf5-java -- help needed

classic Classic list List threaded Threaded
17 messages Options
Reply | Threaded
Open this post in threaded view
|

Packaging libsis-jhdf5-java -- help needed

Pierre Gruet
Hi everyone,

I have been trying to package libsis-jhdf5-java, after Andreas imported the
last upstream version.
This package builds a java package with a .jar file and a jni package with
native code used by the .jar.

I have been able to:
- refresh patches;
- get rid of the private header H5private.h of source package hdf5, which is
not shipped by any package. Only a few simple preprocessor directives of
that file were used;
- update the list of build-depends;
- have the package build, including the override of dh_auto_test that caused
issues previously.

Yet:
- it seems that only a few upstream-provided tests are run in dh_auto_test,
so having the build complete is maybe not so meaningful :-( ;
- I have begun designing tests for the autopkgtest testsuite, using
upstream-provided tests, and while around 30 of them pass, there remains a
lot of failing tests. It seems that the linking of the jni with the jar is
not correctly done at build-time.

At that point I would need help, as this package is complicated: the build
processes of the jar and the jni are somehow entangled and I do not have
enough knowledge of Java packaging to be able to solve the issues I'm facing.

Maybe there does not remain so much to be done; if someone has Java
knowledge and could look to the current packaging I have put into Salsa [1],
this would be really great.

Thank you and have a good week,

All the best,
Pierre

[1] https://salsa.debian.org/med-team/libsis-jhdf5-java

Reply | Threaded
Open this post in threaded view
|

Re: Packaging libsis-jhdf5-java -- help needed

Andreas Tille-2
Hi Gilles,

On Wed, Apr 29, 2020 at 05:10:38PM +0200, Gilles Filippini wrote:

>
> I've cloned the git repo and attempted a build. But the dh_auto_test
> part doesn't execute any test actually:
>
>    debian/rules override_dh_auto_test
> make[1]: Entering directory
> ...
> ===============================================
>     All
>     Tests run: 0, Failures: 0, Skips: 0
> ===============================================
>
>
> ===============================================
> All
> Total tests run: 0, Failures: 0, Skips: 0
> ===============================================

Thanks a lot for checking - any idea how to force execution of
the tests?

Kind regards

     Andreas.

--
http://fam-tille.de

Reply | Threaded
Open this post in threaded view
|

Re: Packaging libsis-jhdf5-java -- help needed

Pierre Gruet
Hi Gilles,

Le 29/04/2020 à 17:53, Gilles Filippini a écrit :

> Andreas Tille a écrit le 29/04/2020 à 17:36 :
>> Hi Gilles,
>>
>> On Wed, Apr 29, 2020 at 05:10:38PM +0200, Gilles Filippini wrote:
>>>
>>> I've cloned the git repo and attempted a build. But the dh_auto_test
>>> part doesn't execute any test actually:
>>>
>>>    debian/rules override_dh_auto_test
>>> make[1]: Entering directory
>>> ...
>>> ===============================================
>>>     All
>>>     Tests run: 0, Failures: 0, Skips: 0
>>> ===============================================
>>>
>>>
>>> ===============================================
>>> All
>>> Total tests run: 0, Failures: 0, Skips: 0
>>> ===============================================
>>
>> Thanks a lot for checking - any idea how to force execution of
>> the tests?
>
> No, I don't know how this is supposed to work :/
>

Thanks for checking; I had also seen that these tests did not run, this will
have to be fixed. More important is the fact that I have put and arranged
the piece of code of dh_auto_test override into a test in
debian/tests/providedTests; here they run, and while the first ones pass,
next ones fail and I feel this is because the java code cannot link to the
native code in the jni package.

Yet, my problem is that I have not been able to investigate more deeply, as
the Java part is build using Gradle but the jni part is built outside of
Gradle, using a single command.
I thus don't know if the link between the Java and native code is correctly
done at that stage.

While I agree, of course, that we will need that the tests in
override_dh_auto_test run, we are already able to reproduce the problem I am
describing by running autopkgtest on the built package.

>
> _g.
>

Thanks again for considering this,

All the best,
Pierre

Reply | Threaded
Open this post in threaded view
|

Re: Packaging libsis-jhdf5-java -- help needed

Gilles Filippini-2
Pierre Gruet a écrit le 30/04/2020 à 17:40 :

> Hi Gilles,
>
> Le 29/04/2020 à 17:53, Gilles Filippini a écrit :
>> Andreas Tille a écrit le 29/04/2020 à 17:36 :
>>> Hi Gilles,
>>>
>>> On Wed, Apr 29, 2020 at 05:10:38PM +0200, Gilles Filippini wrote:
>>>>
>>>> I've cloned the git repo and attempted a build. But the dh_auto_test
>>>> part doesn't execute any test actually:
>>>>
>>>>    debian/rules override_dh_auto_test
>>>> make[1]: Entering directory
>>>> ...
>>>> ===============================================
>>>>     All
>>>>     Tests run: 0, Failures: 0, Skips: 0
>>>> ===============================================
>>>>
>>>>
>>>> ===============================================
>>>> All
>>>> Total tests run: 0, Failures: 0, Skips: 0
>>>> ===============================================
>>>
>>> Thanks a lot for checking - any idea how to force execution of
>>> the tests?
>>
>> No, I don't know how this is supposed to work :/
>>
>
> Thanks for checking; I had also seen that these tests did not run, this will
> have to be fixed. More important is the fact that I have put and arranged
> the piece of code of dh_auto_test override into a test in
> debian/tests/providedTests; here they run, and while the first ones pass,
> next ones fail and I feel this is because the java code cannot link to the
> native code in the jni package.
>
> Yet, my problem is that I have not been able to investigate more deeply, as
> the Java part is build using Gradle but the jni part is built outside of
> Gradle, using a single command.
> I thus don't know if the link between the Java and native code is correctly
> done at that stage.
>
> While I agree, of course, that we will need that the tests in
> override_dh_auto_test run, we are already able to reproduce the problem I am
> describing by running autopkgtest on the built package.

Here is what I obtain running autopkgtest against the binary packages:

autopkgtest [19:41:00]: test providedTests: [-----------------------
xargs: javac: No such file or directory
autopkgtest [19:41:00]: test providedTests: -----------------------]
autopkgtest [19:41:01]: test providedTests:  - - - - - - - - - - results
- - - - - - - - - -
providedTests        FAIL non-zero exit status 127
autopkgtest [19:41:01]: test providedTests:  - - - - - - - - - - stderr
- - - - - - - - - -
xargs: javac: No such file or directory
autopkgtest [19:41:01]: @@@@@@@@@@@@@@@@@@@@ summary
providedTests        FAIL non-zero exit status 127

Is the salsa git repo up to date?

Best,

_g.

Reply | Threaded
Open this post in threaded view
|

Re: Packaging libsis-jhdf5-java -- help needed

Pierre Gruet
Hello Gilles,

Le 30/04/2020 à 23:17, Gilles Filippini a écrit :

>>>
>>> Thanks for checking; I had also seen that these tests did not run, this will
>>> have to be fixed. More important is the fact that I have put and arranged
>>> the piece of code of dh_auto_test override into a test in
>>> debian/tests/providedTests; here they run, and while the first ones pass,
>>> next ones fail and I feel this is because the java code cannot link to the
>>> native code in the jni package.
>>>
>>> Yet, my problem is that I have not been able to investigate more deeply, as
>>> the Java part is build using Gradle but the jni part is built outside of
>>> Gradle, using a single command.
>>> I thus don't know if the link between the Java and native code is correctly
>>> done at that stage.
>>>
>>> While I agree, of course, that we will need that the tests in
>>> override_dh_auto_test run, we are already able to reproduce the problem I am
>>> describing by running autopkgtest on the built package.
>>
>> Here is what I obtain running autopkgtest against the binary packages:
>>
>> autopkgtest [19:41:00]: test providedTests: [-----------------------
>> xargs: javac: No such file or directory
>> autopkgtest [19:41:00]: test providedTests: -----------------------]
>> autopkgtest [19:41:01]: test providedTests:  - - - - - - - - - - results
>> - - - - - - - - - -
>> providedTests        FAIL non-zero exit status 127
>> autopkgtest [19:41:01]: test providedTests:  - - - - - - - - - - stderr
>> - - - - - - - - - -
>> xargs: javac: No such file or directory
>> autopkgtest [19:41:01]: @@@@@@@@@@@@@@@@@@@@ summary
>> providedTests        FAIL non-zero exit status 127
>>
>> Is the salsa git repo up to date?
>
> I've managed to run the tests (adding default-jdk the the Depends field
> into debian/tests/control), and reproduced the failures you mentioned.
>
> I can't tell whether it it the library's fault or the test cases' one,
> but most failures are due to the JNI library not being loaded. Adding
>  System.loadLibrary("sis-jhfd5");
> to the init() method of HDF5DataSetRandomAccessFileTest.java fixed them.
>
> It remains 16 unrelated failures (IllegalArgumentException,
> AssertionError, HDF5DatatypeInterfaceException).
>

Really sorry, I had forgotten to push my last change, which is exactly what
you did (putting default-jdk in the Depends field of the tests). I have just
done it.

Thanks for looking at the issue and proposing a fix. It is very helpful!
I will thus look at it and investigate further the failures that remain.


>
> Best,
>
> _g.
>

Best regards,
Pierre

Reply | Threaded
Open this post in threaded view
|

Re: Packaging libsis-jhdf5-java -- help needed

Pierre Gruet
Hi Gilles,

Le 01/05/2020 à 01:12, Gilles Filippini a écrit :

> Pierre Gruet a écrit le 30/04/2020 à 23:45 :
>> Thanks for looking at the issue and proposing a fix. It is very helpful!
>> I will thus look at it and investigate further the failures that remain.
>
> This package looks weird. Where does the mix-up between libhdf5-java and
> libsis-jhdf5-java come from? There shouldn't have any dependency to
> libhdf5-java as I understand it. The tests failures come from this
> confusion.
>
> Using the attached patch, the package builds correctly and only 10
> failing test cases remain.
>
> BTW the override_dh_auto_clean target should be reintroduced.
>
> Best,
>
> _g.
>

Thank you for this additional feedback.
Concerning the mix-up, upstream was previously using a file from hdf5 to
build sis-jhdf5, I removed it some days ago.
I introduced other dependencies at that time and did not remove them...
thanks for your patch thus!

I was planning to consider a full refresh of debian/rules, including
dh_auto_clean.

Best regards,
Pierre

Reply | Threaded
Open this post in threaded view
|

Re: Packaging libsis-jhdf5-java -- help needed

Pierre Gruet
In reply to this post by Pierre Gruet
Hi Andreas and Gilles,


Andreas, I have just seen you uploaded my commits of libsis-jhdf5-java
to unstable, thanks for that!

I had not written to ask for it as I intended to do further changes
before the upload (the script h5ar in /usr/bin should be removed or
another jar should be compiled, as h5ar is not working as is).
But this is not a problem as the package you uploaded builds well, all
tests pass and the jar and jni are usable. I shall provide another
version soon, that fixes the /usr/bin/h5ar issue.


Gilles, huge thanks again for your help, with your input I could solve
the remaining issues and, afterwards, improve debian/rules consequently.


All the best,
Pierre

Reply | Threaded
Open this post in threaded view
|

Re: Packaging libsis-jhdf5-java -- help needed

Andreas Tille-5
Hi Pierre,

On Thu, May 07, 2020 at 08:16:07AM +0200, Pierre Gruet wrote:
> Andreas, I have just seen you uploaded my commits of libsis-jhdf5-java
> to unstable, thanks for that!
>
> I had not written to ask for it as I intended to do further changes
> before the upload (the script h5ar in /usr/bin should be removed or
> another jar should be compiled, as h5ar is not working as is).
> But this is not a problem as the package you uploaded builds well, all
> tests pass and the jar and jni are usable. I shall provide another
> version soon, that fixes the /usr/bin/h5ar issue.

Yes, I was wondering whether you plan additional changes but fixing the
long standing RC bug seemed to be more important for the moment.  I also
think that observing autobuilders and possibly fix issues there in a
subsequent upload with your additional changes could be a good idea.
 
> Gilles, huge thanks again for your help, with your input I could solve
> the remaining issues and, afterwards, improve debian/rules consequently.

Thanks a lot to you both.  Your work is really appreciated

     Andreas.

--
http://fam-tille.de

Reply | Threaded
Open this post in threaded view
|

Re: Packaging libsis-jhdf5-java -- help needed

Andreas Tille-5
On Thu, May 07, 2020 at 09:43:11AM +0200, Andreas Tille wrote:
> I also
> think that observing autobuilders and possibly fix issues there in a
> subsequent upload with your additional changes could be a good idea.

As expected there is some more work to do which escaped my sponsors
view: See bug #959955. :-)

Kind regards

      Andreas.

--
http://fam-tille.de

Reply | Threaded
Open this post in threaded view
|

Re: Packaging libsis-jhdf5-java -- help needed

Pierre Gruet
Hi Andreas,

Le 07/05/2020 à 16:26, Andreas Tille a écrit :

> On Thu, May 07, 2020 at 09:43:11AM +0200, Andreas Tille wrote:
>> I also
>> think that observing autobuilders and possibly fix issues there in a
>> subsequent upload with your additional changes could be a good idea.
>
> As expected there is some more work to do which escaped my sponsors
> view: See bug #959955. :-)
>
> Kind regards
>
>       Andreas.
>

Wow, I had not seen that, thanks for showing me. I will prepare a new
upload very soon to fix the issue.

Best regards,
Pierre

Reply | Threaded
Open this post in threaded view
|

Re: Packaging libsis-jhdf5-java -- help needed

Pierre Gruet
In reply to this post by Andreas Tille-5
Hi again,

Le 07/05/2020 à 16:26, Andreas Tille a écrit :

> On Thu, May 07, 2020 at 09:43:11AM +0200, Andreas Tille wrote:
>> I also
>> think that observing autobuilders and possibly fix issues there in a
>> subsequent upload with your additional changes could be a good idea.
>
> As expected there is some more work to do which escaped my sponsors
> view: See bug #959955. :-)
>
> Kind regards
>
>       Andreas.
>

I have just uploaded the new version 19.04.0+dfsg-2 of libsis-jhdf5-java
to Salsa: to close bug #959955, I have changed the name of the .so that
is generated to avoid name collisions. All tests from build-time and
autopkgtest testsuite are passing.

All the best,
Pierre

Reply | Threaded
Open this post in threaded view
|

Re: Packaging libsis-jhdf5-java -- help needed

Andreas Tille-6
Hi Pierre,

On Thu, May 07, 2020 at 11:28:32PM +0200, Pierre Gruet wrote:
> I have just uploaded the new version 19.04.0+dfsg-2 of libsis-jhdf5-java
> to Salsa: to close bug #959955, I have changed the name of the .so that
> is generated to avoid name collisions. All tests from build-time and
> autopkgtest testsuite are passing.

Very nice.  Thanks a lot for your work.  I've built and installed the
package.  When noticing that there is no manpage for h5ar I intended to
check whether it might be cheap to create one via help2man.  However,
this ended up in

$ h5ar
cat: /usr/version.txt: Datei oder Verzeichnis nicht gefunden
Error: Unable to access jarfile /usr/lib/sis-jhdf5-h5ar-cli-.jar


I think we need some more patches adjusting pathes to let this script
run successfully.  Are you volunteering to have a look?

Thanks a lot for your work in any case

    Andreas.

--
http://fam-tille.de

Reply | Threaded
Open this post in threaded view
|

Re: Packaging libsis-jhdf5-java -- help needed

Pierre Gruet
Hi Andreas,

Le 08/05/2020 à 22:05, Andreas Tille a écrit :

>
> Very nice.  Thanks a lot for your work.  I've built and installed the
> package.  When noticing that there is no manpage for h5ar I intended to
> check whether it might be cheap to create one via help2man.  However,
> this ended up in
>
> $ h5ar
> cat: /usr/version.txt: Datei oder Verzeichnis nicht gefunden
> Error: Unable to access jarfile /usr/lib/sis-jhdf5-h5ar-cli-.jar
>
>
> I think we need some more patches adjusting pathes to let this script
> run successfully.  Are you volunteering to have a look?
>

Absolutely. This is part of the complementary tasks I was planning to do
(my Salsa push of yesterday was only to fix the RC bug). h5ar was not in
the previous packaging of libsis-jhdf5-java.

I have seen that creating a man page should be easy, and while running
h5ar as you did, I saw there was an issue with that version.txt file
(easy to fix) and another one with the jar file, which is currently not
built as it has not been part of the package libsis-jhdf5-java up to now.
I will thus try to patch the gradle build script (or to edit d/rules) to
build this new jar, and I need to learn a bit more about gradle to do
so, which I am doing those days :-)
Curiously, the jar is provided by upstream, there exists a task to build
it in its gradle script, but it is not built by jh_build.

>
> Thanks a lot for your work in any case
>
>     Andreas.
>

Thanks for having looked at this!
All the best,
Pierre

Reply | Threaded
Open this post in threaded view
|

Re: Packaging libsis-jhdf5-java -- help needed

Andreas Tille-5
Hi Pierre,

On Fri, May 08, 2020 at 10:24:47PM +0200, Pierre Gruet wrote:
>
> Absolutely. This is part of the complementary tasks I was planning to do
> (my Salsa push of yesterday was only to fix the RC bug). h5ar was not in
> the previous packaging of libsis-jhdf5-java.

If you think you might be able to care for the h5ar binary issue in the
next couple of days I would delay the upload to fix the RC bug.  I guess
this will not have any practical drawback if the propagation of this
package to testing will be delayed some more days.
 
> I have seen that creating a man page should be easy, and while running
> h5ar as you did, I saw there was an issue with that version.txt file
> (easy to fix) and another one with the jar file, which is currently not
> built as it has not been part of the package libsis-jhdf5-java up to now.
> I will thus try to patch the gradle build script (or to edit d/rules) to
> build this new jar, and I need to learn a bit more about gradle to do
> so, which I am doing those days :-)

I can confirm that you can't prevent learning something when dealing
with Debian packages. ;-)

> Curiously, the jar is provided by upstream, there exists a task to build
> it in its gradle script, but it is not built by jh_build.

Sounds strange.

> Thanks for having looked at this!

You are welcome - the basic work was done by you.

Kind regards

      Andreas.

--
http://fam-tille.de

Reply | Threaded
Open this post in threaded view
|

Re: Packaging libsis-jhdf5-java -- help needed

Pierre Gruet
Hi Andreas,

Le 08/05/2020 à 22:51, Andreas Tille a écrit :

> Hi Pierre,
>
> On Fri, May 08, 2020 at 10:24:47PM +0200, Pierre Gruet wrote:
>>
>> Absolutely. This is part of the complementary tasks I was planning to do
>> (my Salsa push of yesterday was only to fix the RC bug). h5ar was not in
>> the previous packaging of libsis-jhdf5-java.
>
> If you think you might be able to care for the h5ar binary issue in the
> next couple of days I would delay the upload to fix the RC bug.  I guess
> this will not have any practical drawback if the propagation of this
> package to testing will be delayed some more days.
>

Fine then, we can delay the upload as I plan to care for h5ar those days :-)

>  > [...]
>
> I can confirm that you can't prevent learning something when dealing
> with Debian packages. ;-)
>

I guess this is one of the great things about packaging!

>
> Kind regards
>
>       Andreas.
>

Have a nice day,
Pierre

Reply | Threaded
Open this post in threaded view
|

Re: Packaging libsis-jhdf5-java -- help needed

Pierre Gruet
In reply to this post by Andreas Tille-5
Hi Andreas,

Le 08/05/2020 à 22:51, Andreas Tille a écrit :
>
> If you think you might be able to care for the h5ar binary issue in the
> next couple of days I would delay the upload to fix the RC bug.  I guess
> this will not have any practical drawback if the propagation of this
> package to testing will be delayed some more days.
>

This is done, I have pushed the changes to Salsa. The candidate upload
would:
- close bug 959955 by renaming the .so (we discussed it some days ago);
- provide a *new binary package*, which I called h5ar. It contains the
wrapper script /usr/bin/h5ar and the jar that it wraps. The build of
this jar is done using jh_build as it is not handled by the gradle call
of upstream. I put the wrapper and its jar in a new package in order to
comply with Debian Java policy [0], which makes a clear distinction
between programs and libraries;
- provide a test for this h5ar package in the testsuite;
- provide hardening while building this .so;
- provide tests of the testsuite as package examples.


Thanks in advance for your time and possible future advice on this
packaging (in particular in case you would recommend another name for
the new package h5ar),

Best regards,
Pierre


[0] https://www.debian.org/doc/packaging-manuals/java-policy/ch02.html

Reply | Threaded
Open this post in threaded view
|

Re: Packaging libsis-jhdf5-java -- help needed

Andreas Tille-5
Hi Pierre,

On Tue, May 12, 2020 at 11:11:09PM +0200, Pierre Gruet wrote:

> This is done, I have pushed the changes to Salsa. The candidate upload
> would:
> - close bug 959955 by renaming the .so (we discussed it some days ago);
> - provide a *new binary package*, which I called h5ar. It contains the
> wrapper script /usr/bin/h5ar and the jar that it wraps. The build of
> this jar is done using jh_build as it is not handled by the gradle call
> of upstream. I put the wrapper and its jar in a new package in order to
> comply with Debian Java policy [0], which makes a clear distinction
> between programs and libraries;
> - provide a test for this h5ar package in the testsuite;
> - provide hardening while building this .so;
> - provide tests of the testsuite as package examples.

Thanks a lot for your very sensible changes.
 
> Thanks in advance for your time and possible future advice on this
> packaging (in particular in case you would recommend another name for
> the new package h5ar),

I think the name is fine and I've uploaded the package to new (due to
the new binary package).

Thanks a lot for your work.  As I said we have more Java packaging tasks
and you are kindly invited to help on these. :-)

Kind regards

       Andreas.
 
> [0] https://www.debian.org/doc/packaging-manuals/java-policy/ch02.html

--
http://fam-tille.de