[Help] Failed to upgrade bcftools to version 1.5

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
17 messages Options
Reply | Threaded
Open this post in threaded view
|

[Help] Failed to upgrade bcftools to version 1.5

Andreas Tille-5
Hi,

I intend to upgrade bcdtools to enable python-pysam upgrade.  I commited
my intermediate state where I disabled some patches fiddling around with
config details to use Debian packaged libs and instead tried to install
some additional files in libhts-dev under /usr/lib/htslib that should
help bcftools to find the information it needs.  I blatantly failed with
this attempt and will not have time to continue for the next three days.
So if somebody would like to step in here I (and several other users of
samtools related software) would be really happy.

Kind regards

     Andreas.

--
http://fam-tille.de

Reply | Threaded
Open this post in threaded view
|

Re: [Help] Failed to upgrade bcftools to version 1.5

Afif Elghraoui-6
I was under the impression that htslib 1.5, required for bcftools 1.5,
has a soname bump. There is no pysam release yet to wrap htslib 1.5, so
I think upgrading bcftools/samtools/htslib right now to 1.5 will just
make it impossible to use pysam until a new release comes out, right?

I did not get through updating pysam since my time became scarce and the
time I did spend on it was not so effecient with a slow laptop.

regards
Afif

--
Afif Elghraoui | عفيف الغراوي
http://afif.ghraoui.name

Reply | Threaded
Open this post in threaded view
|

Re: [Help] Failed to upgrade bcftools to version 1.5

Graham Inggs-3
On 24 July 2017 at 06:12, Afif Elghraoui <[hidden email]> wrote:
> I was under the impression that htslib 1.5, required for bcftools 1.5,
> has a soname bump. There is no pysam release yet to wrap htslib 1.5, so
> I think upgrading bcftools/samtools/htslib right now to 1.5 will just
> make it impossible to use pysam until a new release comes out, right?

Also, the htslib 1.4.1 is almost complete at 85%.
https://release.debian.org/transitions/html/auto-htslib.html
I think it would be better to finish that transition before starting
another one, otherwise some packages may get auto-removed from
testing.

Reply | Threaded
Open this post in threaded view
|

Re: [Help] Failed to upgrade bcftools to version 1.5

Andreas Tille-5
In reply to this post by Afif Elghraoui-6
Hi Afif,

On Mon, Jul 24, 2017 at 12:12:04AM -0400, Afif Elghraoui wrote:
> I was under the impression that htslib 1.5, required for bcftools 1.5,
> has a soname bump. There is no pysam release yet to wrap htslib 1.5, so
> I think upgrading bcftools/samtools/htslib right now to 1.5 will just
> make it impossible to use pysam until a new release comes out, right?

I have no idea - I just realised that I'm unable to build latest
python-pysam which ends up in

...
pysam/libcutils.c:10574:20: warning: ‘__pyx_v_retval’ may be used uninitialized in this function [-Wmaybe-uninitialized]
             return PyInt_FromLong((long) value);
                    ^~~~~~~~~~~~~~~~~~~~~~~~~~~~
pysam/libcutils.c:5493:7: note: ‘__pyx_v_retval’ was declared here
   int __pyx_v_retval;
       ^~~~~~~~~~~~~~
x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fno-strict-aliasing -g -O2 -fdebug-prefix-map=/build/python-pysam-0.11.2.2+ds=. -fstack-protector-strong -Wf
x86_64-linux-gnu-gcc -pthread -shared -Wl,-O1 -Wl,-Bsymbolic-functions -Wl,-z,relro -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -Wdate-time -D_FORTIFY_SOURCE=2 -g
/usr/bin/ld: cannot find -lcsamtools
/usr/bin/ld: cannot find -lcbcftools
collect2: error: ld returned 1 exit status


I simply assumed that the cause would be that those missing libs are
part of new upstream versions of the dependencies.

 
> I did not get through updating pysam since my time became scarce and the
> time I did spend on it was not so effecient with a slow laptop.

I would be really happy if somebody more educated than me in the
field of bcftools/htslib/samtools would step in and fix all bugs
(whatever version - I do not care as long as it is done).

Kind regards

        Andreas.

--
http://fam-tille.de

Reply | Threaded
Open this post in threaded view
|

Re: [Help] Failed to upgrade bcftools to version 1.5

Afif Elghraoui-6
In reply to this post by Graham Inggs-3


على الإثنين 24 تـمـوز 2017 ‫03:18، كتب Graham Inggs:
> Also, the htslib 1.4.1 is almost complete at 85%.
> https://release.debian.org/transitions/html/auto-htslib.html
> I think it would be better to finish that transition before starting
> another one, otherwise some packages may get auto-removed from
> testing.

Oh, my mistake. I misremembered the 1.4 transition as 1.5.

regards
Afif
--
Afif Elghraoui | عفيف الغراوي
http://afif.ghraoui.name

Reply | Threaded
Open this post in threaded view
|

Re: [Help] Failed to upgrade bcftools to version 1.5

Andreas Tille-5
Hi,

On Mon, Jul 24, 2017 at 11:23:13PM -0400, Afif Elghraoui wrote:
> > Also, the htslib 1.4.1 is almost complete at 85%.
> > https://release.debian.org/transitions/html/auto-htslib.html
> > I think it would be better to finish that transition before starting
> > another one, otherwise some packages may get auto-removed from
> > testing.
>
> Oh, my mistake. I misremembered the 1.4 transition as 1.5.

So how will we deal with #865006 and issues in python-pysam?

Kind regards

     Andreas.

--
http://fam-tille.de

Reply | Threaded
Open this post in threaded view
|

Re: [Help] Failed to upgrade bcftools to version 1.5

Graham Inggs-3
On 25/07/2017 11:20, Andreas Tille wrote:
> So how will we deal with #865006 and issues in python-pysam?

I made a few minor changes to python-pysam 0.11.2.2 (which builds
against bcftools 1.4.1) as it stands in git, and managed to get it to
build and pass its tests.  If there are no objections, I can push my
changes and upload on Monday.

The remaining bugs blocking the transition are augustus (bug #869126)
and bcftools (bug #868958).  Sascha has already pushed the augustus
changes to git, and it just needs an upload.  The two fixes for bcftools
are in the bug report, someone should review them and upload.

The patches can be grabbed from the Ubuntu package:
https://launchpad.net/ubuntu/+source/bcftools/1.4.1-2ubuntu2

Have a good weekend!

Reply | Threaded
Open this post in threaded view
|

Re: [Help] Failed to upgrade bcftools to version 1.5

Andreas Tille-2
Hi Graham,

On Fri, Jul 28, 2017 at 03:23:40PM +0200, Graham Inggs wrote:
> On 25/07/2017 11:20, Andreas Tille wrote:
> > So how will we deal with #865006 and issues in python-pysam?
>
> I made a few minor changes to python-pysam 0.11.2.2 (which builds against
> bcftools 1.4.1) as it stands in git, and managed to get it to build and pass
> its tests.  If there are no objections, I can push my changes and upload on
> Monday.

There are definitely no objections! :-)
If you push soon I can upload tomorrow - on Sunday I'll start traveling
and will not be able to upload before I'll reach DebConf venue
 
> The remaining bugs blocking the transition are augustus (bug #869126) and
> bcftools (bug #868958).  Sascha has already pushed the augustus changes to
> git, and it just needs an upload.  The two fixes for bcftools are in the bug
> report, someone should review them and upload.
>
> The patches can be grabbed from the Ubuntu package:
> https://launchpad.net/ubuntu/+source/bcftools/1.4.1-2ubuntu2

Sascha, will you care for this?

Kind regards

      Andreas.

--
http://fam-tille.de

Reply | Threaded
Open this post in threaded view
|

Re: [Help] Failed to upgrade bcftools to version 1.5

Sascha Steinbiss-5
Hi all,

[...]
>> The remaining bugs blocking the transition are augustus (bug #869126) and
>> bcftools (bug #868958).  Sascha has already pushed the augustus changes to
>> git, and it just needs an upload.  The two fixes for bcftools are in the bug
>> report, someone should review them and upload.
>
> Sascha, will you care for this?

Good timing, indeed I was going to finish up the augustus package
tonight. I took some time to finally include all the little executables
that come with the main tool, and I wanted to take another close look to
make sure they all have man pages etc.

Once I'm at it I can also take a shot at bcftools. Never worked on that
before though, let's see how it goes.

Cheers
Sascha


signature.asc (499 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: [Help] Failed to upgrade bcftools to version 1.5

Graham Inggs-3
On 28 July 2017 at 16:30, Sascha Steinbiss <[hidden email]> wrote:
> Once I'm at it I can also take a shot at bcftools. Never worked on that
> before though, let's see how it goes.

Please take note you'll need to backtract in git until version
1.4.1-2, add the patches, and upload 1.4.1-3.  I don't know any good
way to do this, perhaps Andreas knows what to do.  Otherwise, it can
wait until Monday when I get back, but it would be nice to have
bcftools uploaded before python-pysam.

Reply | Threaded
Open this post in threaded view
|

Re: [Help] Failed to upgrade bcftools to version 1.5

Andreas Tille-5
Hi Graham,

On Fri, Jul 28, 2017 at 07:27:36PM +0200, Graham Inggs wrote:
> On 28 July 2017 at 16:30, Sascha Steinbiss <[hidden email]> wrote:
> > Once I'm at it I can also take a shot at bcftools. Never worked on that
> > before though, let's see how it goes.
>
> Please take note you'll need to backtract in git until version
> 1.4.1-2, add the patches, and upload 1.4.1-3.  I don't know any good
> way to do this, perhaps Andreas knows what to do.

I'm not a Git expert but

   git format-patch 1.4.1-2

should give a set of patches that could be used for quilt after minimal
editiing (+ quilt refresh)

Kind regards

      Andreas.

--
http://fam-tille.de

Reply | Threaded
Open this post in threaded view
|

Re: [Help] Failed to upgrade bcftools to version 1.5

Sascha Steinbiss-5
In reply to this post by Graham Inggs-3
Hi all,

> Well, I was going to branch off a separate branch from the
> debian/1.4.1-2 tag, add the patches, build and upload 1.4.1-3 from
> there, and then just add the changelog entry to the current master (as
> the patches should already been in the latest upstream code, right?).
> If one wants to reproduce the exact 1.4.1-3 state it should still be
> possible via the branch.

I guess I'm going to do that for now.

> Anyway, a more acute problem is that even with the patches [1] I can't
> get bcftools 1.4.1 to build in a recent i386 sid chroot. I get some new
> failed tests that you did not encounter in the original bug report:
>
> Number of tests:
>     total   .. 617
>     passed  .. 561
>     failed  .. 56

Oops, just noticed that i386 never built in the first place, and the
patches addressed getting the currently out-of-date builds on armel and
mipsel to work. I can confirm from builds on porterboxes that these now
finish, so I'm going to upload a new bcftools 1.4.1 version with the
patches but leave the bug (#868958) open since not all 32-bit archs are
building yet.

OK with everyone?

Cheers
Sascha

Reply | Threaded
Open this post in threaded view
|

Re: [Help] Failed to upgrade bcftools to version 1.5

Graham Inggs-3
On 29/07/2017 10:36, Sascha Steinbiss wrote:
>> Well, I was going to branch off a separate branch from the
>> debian/1.4.1-2 tag, add the patches, build and upload 1.4.1-3 from
>> there, and then just add the changelog entry to the current master (as
>> the patches should already been in the latest upstream code, right?).
>> If one wants to reproduce the exact 1.4.1-3 state it should still be
>> possible via the branch.
>
> I guess I'm going to do that for now.

I think master still needs debian/patches/fix-test_vcf_query.patch and
'Fix autopkgtests by ensuring presence of all deps'.

> Oops, just noticed that i386 never built in the first place, and the
> patches addressed getting the currently out-of-date builds on armel and
> mipsel to work. I can confirm from builds on porterboxes that these now
> finish, so I'm going to upload a new bcftools 1.4.1 version with the
> patches but leave the bug (#868958) open since not all 32-bit archs are
> building yet.

Sorry, I could have been more clear about that.  Bug #868958 was
specifically about a regression in 1.4.1-1 that caused build failures on
32-bit architectures where bcftools had built successfully in the past.
Adrian Bunk has already closed that bug.

Thanks for sorting out git and uploading!

Reply | Threaded
Open this post in threaded view
|

Re: [Help] Failed to upgrade bcftools to version 1.5

Graham Inggs-3
In reply to this post by Afif Elghraoui-6
On 24/07/2017 06:12, Afif Elghraoui wrote:
> I was under the impression that htslib 1.5, required for bcftools 1.5,
> has a soname bump. There is no pysam release yet to wrap htslib 1.5, so
> I think upgrading bcftools/samtools/htslib right now to 1.5 will just
> make it impossible to use pysam until a new release comes out, right?

Python-pysam 0.11.2.2+ds-3 migrated to testing on 2017-08-06, completing
the htslib/samtools/bcftools 1.4.1 transition.

Now htslib 1.5-1 and samtools 1.5-1 have been uploaded and python-pysam
FTBFS again (see #871083):
ValueError: versions of pysam.samtools and samtools differ: 1.4.1 != 1.5

Should RC bugs be filed against htslib and samtools to prevent them from
migrating to testing?

Reply | Threaded
Open this post in threaded view
|

Re: [Help] Failed to upgrade bcftools to version 1.5

Afif Elghraoui-6
Hi, Graham,

[adding ticket log to Cc]

على الإثنين  7 آب 2017 ‫10:40، كتب Graham Inggs:
> Python-pysam 0.11.2.2+ds-3 migrated to testing on 2017-08-06, completing
> the htslib/samtools/bcftools 1.4.1 transition.
>
> Now htslib 1.5-1 and samtools 1.5-1 have been uploaded and python-pysam
> FTBFS again (see #871083):
> ValueError: versions of pysam.samtools and samtools differ: 1.4.1 != 1.5
>
> Should RC bugs be filed against htslib and samtools to prevent them from
> migrating to testing?

samtools/bcftools/htslib/pysam generally need to be in sync, regardless
of soname changes. I'm currently preparing bcftools 1.5, but there's
still no corresponding pysam release.

Maybe RC bugs are a good idea. They probably should have been uploaded
to experimental to begin with until pysam caught up. I'll upload
bcftools there once I'm done barring any objection.

Thanks and regards
Afif

--
Afif Elghraoui | عفيف الغراوي
http://afif.ghraoui.name

Reply | Threaded
Open this post in threaded view
|

Re: [Help] Failed to upgrade bcftools to version 1.5

Andreas Tille-5
Hi,

On Mon, Aug 07, 2017 at 11:05:22AM -0400, Afif Elghraoui wrote:
>
> Maybe RC bugs are a good idea. They probably should have been uploaded
> to experimental

Sorry for not beeing less carefully when I gave the green light to
Matthias Klumpp for uploading his fix.  I agree it should have been
rather to experimental.  Do you think it would be sensible to file
some RC bug against it to not enter testing?

Kind regards

         Andreas.

--
http://fam-tille.de

Reply | Threaded
Open this post in threaded view
|

Re: [Help] Failed to upgrade bcftools to version 1.5

Graham Inggs-3
On 07/08/2017 17:49, Andreas Tille wrote:
> Do you think it would be sensible to file
> some RC bug against it to not enter testing?

I cloned #871083 against python-pysam to #871314 and #871315 against
htslib and samtools respectively.