Bug#960756: python-biopython FTBFS on 32bit: test_NCBI_BLAST_tools.BlastDB failures

classic Classic list List threaded Threaded
8 messages Options
Reply | Threaded
Open this post in threaded view
|

Bug#960756: python-biopython FTBFS on 32bit: test_NCBI_BLAST_tools.BlastDB failures

Adrian Bunk-3
Source: python-biopython
Version: 1.76+dfsg-2
Severity: serious
Tags: ftbfs

https://buildd.debian.org/status/package.php?p=python-biopython&suite=sid

...
======================================================================
FAIL: test_fasta_db_nucl (test_NCBI_BLAST_tools.BlastDB)
Test makeblastdb wrapper with nucleotide database.
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/<<PKGBUILDDIR>>/.pybuild/cpython3_3.8/build/Tests/test_NCBI_BLAST_tools.py", line 249, in test_fasta_db_nucl
    self.assertTrue(os.path.isfile("GenBank/NC_005816.fna.nhd"))
AssertionError: False is not true

======================================================================
FAIL: test_fasta_db_prot (test_NCBI_BLAST_tools.BlastDB)
Test makeblastdb wrapper with protein database.
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/<<PKGBUILDDIR>>/.pybuild/cpython3_3.8/build/Tests/test_NCBI_BLAST_tools.py", line 208, in test_fasta_db_prot
    self.assertTrue(os.path.isfile("GenBank/NC_005816.faa.phd"))
AssertionError: False is not true

----------------------------------------------------------------------
Ran 518 tests in 330.272 seconds

FAILED (failures = 1)
Skipping any tests requiring internet access
Python version: 3.8.3 (default, May 14 2020, 11:03:12)
[GCC 9.3.0]
Operating system: posix linux
E: pybuild pybuild:352: test: plugin custom failed with: exit code=1: set -e; \
                             mkdir -p /<<PKGBUILDDIR>>/.pybuild/cpython3_3.8/build/home; \
                             mkdir -p /<<PKGBUILDDIR>>/.pybuild/cpython3_3.8/build/Doc/examples; \
                             cp -a Doc/Tutorial.tex /<<PKGBUILDDIR>>/.pybuild/cpython3_3.8/build/Doc; \
                             cp -a Doc/Tutorial /<<PKGBUILDDIR>>/.pybuild/cpython3_3.8/build/Doc; \
                             cp -a Doc/examples /<<PKGBUILDDIR>>/.pybuild/cpython3_3.8/build/Doc; \
                             cp -a Tests /<<PKGBUILDDIR>>/.pybuild/cpython3_3.8/build; \
                             cd /<<PKGBUILDDIR>>/.pybuild/cpython3_3.8/build/Tests; \
                             env DIALIGN2_DIR=/usr/share/dialign EMBOSS_ROOT=/usr/lib/emboss HOME=/<<PKGBUILDDIR>>/.pybuild/cpython3_3.8/build/home python3.8 run_tests.py --offline
dh_auto_test: error: pybuild --test -i python{version} -p 3.8 --test --system=custom "--test-args=set -e; \\\
                             mkdir -p {build_dir}/home; \\\
                             mkdir -p {build_dir}/Doc/examples; \\\
                             cp -a Doc/Tutorial.tex {build_dir}/Doc; \\\
                             cp -a Doc/Tutorial {build_dir}/Doc; \\\
                             cp -a Doc/examples {build_dir}/Doc; \\\
                             cp -a Tests {build_dir}; \\\
                             cd {build_dir}/Tests; \\\
                             env DIALIGN2_DIR=/usr/share/dialign EMBOSS_ROOT=/usr/lib/emboss HOME={build_dir}/home {interpreter} run_tests.py --offline" returned exit code 13
make[1]: *** [debian/rules:83: override_dh_auto_test] Error 25

Reply | Threaded
Open this post in threaded view
|

Bug#960756: python-biopython FTBFS on 32bit: test_NCBI_BLAST_tools.BlastDB failures

Andreas Tille-5
Control: tags -1 upstream
Control: forwarded -1 Peter Cock <[hidden email]>

Hi Peter,

it seems the patch applied does not work for 32bit architectures.

Kind regards

     Andreas.

On Sat, May 16, 2020 at 02:25:20PM +0300, Adrian Bunk wrote:

> Source: python-biopython
> Version: 1.76+dfsg-2
> Severity: serious
> Tags: ftbfs
>
> https://buildd.debian.org/status/package.php?p=python-biopython&suite=sid
>
> ...
> ======================================================================
> FAIL: test_fasta_db_nucl (test_NCBI_BLAST_tools.BlastDB)
> Test makeblastdb wrapper with nucleotide database.
> ----------------------------------------------------------------------
> Traceback (most recent call last):
>   File "/<<PKGBUILDDIR>>/.pybuild/cpython3_3.8/build/Tests/test_NCBI_BLAST_tools.py", line 249, in test_fasta_db_nucl
>     self.assertTrue(os.path.isfile("GenBank/NC_005816.fna.nhd"))
> AssertionError: False is not true
>
> ======================================================================
> FAIL: test_fasta_db_prot (test_NCBI_BLAST_tools.BlastDB)
> Test makeblastdb wrapper with protein database.
> ----------------------------------------------------------------------
> Traceback (most recent call last):
>   File "/<<PKGBUILDDIR>>/.pybuild/cpython3_3.8/build/Tests/test_NCBI_BLAST_tools.py", line 208, in test_fasta_db_prot
>     self.assertTrue(os.path.isfile("GenBank/NC_005816.faa.phd"))
> AssertionError: False is not true
>
> ----------------------------------------------------------------------
> Ran 518 tests in 330.272 seconds
>
> FAILED (failures = 1)
> Skipping any tests requiring internet access
> Python version: 3.8.3 (default, May 14 2020, 11:03:12)
> [GCC 9.3.0]
> Operating system: posix linux
> E: pybuild pybuild:352: test: plugin custom failed with: exit code=1: set -e; \
>                              mkdir -p /<<PKGBUILDDIR>>/.pybuild/cpython3_3.8/build/home; \
>                              mkdir -p /<<PKGBUILDDIR>>/.pybuild/cpython3_3.8/build/Doc/examples; \
>                              cp -a Doc/Tutorial.tex /<<PKGBUILDDIR>>/.pybuild/cpython3_3.8/build/Doc; \
>                              cp -a Doc/Tutorial /<<PKGBUILDDIR>>/.pybuild/cpython3_3.8/build/Doc; \
>                              cp -a Doc/examples /<<PKGBUILDDIR>>/.pybuild/cpython3_3.8/build/Doc; \
>                              cp -a Tests /<<PKGBUILDDIR>>/.pybuild/cpython3_3.8/build; \
>                              cd /<<PKGBUILDDIR>>/.pybuild/cpython3_3.8/build/Tests; \
>                              env DIALIGN2_DIR=/usr/share/dialign EMBOSS_ROOT=/usr/lib/emboss HOME=/<<PKGBUILDDIR>>/.pybuild/cpython3_3.8/build/home python3.8 run_tests.py --offline
> dh_auto_test: error: pybuild --test -i python{version} -p 3.8 --test --system=custom "--test-args=set -e; \\\
>                              mkdir -p {build_dir}/home; \\\
>                              mkdir -p {build_dir}/Doc/examples; \\\
>                              cp -a Doc/Tutorial.tex {build_dir}/Doc; \\\
>                              cp -a Doc/Tutorial {build_dir}/Doc; \\\
>                              cp -a Doc/examples {build_dir}/Doc; \\\
>                              cp -a Tests {build_dir}; \\\
>                              cd {build_dir}/Tests; \\\
>                              env DIALIGN2_DIR=/usr/share/dialign EMBOSS_ROOT=/usr/lib/emboss HOME={build_dir}/home {interpreter} run_tests.py --offline" returned exit code 13
> make[1]: *** [debian/rules:83: override_dh_auto_test] Error 25
>
> _______________________________________________
> Debian-med-packaging mailing list
> [hidden email]
> https://alioth-lists.debian.net/cgi-bin/mailman/listinfo/debian-med-packaging

--
http://fam-tille.de

Reply | Threaded
Open this post in threaded view
|

Bug#960756: python-biopython FTBFS on 32bit: test_NCBI_BLAST_tools.BlastDB failures

Peter Cock
Perhaps makeblastdb itself failed (and our wrapper didn't notice)? Those are the first files looked for after calling makeblastdb, to see if it could make a BLAST database.  Are there any GenBank/NC_005816.fna.n* or GenBank/NC_005816.faa.p* files present?

If it helps, the commands our script was trying to run were:

$ makeblastdb -dbtype nucl -in GenBank/NC_005816.fna \
-parse_seqids -hash_index -max_file_sz 20MB  -taxid 10

and:

$ makeblastdb -dbtype prot -in GenBank/NC_005816.faa \
-parse_seqids -hash_index -max_file_sz 20MB -taxid 10

I'm not sure I have remote access to any 32bit machines right now...

Peter

On Sat, May 16, 2020 at 2:22 PM Andreas Tille <[hidden email]> wrote:
Control: tags -1 upstream
Control: forwarded -1 Peter Cock <[hidden email]>

Hi Peter,

it seems the patch applied does not work for 32bit architectures.

Kind regards

     Andreas.

On Sat, May 16, 2020 at 02:25:20PM +0300, Adrian Bunk wrote:
> Source: python-biopython
> Version: 1.76+dfsg-2
> Severity: serious
> Tags: ftbfs
>
> https://buildd.debian.org/status/package.php?p=python-biopython&suite=sid
>
> ...
> ======================================================================
> FAIL: test_fasta_db_nucl (test_NCBI_BLAST_tools.BlastDB)
> Test makeblastdb wrapper with nucleotide database.
> ----------------------------------------------------------------------
> Traceback (most recent call last):
>   File "/<<PKGBUILDDIR>>/.pybuild/cpython3_3.8/build/Tests/test_NCBI_BLAST_tools.py", line 249, in test_fasta_db_nucl
>     self.assertTrue(os.path.isfile("GenBank/NC_005816.fna.nhd"))
> AssertionError: False is not true
>
> ======================================================================
> FAIL: test_fasta_db_prot (test_NCBI_BLAST_tools.BlastDB)
> Test makeblastdb wrapper with protein database.
> ----------------------------------------------------------------------
> Traceback (most recent call last):
>   File "/<<PKGBUILDDIR>>/.pybuild/cpython3_3.8/build/Tests/test_NCBI_BLAST_tools.py", line 208, in test_fasta_db_prot
>     self.assertTrue(os.path.isfile("GenBank/NC_005816.faa.phd"))
> AssertionError: False is not true
>
> ----------------------------------------------------------------------
> Ran 518 tests in 330.272 seconds
>
> FAILED (failures = 1)
> Skipping any tests requiring internet access
> Python version: 3.8.3 (default, May 14 2020, 11:03:12)
> [GCC 9.3.0]
> Operating system: posix linux
> E: pybuild pybuild:352: test: plugin custom failed with: exit code=1: set -e; \
>                              mkdir -p /<<PKGBUILDDIR>>/.pybuild/cpython3_3.8/build/home; \
>                              mkdir -p /<<PKGBUILDDIR>>/.pybuild/cpython3_3.8/build/Doc/examples; \
>                              cp -a Doc/Tutorial.tex /<<PKGBUILDDIR>>/.pybuild/cpython3_3.8/build/Doc; \
>                              cp -a Doc/Tutorial /<<PKGBUILDDIR>>/.pybuild/cpython3_3.8/build/Doc; \
>                              cp -a Doc/examples /<<PKGBUILDDIR>>/.pybuild/cpython3_3.8/build/Doc; \
>                              cp -a Tests /<<PKGBUILDDIR>>/.pybuild/cpython3_3.8/build; \
>                              cd /<<PKGBUILDDIR>>/.pybuild/cpython3_3.8/build/Tests; \
>                              env DIALIGN2_DIR=/usr/share/dialign EMBOSS_ROOT=/usr/lib/emboss HOME=/<<PKGBUILDDIR>>/.pybuild/cpython3_3.8/build/home python3.8 run_tests.py --offline
> dh_auto_test: error: pybuild --test -i python{version} -p 3.8 --test --system=custom "--test-args=set -e; \\\
>                              mkdir -p {build_dir}/home; \\\
>                              mkdir -p {build_dir}/Doc/examples; \\\
>                              cp -a Doc/Tutorial.tex {build_dir}/Doc; \\\
>                              cp -a Doc/Tutorial {build_dir}/Doc; \\\
>                              cp -a Doc/examples {build_dir}/Doc; \\\
>                              cp -a Tests {build_dir}; \\\
>                              cd {build_dir}/Tests; \\\
>                              env DIALIGN2_DIR=/usr/share/dialign EMBOSS_ROOT=/usr/lib/emboss HOME={build_dir}/home {interpreter} run_tests.py --offline" returned exit code 13
> make[1]: *** [debian/rules:83: override_dh_auto_test] Error 25
>
> _______________________________________________
> Debian-med-packaging mailing list
> [hidden email]
> https://alioth-lists.debian.net/cgi-bin/mailman/listinfo/debian-med-packaging

--
http://fam-tille.de
Reply | Threaded
Open this post in threaded view
|

Bug#960756: python-biopython FTBFS on 32bit: test_NCBI_BLAST_tools.BlastDB failures

Andreas Tille-5
In reply to this post by Adrian Bunk-3
Hi,

any voluntee to follow this hint of upstream?

> Perhaps makeblastdb itself failed (and our wrapper didn't notice)? Those
> are the first files looked for after calling makeblastdb, to see if it
> could make a BLAST database.  Are there any GenBank/NC_005816.fna.n* or
> GenBank/NC_005816.faa.p* files present?
>
> If it helps, the commands our script was trying to run were:
>
> $ makeblastdb -dbtype nucl -in GenBank/NC_005816.fna \
> -parse_seqids -hash_index -max_file_sz 20MB  -taxid 10
>
> and:
>
> $ makeblastdb -dbtype prot -in GenBank/NC_005816.faa \
> -parse_seqids -hash_index -max_file_sz 20MB -taxid 10
>
> I'm not sure I have remote access to any 32bit machines right now...

Kind regards

       Andreas.

--
http://fam-tille.de

Reply | Threaded
Open this post in threaded view
|

Bug#960756: [Debian-med-packaging] Bug#960756: python-biopython FTBFS on 32bit: test_NCBI_BLAST_tools.BlastDB failures

Andreas Tille-2
In reply to this post by Adrian Bunk-3
Hi Étienne,

I'm just including the Uploaders of ncbi-blast+ into this conversation
to make sure the information reaches the experts.  We should probably
reassign the bug but I'll leave this to those who know better.

Thanks a lot for your analysis

      Andreas.

On Tue, Jun 09, 2020 at 02:25:58PM +0200, Étienne Mollier wrote:

> Hi all,
>
> Andreas Tille, on 2020-06-08 16:01:33 +0200:
> > any voluntee to follow this hint of upstream?
>
> Having a look a this issue, here is what I can tell so far.
>
> > > Perhaps makeblastdb itself failed (and our wrapper didn't notice)? Those
> > > are the first files looked for after calling makeblastdb, to see if it
> > > could make a BLAST database.  Are there any GenBank/NC_005816.fna.n* or
> > > GenBank/NC_005816.faa.p* files present?
> > >
> > > If it helps, the commands our script was trying to run were:
> > >
> > > $ makeblastdb -dbtype nucl -in GenBank/NC_005816.fna \
> > > -parse_seqids -hash_index -max_file_sz 20MB  -taxid 10
> > >
> > > and:
> > >
> > > $ makeblastdb -dbtype prot -in GenBank/NC_005816.faa \
> > > -parse_seqids -hash_index -max_file_sz 20MB -taxid 10
>
> On my i686 machine, both of these commands end up in error,
> failing to allocate memory:
>
> $ makeblastdb -dbtype nucl -in GenBank/NC_005816.fna -parse_seqids -hash_index -max_file_sz 20MB  -taxid 10
>
>
> Building a new DB, current time: 06/09/2020 08:28:08
> New DB name:   /tmp/python-biopthon/Tests/GenBank/NC_005816.fna
> New DB title:  GenBank/NC_005816.fna
> Sequence type: Nucleotide
> Deleted existing Nucleotide BLAST database named /tmp/python-biopthon/Tests/GenBank/NC_005816.fna
> Keep MBits: T
> Maximum file size: 20000000B
> Adding sequences from FASTA; added 1 sequences in 0.284663 seconds.
>
> No volumes were created.
>
> BLAST Database creation error: mdb_env_open: Cannot allocate memory
>
> Looking up the strace to see what happens exactly from a kernel
> point of view, the program attempts to map 3647256576 bytes of
> memory in which the stub of database will be built:
>
> lstat64("/tmp/python-biopthon/Tests/GenBank/NC_005816.fna.ndb", 0xbfc2e5ac) = -1 ENOENT (No such file or directory)
> openat(AT_FDCWD, "/tmp/python-biopthon/Tests/GenBank/NC_005816.fna.ndb", O_RDWR|O_CREAT, 0664) = 4
> fstatfs(4, {f_type=XFS_SB_MAGIC, f_bsize=4096, f_blocks=73645943, f_bfree=64080178, f_bavail=64080178, f_files=147363840, f_ffree=147171712, f_fsid={val=[65027, 0]}, f_namelen=255, f_frsize=4096, f_flags=ST_VALID|ST_NOATIME}) = 0
> pread64(4, "", 92, 0)                   = 0
> pwrite64(4, "\0\0\0\0\0\0\10\0\0\0\0\0\336\300\357\276\1\0\0\0\0\0\0\0\0\270d\331\0\20\0\0"..., 8192, 0) = 8192
> mmap2(NULL, 3647256576, PROT_READ, MAP_SHARED, 4, 0) = -1 ENOMEM (Cannot allocate memory)
>            ~~~~~~~~~~
>
> To rule out a few issues that could have caused more or less
> artificial memory starvation situations, I tried to bring the
> following changes to my configuration:
>
>   - append an additional 4 GiB of swap through a file;
>
>   - move to a PAE aware kernel since my original configuration
>     had no use for virtual memory extension past the 3 GiB limit
>     anyway:
> $ uname -sr
> Linux 4.19.0-9-686-pae
> $ grep PAE /boot/config-`uname -r`
> CONFIG_X86_PAE=y
>
>   - check RLIMIT_DATA to make sure they were not blocking:
> $ prlimit   # filtered
> AS         address space limit unlimited unlimited bytes
> DATA       max data size       unlimited unlimited bytes
>
>   - increase the vm.max_map_count by two orders of magnitude
>     compared to the default (65536), just in case:
> $ cat /proc/sys/vm/max_map_count
> 1000000
>
>   - enable memory overcommit and allow unreasonable levels of
>     commit ratios:
> $ grep . /proc/sys/vm/overcommit_*
> /proc/sys/vm/overcommit_kbytes:0
> /proc/sys/vm/overcommit_memory:1
> /proc/sys/vm/overcommit_ratio:200
>     but that shouldn't be important given the fact that in such
>     mmap configuration, the memory does not need to be
>     committed anyway, that was just to rule out that point too.
>
> For comparison, on 64 bits systems, the size of the mmap is of
> precisely 300 GB, and the command works very well whatever the
> actual size of physical memory is available on the host.
>
> My current impression is that makeblastdb is unable to work
> properly on most 32 bits machines, because the amount of memory
> needing to be addressed by the process looks like it might
> exceed too easily 32 bits architectural limits.
>
> Have a nice day,
> --
> Étienne Mollier <[hidden email]>
> Fingerprint:  5ab1 4edf 63bb ccff 8b54  2fa9 59da 56fe fff3 882d
> Help find cures against the Covid-19 !  Give CPU cycles:
>   * Rosetta@home: https://boinc.bakerlab.org/rosetta/
>   * Folding@home: https://foldingathome.org/



--
http://fam-tille.de

Reply | Threaded
Open this post in threaded view
|

Bug#960756: [Debian-med-packaging] Bug#960756: python-biopython FTBFS on 32bit: test_NCBI_BLAST_tools.BlastDB failures

Aaron M. Ucko
Andreas Tille <[hidden email]> writes:

> On Tue, Jun 09, 2020 at 02:25:58PM +0200, Étienne Mollier wrote:
>> My current impression is that makeblastdb is unable to work
>> properly on most 32 bits machines, because the amount of memory
>> needing to be addressed by the process looks like it might
>> exceed too easily 32 bits architectural limits.

That's entirely plausible; upstream tends to assume 64-bit systems
nowadays.  Explicitly supplying -blastdb_version 4 (the default prior to
BLAST+ 2.10.x) may help.

Thanks for calling this report to my attention!

--
Aaron M. Ucko, KB1CJC (amu at alum.mit.edu, ucko at debian.org)
http://www.mit.edu/~amu/ | http://stuff.mit.edu/cgi/finger/?amu@...

Reply | Threaded
Open this post in threaded view
|

Bug#960756: [Debian-med-packaging] Bug#960756: python-biopython FTBFS on 32bit: test_NCBI_BLAST_tools.BlastDB failures

olivier sallou-2


Le mar. 9 juin 2020 à 16:47, Aaron M. Ucko <[hidden email]> a écrit :
Andreas Tille <[hidden email]> writes:

> On Tue, Jun 09, 2020 at 02:25:58PM +0200, Étienne Mollier wrote:
>> My current impression is that makeblastdb is unable to work
>> properly on most 32 bits machines, because the amount of memory
>> needing to be addressed by the process looks like it might
>> exceed too easily 32 bits architectural limits.

That's entirely plausible; upstream tends to assume 64-bit systems
nowadays.  Explicitly supplying -blastdb_version 4 (the default prior to
BLAST+ 2.10.x) may help.

options:

* simply remove 32bit package from Debian.... (though should work on *small* datasets).
more and more systems are going to 64bit only, and we don't have the task force to fix 32bits issues on such software (too much effort)
OR
* ignore issue as it is related to memory management but will work on  relatively *small* jobs, so software is ~OK 'till a certain point.

Olivier

Thanks for calling this report to my attention!

--
Aaron M. Ucko, KB1CJC (amu at alum.mit.edu, ucko at debian.org)
http://www.mit.edu/~amu/ | http://stuff.mit.edu/cgi/finger/?amu@...


--
gpg key id: 4096R/326D8438  (keyring.debian.org)
Key fingerprint = 5FB4 6F83 D3B9 5204 6335  D26D 78DC 68DB 326D 8438
Reply | Threaded
Open this post in threaded view
|

Bug#960756: [Debian-med-packaging] Bug#960756: python-biopython FTBFS on 32bit: test_NCBI_BLAST_tools.BlastDB failures

Andreas Tille-5
On Tue, Jun 09, 2020 at 06:28:05PM +0200, olivier sallou wrote:
> options:
>
> * simply remove 32bit package from Debian.... (though should work on
> *small* datasets).

That would affect a lot of rdepends and thus a lot of work filling
ROM bugs.

> more and more systems are going to 64bit only, and we don't have the task
> force to fix 32bits issues on such software (too much effort)
> OR
> * ignore issue as it is related to memory management but will work on
> relatively *small* jobs, so software is ~OK 'till a certain point.

I agree that its use is definitely limited - but if we can get a simple
workaround (as Aaron suggested - implementation welcome I'm busy with
real life the next couple of days) this would just save work on our side
and might be the prefered solution.

Kind regards

        Andreas.

--
http://fam-tille.de