Bug#919778: mash FTBFS on armhf when built on arm64 hardware

classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

Bug#919778: mash FTBFS on armhf when built on arm64 hardware

Adrian Bunk-3
Source: mash
Version: 2.1-2
Severity: serious
Tags: ftbfs

https://buildd.debian.org/status/fetch.php?pkg=mash&arch=armhf&ver=2.1-2&stamp=1545838322&raw=0

...
   dh_auto_test -a
        make -j8 test VERBOSE=1
make[1]: Entering directory '/<<PKGBUILDDIR>>'
cd test ; ../mash sketch -o genomes.msh genome1.fna genome2.fna genome3.fna
cd test ; ../mash sketch -r -I reads reads1.fastq reads2.fastq -o reads.msh
Sketching genome1.fna...
Sketching genome2.fna...
Sketching genome3.fna...
Bus error
make[1]: *** [Makefile:94: test/reads.msh] Error 135


Backtrace:

#0  0x00638d2e in MurmurHash3_x64_128 (key=0xf523a009, len=len@entry=21, seed=<optimized out>, out=out@entry=0xf6b2dbd4)
    at src/mash/MurmurHash3.cpp:277
#1  0x00635b4c in getHash (seq=<optimized out>, length=length@entry=21, seed=<optimized out>, use64=true)
    at src/mash/hash.cpp:23
#2  0x00639a50 in addMinHashes (minHashHeap=...,
    seq=0xf523a008 "AGCTTTTCATTCTGACTGCAACGGGCAATATGTCTCTGTGTGGATTAAAAAAAGAGTGTCTGATAGCAGCTTCTGAACTGGTTACCTGCCGTGAGTAAATTAAAATTTTATTGACTTAGGTCACTAAATACTTTAACCAATATAGGCATAGCGCACAGACAGATAAAAATTACAGAGTACACAACATCCATGAAACGCAT"...,
    length=<optimized out>, parameters=...) at src/mash/Sketch.cpp:601
#3  0x0063a7d6 in sketchFile (input=0x692268) at src/mash/Sketch.cpp:1264
#4  0x006400a0 in ThreadPool<Sketch::SketchInput, Sketch::SketchOutput>::thread (arg=0xffeea370)
    at src/mash/ThreadPool.hxx:182
#5  0xf6cd2bbe in start_thread () from /lib/arm-linux-gnueabihf/libpthread.so.0
#6  0xf6bc94dc in ?? () from /lib/arm-linux-gnueabihf/libc.so.6



void MurmurHash3_x64_128 ( const void * key, const int len,
                           const uint32_t seed, void * out )
{
  const uint8_t * data = (const uint8_t*)key;
...
  const uint64_t * blocks = (const uint64_t *)(data);
...


key=0xf523a009 is not properly aligned for that.

Reply | Threaded
Open this post in threaded view
|

Bug#919778: [Debian-med-packaging] Bug#919778: mash FTBFS on armhf when built on arm64 hardware

Sascha Steinbiss-5
tags 919778 help
forwarded 919778 https://github.com/marbl/Mash/issues/108
thanks

Hi,

thanks for reporting this issue!
I have forwarded this upstream but I'm afraid I won't be able to do much
about it in the near future. With the upcoming freeze in mind, and given
the fact that upstream doesn't seem to be too active about our recent
bug reports, TBH I would be tempted to remove support for the
architectures with missing builds from the mash package and file RM for
the old binary packages until there is a solution.
Some other packages depend on mash so I would like to see it in buster,
since (as always, in my own humble opinion) the problematic
architectures (arm, s390, mips) are not within the typical target
audience of mash.

Also tagging this as help -- so if there's anyone with expertise in this
kind of porting work, I would surely appreciate help :)
BTW Fabian, did you get any further with #918566 as this may be related?

Cheers
Sascha


On 19.01.19 15:42, Adrian Bunk wrote:

> Source: mash
> Version: 2.1-2
> Severity: serious
> Tags: ftbfs
>
> https://buildd.debian.org/status/fetch.php?pkg=mash&arch=armhf&ver=2.1-2&stamp=1545838322&raw=0
>
> ...
>    dh_auto_test -a
> make -j8 test VERBOSE=1
> make[1]: Entering directory '/<<PKGBUILDDIR>>'
> cd test ; ../mash sketch -o genomes.msh genome1.fna genome2.fna genome3.fna
> cd test ; ../mash sketch -r -I reads reads1.fastq reads2.fastq -o reads.msh
> Sketching genome1.fna...
> Sketching genome2.fna...
> Sketching genome3.fna...
> Bus error
> make[1]: *** [Makefile:94: test/reads.msh] Error 135
>
>
> Backtrace:
>
> #0  0x00638d2e in MurmurHash3_x64_128 (key=0xf523a009, len=len@entry=21, seed=<optimized out>, out=out@entry=0xf6b2dbd4)
>     at src/mash/MurmurHash3.cpp:277
> #1  0x00635b4c in getHash (seq=<optimized out>, length=length@entry=21, seed=<optimized out>, use64=true)
>     at src/mash/hash.cpp:23
> #2  0x00639a50 in addMinHashes (minHashHeap=...,
>     seq=0xf523a008 "AGCTTTTCATTCTGACTGCAACGGGCAATATGTCTCTGTGTGGATTAAAAAAAGAGTGTCTGATAGCAGCTTCTGAACTGGTTACCTGCCGTGAGTAAATTAAAATTTTATTGACTTAGGTCACTAAATACTTTAACCAATATAGGCATAGCGCACAGACAGATAAAAATTACAGAGTACACAACATCCATGAAACGCAT"...,
>     length=<optimized out>, parameters=...) at src/mash/Sketch.cpp:601
> #3  0x0063a7d6 in sketchFile (input=0x692268) at src/mash/Sketch.cpp:1264
> #4  0x006400a0 in ThreadPool<Sketch::SketchInput, Sketch::SketchOutput>::thread (arg=0xffeea370)
>     at src/mash/ThreadPool.hxx:182
> #5  0xf6cd2bbe in start_thread () from /lib/arm-linux-gnueabihf/libpthread.so.0
> #6  0xf6bc94dc in ?? () from /lib/arm-linux-gnueabihf/libc.so.6
>
>
>
> void MurmurHash3_x64_128 ( const void * key, const int len,
>                            const uint32_t seed, void * out )
> {
>   const uint8_t * data = (const uint8_t*)key;
> ...
>   const uint64_t * blocks = (const uint64_t *)(data);
> ...
>
>
> key=0xf523a009 is not properly aligned for that.
>
> _______________________________________________
> Debian-med-packaging mailing list
> [hidden email]
> https://alioth-lists.debian.net/cgi-bin/mailman/listinfo/debian-med-packaging
>


signature.asc (499 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Bug#919778: [Debian-med-packaging] Bug#919778: Bug#919778: mash FTBFS on armhf when built on arm64 hardware

Graham Inggs-3
Ubuntu has a patch that might help.

https://patches.ubuntu.com/m/mash/mash_2.0-2ubuntu2.patch

Reply | Threaded
Open this post in threaded view
|

Bug#919778: Bug#918566: Lost of code copies of MurmurHash3 (Was: Bug#918566: mash FTBFS on big endian: test failures)

Fabian Klötzl
In reply to this post by Adrian Bunk-3

On 07.02.19 14:58, Andreas Tille wrote:

>
> Unfortunately the -DARCH32 flag (which is set according to the build log[1]) does
> not help on i386 architecture:
>
>
> ./mash info -d test/reads.msh > test/reads.json
> diff test/genomes.json test/ref/genomes.json
> 7c7
> <  "hashType" : "MurmurHash3_x86_32",
> ---
>>   "hashType" : "MurmurHash3_x64_128",

mash uses two different hash functions depending on the architecture:
MurmurHash3_x86_32 for 32 bit and MurmurHash3_x64_128 for 64 bit
architectures; These produce different hash values. The tests compare
the results of one run with a precomputed result. However, the tests
check against the 64bit reference. Thus they are bound to fail on 32bit.
Hence my suggestion to turn the tests off for 32bit.


> There are also build issues of libmurmurhash on some architectures[2]
>
This is curious. One of the three hash functions is wrong, but at least
consistently wrong. I'll look into that.

Fabian

Reply | Threaded
Open this post in threaded view
|

Bug#919778: Bug#918566: Lost of code copies of MurmurHash3 (Was: Bug#918566: mash FTBFS on big endian: test failures)

Fabian Klötzl
In reply to this post by Adrian Bunk-3
Hi,

On 07.02.19 16:00, Andreas Tille wrote:
> I've obviously missed that part of your mail.  It is fixed now and I'll
> re-upload.

No worries. The alternative would have been to create a separate set of
unit tests for the 32bit architectures.

>
> Meanwhile I've fixed the symbols file in Git.  Please git pull.
>

While your at it, could you also push the tags? Thanks in advance
Fabian