Bug#915333: git-annex: Illegal Instruction on armel (Fujitsu Q700 like QNAP TS-21x/TS-22x)

classic Classic list List threaded Threaded
13 messages Options
Reply | Threaded
Open this post in threaded view
|

Bug#915333: git-annex: Illegal Instruction on armel (Fujitsu Q700 like QNAP TS-21x/TS-22x)

Lukas Straub
Package: git-annex
Version: 7.20181121-1
Severity: important

Dear Maintainer,
I wanted to use git-annex on this machine (Fujitsu Q700 NAS), but it Crashes with
"Illegal Instruction". I don't have these problems with other Packages or with the
upstream build of git-annex.

Many Thanks for Maintaining the Debian Package.

Regards,
Lukas Straub


/proc/cpuinfo:
processor       : 0
model name      : Feroceon 88FR131 rev 1 (v5l)
BogoMIPS        : 400.00
Features        : swp half thumb fastmult edsp
CPU implementer : 0x56
CPU architecture: 5TE
CPU variant     : 0x2
CPU part        : 0x131
CPU revision    : 1

Hardware        : Marvell Kirkwood (Flattened Device Tree)
Revision        : 0000
Serial          : 0000000000000000

-- System Information:
Debian Release: buster/sid
  APT prefers testing
  APT policy: (500, 'testing')
Architecture: armel (armv5tel)

Kernel: Linux 4.18.0-0.bpo.1-marvell
Locale: LANG=en_US.UTF-8, LC_CTYPE=en_US.UTF-8 (charmap=UTF-8), LANGUAGE=en_US.UTF-8 (charmap=UTF-8)
Shell: /bin/sh linked to /bin/dash
Init: systemd (via /run/systemd/system)

Versions of packages git-annex depends on:
ii  curl            7.62.0-1
ii  git             1:2.19.2-1
ii  libatomic1      8.2.0-9
ii  libc6           2.27-8
ii  libffi6         3.2.1-9
ii  libgmp10        2:6.1.2+dfsg-3
ii  libmagic1       1:5.34-2
ii  libsqlite3-0    3.25.3-2
ii  libxml2         2.9.4+dfsg1-7+b2
ii  netbase         5.5
ii  openssh-client  1:7.9p1-4
ii  rsync           3.1.2-2.2
ii  zlib1g          1:1.2.11.dfsg-1

Versions of packages git-annex recommends:
ii  aria2              1.34.0-3
ii  bind9-host         1:9.11.5+dfsg-1
ii  git-remote-gcrypt  1.1-1
ii  gnupg              2.2.11-1
ii  lsof               4.89+dfsg-0.1
ii  nocache            1.0-1

Versions of packages git-annex suggests:
pn  adb             <none>
pn  bup             <none>
pn  libnss-mdns     <none>
pn  magic-wormhole  <none>
pn  tahoe-lafs      <none>
ii  tor             0.3.4.9-5
pn  uftp            <none>
pn  xdot            <none>
pn  youtube-dl      <none>

-- no debconf information

Reply | Threaded
Open this post in threaded view
|

Bug#915333: Status

Lukas Straub
Hello Everyone,
Any status on this?

Regards,
Lukas Straub

Reply | Threaded
Open this post in threaded view
|

Bug#915333: Status

Sean Whitton
Hello,

On Sat 29 Dec 2018 at 11:10am +0100, Lukas Straub wrote:

> Hello Everyone,
> Any status on this?

Unless someone has exactly the device mentioned in the original report,
it's not reproducible so not fixable, unfortunately.

--
Sean Whitton

signature.asc (847 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Bug#915333: git-annex: Illegal Instruction on armel (Fujitsu Q700 like QNAP TS-21x/TS-22x)

Julian Calaby
In reply to this post by Lukas Straub
Package: git-annex
Version: 7.20190122-1
Followup-For: Bug #915333

Dear Maintainer,

This also happens on Marvel Kirkwood systems, e.g. a GoFlex NAS.

I suspect that this is a code generation problem further upstream and not
necessarily a git-annex specific problem and probably happens on all
sufficiently low-end armel CPUs.

I would the armel buildds might exhibit this issue, so adding some auto package
tests that would be run in that environment should let you know once the bug is
fixed, even if it doesn't help you debug it.

Thanks,

Julian Calaby

Reply | Threaded
Open this post in threaded view
|

Bug#915333: git-annex: Illegal Instruction on armel (Fujitsu Q700 like QNAP TS-21x/TS-22x)

Bernhard Übelacker-3
Hello Everyone,
I own a qnap ts-119pII with a similar cpu.

See attached file with several debugging attempts.

With my limited assembly knowledge I got to this instruction,
where the backtrace command still shows all of the stack:

    (gdb) bt
    #0  0x03718a2c in stg_returnToStackTop$def ()
    #1  0x03700118 in schedule ()
    #2  0x03701670 in scheduleWaitThread ()
    #3  0x037183a8 in ioManagerStart ()
    #4  0x036fe9ac in hs_init_ghc ()
    #5  0x0370df40 in hs_main ()
    #6  0x0000d838 in main (argc=2, argv=0xbefff6c4) at /tmp/ghc23822_0/ghc_295.c:9

Then execution continues in stg_enter_info$def and stg_ap_v_info$def.
In the latter following instruction is reached:

    0x0371ef80 in stg_ap_v_info$def ()
    1: x/i $pc
    => 0x371ef80 <stg_ap_v_info$def+292>:   ldrsh   r2, [r0, #-10]
    (gdb) nexti
    0x0371ef84 in stg_ap_v_info$def ()
    1: x/i $pc
    => 0x371ef84 <stg_ap_v_info$def+296>:   uxth    r1, r2
    (gdb) nexti

    Thread 1 "git-annex" received signal SIGILL, Illegal instruction.
    0x0371ef84 in stg_ap_v_info$def ()
    1: x/i $pc
    => 0x371ef84 <stg_ap_v_info$def+296>:   uxth    r1, r2
    (gdb) bt
    #0  0x0371ef84 in stg_ap_v_info$def ()
    #1  0x03700118 in schedule ()
    Backtrace stopped: previous frame inner to this frame (corrupt stack?)

If I read [1] right, then the UXTH instruction is just supported
on ARMv6 or later.

Kind regards,
Bernhard

[1] http://infocenter.arm.com/help/topic/com.arm.doc.dui0489i/CIHHJCFE.html

debugging.txt (38K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Bug#915333: git-annex: Illegal Instruction on armel (Fujitsu Q700 like QNAP TS-21x/TS-22x)

Julian Calaby
In reply to this post by Lukas Straub
Package: git-annex
Version: 7.20190129-2
Followup-For: Bug #915333

Dear Maintainer,

FYI this still exists in 7.20190129-2

Thanks,

Julian Calaby

Reply | Threaded
Open this post in threaded view
|

Bug#915333: git-annex: Illegal Instruction on armel (Fujitsu Q700 like QNAP TS-21x/TS-22x)

Adrian Bunk-3
In reply to this post by Bernhard Übelacker-3
Control: severity -1 serious
Control: reassign -1 ghc 8.4.4+dfsg1-1
Control: affects -1 git-annex

On Thu, Jan 31, 2019 at 08:12:17PM +0100, Bernhard Übelacker wrote:
> Hello Everyone,
> I own a qnap ts-119pII with a similar cpu.
>
> See attached file with several debugging attempts.

Thanks a lot for this.

>...
> If I read [1] right, then the UXTH instruction is just supported
> on ARMv6 or later.

Looking at the code, the bug seems to be in
https://sources.debian.org/src/ghc/8.4.4+dfsg1-2/debian/patches/llvm-arm-unknown-linux-gnueabi.patch/

ARM1136JF-S is ARM11, which is ARMv6.

arm9e would be correct here instead of arm1136jf-s.

Due to the static-only nature of the ghc ecosystem the fix would then
require a complete rebuild of all Haskell packages on armel, but with
the buildds otherwise mostly idle now this should be finished within
2-3 days.

> Kind regards,
> Bernhard
>
> [1] http://infocenter.arm.com/help/topic/com.arm.doc.dui0489i/CIHHJCFE.html

cu
Adrian

--

       "Is there not promise of rain?" Ling Tan asked suddenly out
        of the darkness. There had been need of rain for many days.
       "Only a promise," Lao Er said.
                                       Pearl S. Buck - Dragon Seed

Reply | Threaded
Open this post in threaded view
|

Bug#915333: git-annex: Illegal Instruction on armel (Fujitsu Q700 like QNAP TS-21x/TS-22x)

Darshaka Pathirana-2
Hey,

On Mon, 11 Mar 2019 12:05:55 +0200 Adrian Bunk <[hidden email]> wrote:
> On Thu, Jan 31, 2019 at 08:12:17PM +0100, Bernhard Übelacker wrote:
> > Hello Everyone,
> > I own a qnap ts-119pII with a similar cpu.
> >
> > See attached file with several debugging attempts.

I took some time and wanted to reproduce and test this issue.

> > If I read [1] right, then the UXTH instruction is just supported
> > on ARMv6 or later.
>
> Looking at the code, the bug seems to be in
> https://sources.debian.org/src/ghc/8.4.4+dfsg1-2/debian/patches/llvm-arm-unknown-linux-gnueabi.patch/
>
> ARM1136JF-S is ARM11, which is ARMv6.
> arm9e would be correct here instead of arm1136jf-s.
>
> Due to the static-only nature of the ghc ecosystem the fix would then
> require a complete rebuild of all Haskell packages on armel, but with
> the buildds otherwise mostly idle now this should be finished within
> 2-3 days.
First I tried to reproduce the crash on one of our porterboxes[4]
`amdahl` (where I could not reproduce the bug, git-annex runs just
fine), but I only much later figured out that it run armv8l:

[4] https://db.debian.org/machines.cgi?host=amdahl

  dpat@amdahl ~ % export sessionid=913555
  dpat@amdahl ~ % schroot -b -c sid_armel-dchroot -n $sessionid
  I: 00check: Untarring chroot environment.  This might take a minute or two.
  I: 99porterbox-extra-sources: o To install build dependencies run
  I: 99porterbox-extra-sources:   dd-schroot-cmd -c 915333 apt-get update
  I: 99porterbox-extra-sources:   followed by build-dep/install as appropriate in the host system.
  I: 99porterbox-extra-sources: o If you started this session with schroot -b, please do not forget to run
  I: 99porterbox-extra-sources:   schroot --end-session -c 915333
  I: 99porterbox-extra-sources:   when you no longer need this environment.
  915333
  schroot -b -c sid_armel-dchroot -n $sessionid  9.08s user 5.17s system 126% cpu 11.293 total
  (sid_armel-dchroot)dpat@amdahl ~ % uname -a
  Linux amdahl 4.9.0-8-arm64 #1 SMP Debian 4.9.144-3.1 (2019-02-19) armv8l GNU/Linux

I also tried on `abel` but its a armv7l.

I then wanted to setup a qemu-arm environment but I am stuck finding out which
machine/cpu option would be correct. I found [1] and [2] which states that the CPU
"Feroceon 88FR131 rev 1 (v5l)" is a actually a Marvell Kirkwood 88F6281.
(Ok, [3] says that qnap ts-11x runs on Kirkwood system-on-chip (SoC) from Marvell).

[1] https://www.chainsawonatireswing.com/2012/01/07/find-out-which-cpu-your-synology-diskstation-uses/
[2] https://www.7-cpu.com/cpu/Kirkwood.html
[3] https://www.debian.org/releases/stable/armel/ch02s01.html.en (see 2.1.4)

So, is it possible to emulate a armv5 and if yes, how? Thanks.

Regards from the Debian BSP201904, Salzburg,
 - Darsha


signature.asc (849 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Bug#915333: git-annex: Illegal Instruction on armel (Fujitsu Q700 like QNAP TS-21x/TS-22x)

Adrian Bunk-3
On Sun, Apr 07, 2019 at 02:01:33PM +0200, Darshaka Pathirana wrote:

> Hey,
>
> On Mon, 11 Mar 2019 12:05:55 +0200 Adrian Bunk <[hidden email]> wrote:
> > On Thu, Jan 31, 2019 at 08:12:17PM +0100, Bernhard Übelacker wrote:
> > > Hello Everyone,
> > > I own a qnap ts-119pII with a similar cpu.
> > >
> > > See attached file with several debugging attempts.
>
> I took some time and wanted to reproduce and test this issue.
>...

What are you planning to do after reproducing it?

Bernhard did debug it back in January,
and I described what line the problem is.

> Regards from the Debian BSP201904, Salzburg,
>  - Darsha

cu
Adrian

--

       "Is there not promise of rain?" Ling Tan asked suddenly out
        of the darkness. There had been need of rain for many days.
       "Only a promise," Lao Er said.
                                       Pearl S. Buck - Dragon Seed

Reply | Threaded
Open this post in threaded view
|

Bug#915333: git-annex: Illegal Instruction on armel (Fujitsu Q700 like QNAP TS-21x/TS-22x)

Antoine Beaupré-2
user [hidden email]
usertags 915333 + bsp-2019-04-ca-toronto
thank you

On 2019-04-09 08:46:10, Adrian Bunk wrote:

> On Sun, Apr 07, 2019 at 02:01:33PM +0200, Darshaka Pathirana wrote:
>> Hey,
>>
>> On Mon, 11 Mar 2019 12:05:55 +0200 Adrian Bunk <[hidden email]> wrote:
>> > On Thu, Jan 31, 2019 at 08:12:17PM +0100, Bernhard Übelacker wrote:
>> > > Hello Everyone,
>> > > I own a qnap ts-119pII with a similar cpu.
>> > >
>> > > See attached file with several debugging attempts.
>>
>> I took some time and wanted to reproduce and test this issue.
>>...
>
> What are you planning to do after reproducing it?
>
> Bernhard did debug it back in January,
> and I described what line the problem is.

... and it looks like it's a bug / limitation in ghc itself...

Has anyone considered forwarding this upstream or looked at whether it's
been reported?

Is this a regression?

From the Toronto BSP's point of view, this is looking more and more like
a "well, this means we won't support that hardware for git-annex" kind
of bug.

A.

--
Never underestimate the bandwidth of a station wagon full of tapes
hurtling down the highway.
                        - Andrew S. Tanenbaum, "Computer Networks"

Reply | Threaded
Open this post in threaded view
|

Bug#915333: git-annex: Illegal Instruction on armel (Fujitsu Q700 like QNAP TS-21x/TS-22x)

Adrian Bunk-3
On Sat, Apr 27, 2019 at 11:14:33AM -0400, Antoine Beaupré wrote:

> user [hidden email]
> usertags 915333 + bsp-2019-04-ca-toronto
> thank you
>
> On 2019-04-09 08:46:10, Adrian Bunk wrote:
> > On Sun, Apr 07, 2019 at 02:01:33PM +0200, Darshaka Pathirana wrote:
> >> Hey,
> >>
> >> On Mon, 11 Mar 2019 12:05:55 +0200 Adrian Bunk <[hidden email]> wrote:
> >> > On Thu, Jan 31, 2019 at 08:12:17PM +0100, Bernhard Übelacker wrote:
> >> > > Hello Everyone,
> >> > > I own a qnap ts-119pII with a similar cpu.
> >> > >
> >> > > See attached file with several debugging attempts.
> >>
> >> I took some time and wanted to reproduce and test this issue.
> >>...
> >
> > What are you planning to do after reproducing it?
> >
> > Bernhard did debug it back in January,
> > and I described what line the problem is.
>
> ... and it looks like it's a bug / limitation in ghc itself...
>
> Has anyone considered forwarding this upstream or looked at whether it's
> been reported?

Why should a bug in a Debian patch be reported upstream?

> Is this a regression?
>...

Yes.

cu
Adrian

--

       "Is there not promise of rain?" Ling Tan asked suddenly out
        of the darkness. There had been need of rain for many days.
       "Only a promise," Lao Er said.
                                       Pearl S. Buck - Dragon Seed

Reply | Threaded
Open this post in threaded view
|

Bug#915333: git-annex: Illegal Instruction on armel (Fujitsu Q700 like QNAP TS-21x/TS-22x)

Adrian Bunk-3
In reply to this post by Lukas Straub
On Sat, Apr 27, 2019 at 09:35:51AM -0700, Sean Whitton wrote:

> Hello,
>
> On Sat 27 Apr 2019 at 11:14AM -04, Antoine Beaupré wrote:
>
> > ... and it looks like it's a bug / limitation in ghc itself...
>
> Right, it has been reassigned to ghc.
>
> > From the Toronto BSP's point of view, this is looking more and more like
> > a "well, this means we won't support that hardware for git-annex" kind
> > of bug.

The "too buggy for shipping in a stable release" kind of bug.

> This would mean reassigning it to git-annex, wouldn't it?

No, there is nothing specific to git-annex here.

It means that likely all Haskell packages are currently miscompiled
and won't work on the armel baseline.

git-annex is just the most popular of these, so several people
running unstable/testing on the armel baseline quickly noticed.

The vast majority of users would only run into these miscompilations
if it doesn't get fixed before buster is released.

> Sean Whitton

cu
Adrian

--

       "Is there not promise of rain?" Ling Tan asked suddenly out
        of the darkness. There had been need of rain for many days.
       "Only a promise," Lao Er said.
                                       Pearl S. Buck - Dragon Seed

Reply | Threaded
Open this post in threaded view
|

Bug#915333: git-annex: Illegal Instruction on armel (Fujitsu Q700 like QNAP TS-21x/TS-22x)

Adrian Bunk-3
In reply to this post by Lukas Straub
On Sun, May 12, 2019 at 03:16:10PM +0300, Ilias Tsitsimpis wrote:

> On Mon, Mar 11, 2019 at 12:05PM, Adrian Bunk wrote:
> > On Thu, Jan 31, 2019 at 08:12:17PM +0100, Bernhard Übelacker wrote:
> > > See attached file with several debugging attempts.
> >
> > Looking at the code, the bug seems to be in
> > https://sources.debian.org/src/ghc/8.4.4+dfsg1-2/debian/patches/llvm-arm-unknown-linux-gnueabi.patch/
>
> Hey all,
>
> Thank you Bernhard and Adrian for debugging this. I have opened an unblock
> request for ghc to see if RT will allow us to fix this for buster.
>
> [#928882] https://bugs.debian.org/928882

Thanks a lot for this.

> Running the `utils/llvm-targets/gen-data-layout.sh` script[1], outputs
> the following triplet for arm-unknown-linux-gnueabi:
>
>   ("arm-unknown-linux-gnueabi", ("e-m:e-p:32:32-i64:64-v128:64:128-a:0:32-n32-S64", "arm7tdmi", "+soft-float -fp-only-sp -d16 -vfp2 -vfp3 -fp16 -vfp4 -fp-armv8 -neon -crypto +strict-align"))
>
> The ARM7TDMI processor core implements the ARM architecture v4T, so it
> looks like the correct value for armel, but it would be great if someone
> from the debian-arm team could verify that the rest of the flags are OK.
>...

For buster the baseline of the Debian armel port was raised
from v4T to v5TE.

v4T is still fine on the armel port (only slightly slower).

For an upstreamable change arm7tdmi might be better since v4T
is the lowest with Thumb (and Thumb interworking) support.

cu
Adrian

--

       "Is there not promise of rain?" Ling Tan asked suddenly out
        of the darkness. There had been need of rain for many days.
       "Only a promise," Lao Er said.
                                       Pearl S. Buck - Dragon Seed