Bug#900821: linux-image-4.9.0-6-amd64: apache reads wrong data over cifs filesystems served by samba

classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

Bug#900821: linux-image-4.9.0-6-amd64: apache reads wrong data over cifs filesystems served by samba

Santiago Garcia Mantinan
Package: src:linux
Version: 4.9.88-1+deb9u1
Severity: important

Dear Maintainer,

I've found that when you mount a filesystem being served by samba on a host
running apache and serve the files on this filesystem over apache, you'll
get garbage mixed with the file content.

This means that you get the right length but the file's content gets
corrupted.

This only happens when serving the files from samba, if you serve them from
Windows the problem doesn't appear.

I have found this problem in a pure Debian stable installation (Stretch),
but I have tested this on a pure testing (Buster) installation with even
worst results, the download breaks and the kernel shows this:

[  649.547840] WARNING: CPU: 6 PID: 1573 at /build/linux-43CEzF/linux-4.16.12/lib/iov_iter.c:695 copy_page_to_iter+0x1dd/0x2f0
[  649.547844] Modules linked in: cmac arc4 md4 nls_utf8 cifs ccm dns_resolver fscache amd64_edac_mod edac_mce_amd radeon ccp rng_core joydev kvm sg evdev ttm k10temp drm_kms_helper serio_raw pcspkr shpchp drm irqbypass i2c_algo_bit hpilo hpwdt ipmi_si ipmi_devintf button ipmi_msghandler ip_tables x_tables autofs4 ext4 crc16 mbcache jbd2 crc32c_generic fscrypto ecb crypto_simd cryptd glue_helper aes_x86_64 hid_generic usbhid hid sd_mod ohci_pci qla2xxx hpsa nvme_fc scsi_transport_fc scsi_transport_sas psmouse uhci_hcd ohci_hcd ehci_pci nvme_fabrics ehci_hcd scsi_mod nvme_core usbcore bnx2 i2c_piix4 usb_common
[  649.547943] CPU: 6 PID: 1573 Comm: wget Tainted: G        W        4.16.0-2-amd64 #1 Debian 4.16.12-1
[  649.547945] Hardware name: HP ProLiant BL465c G6  , BIOS A13 12/08/2009
[  649.547953] RIP: 0010:copy_page_to_iter+0x1dd/0x2f0
[  649.547956] RSP: 0018:ffffad6602defc58 EFLAGS: 00010297
[  649.547960] RAX: 0000000000008000 RBX: ffffd65a085b1000 RCX: 0000000000000003
[  649.547963] RDX: 0000000000008075 RSI: 017fffc000008000 RDI: 00000000085b1000
[  649.547965] RBP: 000000000000148b R08: 0000000000002000 R09: ffff9ca6e457cd24
[  649.547968] R10: ffff9ca6e20df8e8 R11: 000000000000548b R12: ffffad6602defdf0
[  649.547970] R13: 0000000000006bea R14: 0000000000000040 R15: 0000000000000001
[  649.547974] FS:  00007f0978403780(0000) GS:ffff9ca6e7cc0000(0000) knlGS:0000000000000000
[  649.547977] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  649.547980] CR2: 00005592e97b3078 CR3: 0000000223f6c000 CR4: 00000000000006e0
[  649.547983] Call Trace:
[  649.548001]  skb_copy_datagram_iter+0x175/0x280
[  649.548010]  tcp_recvmsg+0x279/0xb90
[  649.548019]  ? set_fd_set+0x38/0x50
[  649.548024]  ? core_sys_select+0x2a4/0x2d0
[  649.548032]  inet_recvmsg+0x58/0xd0
[  649.548038]  sock_read_iter+0x94/0xf0
[  649.548047]  new_sync_read+0xe9/0x140
[  649.548060]  vfs_read+0x89/0x130
[  649.548066]  SyS_read+0x52/0xc0
[  649.548075]  do_syscall_64+0x6c/0x130
[  649.548082]  entry_SYSCALL_64_after_hwframe+0x3d/0xa2
[  649.548089] RIP: 0033:0x7f0976eb7061
[  649.548091] RSP: 002b:00007ffec8800db8 EFLAGS: 00000246 ORIG_RAX: 0000000000000000
[  649.548095] RAX: ffffffffffffffda RBX: 0000000000000003 RCX: 00007f0976eb7061
[  649.548097] RDX: 0000000000002000 RSI: 00005592e97afc70 RDI: 0000000000000003
[  649.548100] RBP: 000000000010113b R08: 00007ffec8800cd0 R09: 00007f0978403780
[  649.548102] R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000002000
[  649.548105] R13: 00005592e97afc70 R14: 0000000000000000 R15: 00005592e97b1c80
[  649.548108] Code: ff ff 48 89 c5 41 83 ae 28 0a 00 00 01 48 83 c4 10 48 89 e8 5b 5d 41 5c 41 5d 41 5e 41 5f c3 0f b6 49 69 48 d3 e0 e9 a6 fe ff ff <0f> 0b 31 ed eb dc 85 c9 0f 84 ad 00 00 00 31 ed eb d0 4d 01 f5
[  649.548180] ---[ end trace 5c988a789d68247f ]---

Doing several md5sums of the files directly on the cifs filesystem will
allways result in the same md5, also doing dd if=file|md5sum, however
wget http://localhost/file -O -|md5sum
will result on a different code each time.

The same tests running the same Stretch machine with Jessie's kernel will
work Ok.

Like I've said I've been able to replicate this on standard Stretch and
Buster configs. These are the steps to replicate...

install:
        apt-get install samba apache2 cifs-utils
add to smb.conf to create a ftp share and then: service smbd reload
[ftp]
   writable = no
   locking = no
   path = /srv/ftp
   public = yes
   browseable = no
generate a file to be served:
        dd if=/dev/zero of=/srv/ftp/100Mzero bs=1024k count=100
mount the share on the web directory to serve it:
        mount.cifs //localhost/ftp /var/www/html/
test the local access of the cifs:
        md5sum /srv/ftp/100Mzero
2f282b84e7e608d5852449ed940bfc51  /srv/ftp/100Mzero
Acces the file over apache:
        wget http://localhost/100Mzero -O - 2>/dev/null|md5sum
2b0ac997ed705924db55cf5f45ad3c88  -

Like I said, changing to a Jessie's kernel this works ok, changing to a
Buster 4.16 kernel or testing on a full Buster setup gives similar problem
but http transfer is interrupted and kernel shows previous message.  Also
serving the file from Windows works ok.  So this is a problem when serving
from Samba (both Stretch or Buster versions) to a Apache ( md5sum or dd of
the file directily over cifs mount works ok).

If you need any other info to replicate don't hesitate to ask.

Thanks in advance.

Regards.

Reply | Threaded
Open this post in threaded view
|

Bug#900821: linux-image-4.9.0-6-amd64: apache reads wrong data over cifs filesystems served by samba

Salvatore Bonaccorso-4
Control: found -1 4.9.110-1
Control: tags -1 + confirmed

Hi

The issue seem to be still present in 4.9.110-1 but I have trouble to
reproduce it on a sid system running 4.17.8-1. So this might give us
some indication on a possible fix.

Regards,
Salvatore

Reply | Threaded
Open this post in threaded view
|

Bug#900821: linux-image-4.9.0-6-amd64: apache reads wrong data over cifs filesystems served by samba

Santiago Garcia Mantinan
In reply to this post by Santiago Garcia Mantinan
Hi!

I have rechecked everything again.

Salvatore, I'm testing on an up to date buster running kernel 4.17.17-1 and
I still see the kernel warning messages and the downloads are breaking and wget
still shows this king of messages:
2018-08-29 13:45:31 (122 MB/s) - Read error at byte 1056768/6538880 (Bad address). Retrying.

So I see no progresses with newer versions or anything like that.

Don't know what are the differences between your setup and mine, maybe it is
the file length?

What seems to work ok is the workaround of setting EnableSendfile to on,
this avoids the original problem I had found on Stretch and also the
problems I later found on buster with the kernel warnings and broken
downloads.

Hope this helps.

Regards.
--
Manty/BestiaTester -> http://manty.net

Reply | Threaded
Open this post in threaded view
|

Bug#900821: linux-image-4.9.0-6-amd64: apache reads wrong data over cifs filesystems served by samba

Karel Kozlík
In reply to this post by Santiago Garcia Mantinan
Hi,

I am facing this problem with debian stable, but kernel from backports:

ii  linux-image-4.19.0-0.bpo.1-amd64-unsigned 4.19.12-1~bpo9+1
       amd64        Linux 4.19 for 64-bit PCs

Linux version 4.19.0-0.bpo.1-amd64 ([hidden email]) (gcc
version 6.3.0 20170516 (Debian 6.3.0-18+deb9u1)) #1 SMP Debian
4.19.12-1~bpo9+1 (2018-12-30)

The problem appear around twice a day while copying data via rsync to
software raid disk array.

Jan 15 22:31:48 kk-router kernel: [ 6406.089208] general protection
fault: 0000 [#1] SMP PTI
Jan 15 22:31:48 kk-router kernel: [ 6406.089330] CPU: 2 PID: 1183 Comm:
sshd Tainted: G            E     4.19.0-0.bpo.1-amd64 #1 Debian
4.19.12-1~bpo9+1
Jan 15 22:31:48 kk-router kernel: [ 6406.089598] Hardware name: Gigabyte
Technology Co., Ltd. Default string/J3455N-D3H, BIOS F2 03/07/2017
Jan 15 22:31:48 kk-router kernel: [ 6406.089751] RIP:
0010:__check_object_size+0x7b/0x1a0
Jan 15 22:31:48 kk-router kernel: [ 6406.089834] Code: 00 00 80 48 2b 15
5d 0e c7 00 48 01 c2 48 c1 ea 0c 48 c1 e2 06 48 03 15 3b 0e c7 00 48 8b
42 08 48 8d 48 ff a8 01 48 0f 45 d1 <48> 8b 4a 08 48 8d 41 ff 83 e1 01
48 0f 44 c2 48 8b 00 f6 c4 01 75
Jan 15 22:31:48 kk-router kernel: [ 6406.090115] RSP:
0018:ffff96f9415dfc58 EFLAGS: 00010202
Jan 15 22:31:48 kk-router kernel: [ 6406.090202] RAX: efffd06949355a01
RBX: ffff896a0d56cc02 RCX: efffd06949355a00
Jan 15 22:31:48 kk-router kernel: [ 6406.090314] RDX: efffd06949355a00
RSI: ffff896a3fff0000 RDI: ffff896a8d56cc02
Jan 15 22:31:48 kk-router kernel: [ 6406.090426] RBP: 00000000000005a8
R08: 00000000000005a8 R09: 00000000000005a8
Jan 15 22:31:48 kk-router kernel: [ 6406.090538] R10: 0000000000000000
R11: 0000000000000000 R12: 0000000000000001
Jan 15 22:31:48 kk-router kernel: [ 6406.090650] R13: ffff896a0d56d1aa
R14: 00000000000005a8 R15: ffff896a0d56cc02
Jan 15 22:31:48 kk-router kernel: [ 6406.090764] FS:
00007ff8efe39d40(0000) GS:ffff896a37b00000(0000) knlGS:0000000000000000
Jan 15 22:31:48 kk-router kernel: [ 6406.090890] CS:  0010 DS: 0000 ES:
0000 CR0: 0000000080050033
Jan 15 22:31:48 kk-router kernel: [ 6406.090982] CR2: 00007fadf5efbfb0
CR3: 0000000265ebc000 CR4: 00000000003406e0
Jan 15 22:31:48 kk-router kernel: [ 6406.091095] Call Trace:
Jan 15 22:31:48 kk-router kernel: [ 6406.091152]
skb_copy_datagram_iter+0x75/0x260
Jan 15 22:31:48 kk-router kernel: [ 6406.091232]  tcp_recvmsg+0x72b/0xca0
Jan 15 22:31:48 kk-router kernel: [ 6406.091300]  ? aa_sk_perm+0x44/0x130
Jan 15 22:31:48 kk-router kernel: [ 6406.091366]  inet_recvmsg+0x5b/0xd0
Jan 15 22:31:48 kk-router kernel: [ 6406.091430]  sock_read_iter+0x94/0xf0
Jan 15 22:31:48 kk-router kernel: [ 6406.091498]  new_sync_read+0xfa/0x160
Jan 15 22:31:48 kk-router kernel: [ 6406.091565]  vfs_read+0x91/0x130
Jan 15 22:31:48 kk-router kernel: [ 6406.091624]  ksys_read+0x52/0xc0
Jan 15 22:31:48 kk-router kernel: [ 6406.091685]  do_syscall_64+0x55/0x110
Jan 15 22:31:48 kk-router kernel: [ 6406.091752]
entry_SYSCALL_64_after_hwframe+0x44/0xa9
Jan 15 22:31:48 kk-router kernel: [ 6406.091837] RIP: 0033:0x7ff8edfad6d0
Jan 15 22:31:48 kk-router kernel: [ 6406.091900] Code: b6 fe ff ff 48 8d
3d 17 be 08 00 48 83 ec 08 e8 06 db 01 00 66 0f 1f 44 00 00 83 3d 39 30
2c 00 00 75 10 b8 00 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 31 c3 48 83
ec 08 e8 de 9b 01 00 48 89 04 24
Jan 15 22:31:48 kk-router kernel: [ 6406.092180] RSP:
002b:00007ffe060775a8 EFLAGS: 00000246 ORIG_RAX: 0000000000000000
Jan 15 22:31:48 kk-router kernel: [ 6406.092301] RAX: ffffffffffffffda
RBX: 0000000000000003 RCX: 00007ff8edfad6d0
Jan 15 22:31:48 kk-router kernel: [ 6406.092413] RDX: 0000000000004000
RSI: 00007ffe060775b0 RDI: 0000000000000003
Jan 15 22:31:48 kk-router kernel: [ 6406.092525] RBP: 00005572caec33e0
R08: 0000000000000000 R09: 0000000000004500
Jan 15 22:31:48 kk-router kernel: [ 6406.092637] R10: 00007ffe0607b530
R11: 0000000000000246 R12: 0000000000000000
Jan 15 22:31:48 kk-router kernel: [ 6406.092748] R13: 00007ffe0607b63f
R14: 00005572c925cb67 R15: 0000000000000003
Jan 15 22:31:48 kk-router kernel: [ 6406.092862] Modules linked in:
snd_hda_codec_hdmi(E) snd_hda_codec_realtek(E) snd_hda_codec_generic(E)
nls_ascii(E) intel_rapl(E) nls_cp437(E) x86_pkg_temp_thermal(E)
intel_powerclamp(E) vfat(E) coretemp(E) fat(E) efi_pstore(E) kvm(E)
irqbypass(E) snd_soc_skl(E) snd_soc_skl_ipc(E) snd_soc_sst_ipc(E)
crct10dif_pclmul(E) snd_soc_sst_dsp(E) snd_hda_ext_core(E) ppdev(E)
snd_soc_acpi_intel_match(E) snd_soc_acpi(E) crc32_pclmul(E) i915(E)
snd_soc_core(E) snd_compress(E) snd_hda_intel(E) ghash_clmulni_intel(E)
intel_cstate(E) drm_kms_helper(E) intel_rapl_perf(E) snd_hda_codec(E)
drm(E) evdev(E) snd_hda_core(E) i2c_algo_bit(E) snd_hwdep(E) efivars(E)
pcspkr(E) snd_pcm(E) lpc_ich(E) snd_timer(E) snd(E) mei_me(E)
soundcore(E) mei(E) sg(E) button(E) parport_pc(E) parport(E) video(E)
pcc_cpufreq(E) nfsd(E) auth_rpcgss(E)
Jan 15 22:31:48 kk-router kernel: [ 6406.094025]  nfs_acl(E) lockd(E)
grace(E) sunrpc(E) efivarfs(E) ip_tables(E) x_tables(E) autofs4(E)
ext4(E) crc16(E) mbcache(E) jbd2(E) fscrypto(E) ecb(E) raid10(E)
raid1(E) raid0(E) multipath(E) linear(E) raid456(E) async_raid6_recov(E)
async_memcpy(E) async_pq(E) async_xor(E) async_tx(E) xor(E) raid6_pq(E)
libcrc32c(E) crc32c_generic(E) md_mod(E) hid_generic(E) usbhid(E) hid(E)
sd_mod(E) crc32c_intel(E) xhci_pci(E) aesni_intel(E) xhci_hcd(E)
aes_x86_64(E) crypto_simd(E) usbcore(E) cryptd(E) usb_common(E)
glue_helper(E) ahci(E) libahci(E) r8169(E) libata(E) i2c_i801(E)
libphy(E) scsi_mod(E) fan(E) thermal(E)
Jan 15 22:31:48 kk-router kernel: [ 6406.094981] ---[ end trace
51787b6133d8b572 ]---
Jan 15 22:31:49 kk-router kernel: [ 6407.221689] RIP:
0010:__check_object_size+0x7b/0x1a0
Jan 15 22:31:49 kk-router kernel: [ 6407.221737] Code: 00 00 80 48 2b 15
5d 0e c7 00 48 01 c2 48 c1 ea 0c 48 c1 e2 06 48 03 15 3b 0e c7 00 48 8b
42 08 48 8d 48 ff a8 01 48 0f 45 d1 <48> 8b 4a 08 48 8d 41 ff 83 e1 01
48 0f 44 c2 48 8b 00 f6 c4 01 75
Jan 15 22:31:49 kk-router kernel: [ 6407.222203] RSP:
0018:ffff96f9415dfc58 EFLAGS: 00010202
Jan 15 22:31:49 kk-router kernel: [ 6407.222236] RAX: efffd06949355a01
RBX: ffff896a0d56cc02 RCX: efffd06949355a00
Jan 15 22:31:49 kk-router kernel: [ 6407.222278] RDX: efffd06949355a00
RSI: ffff896a3fff0000 RDI: ffff896a8d56cc02
Jan 15 22:31:49 kk-router kernel: [ 6407.222662] RBP: 00000000000005a8
R08: 00000000000005a8 R09: 00000000000005a8
Jan 15 22:31:49 kk-router kernel: [ 6407.222703] R10: 0000000000000000
R11: 0000000000000000 R12: 0000000000000001
Jan 15 22:31:49 kk-router kernel: [ 6407.223088] R13: ffff896a0d56d1aa
R14: 00000000000005a8 R15: ffff896a0d56cc02
Jan 15 22:31:49 kk-router kernel: [ 6407.223130] FS:
00007ff8efe39d40(0000) GS:ffff896a37b80000(0000) knlGS:0000000000000000
Jan 15 22:31:49 kk-router kernel: [ 6407.223519] CS:  0010 DS: 0000 ES:
0000 CR0: 0000000080050033
Jan 15 22:31:49 kk-router kernel: [ 6407.223553] CR2: 00007f74c23605e0
CR3: 0000000265ebc000 CR4: 00000000003406e0

Reply | Threaded
Open this post in threaded view
|

Bug#900821: linux-image-4.9.0-6-amd64: apache reads wrong data over cifs filesystems served by samba

Stefan Fritsch
In reply to this post by Santiago Garcia Mantinan
Hi,

by default, apache uses mmap, so probably mmap is broken on cifs. An
alternate workaround should be to set EnableMMAP off in the apache
config.

Cheers,
Stefan