NFS rename sometimes hangs for 15 seconds after upgrade to Debian 8

classic Classic list List threaded Threaded
17 messages Options
Reply | Threaded
Open this post in threaded view
|

NFS rename sometimes hangs for 15 seconds after upgrade to Debian 8

Vincent Lefevre-10
At my lab, with NFS accounts, some machines have been upgraded to
Debian 8, and I get regular hangs on these machines, while they
never occurred before on the same machine and still never occur
on a machine that is still under Debian 7.

This happens with one of my scripts, which does a lot of "mv",
and more often when the target has just been modified from another
machine.

A strace showed that it is the "rename" system call that hangs:

52026      0.000208 execve("/bin/mv", ["mv", "-v", "ssh/ssh-reconfigure", "/home/vlefevre/bin/ssh-reconfigu"...], [/* 112 vars */]) = 0
[...]
52026      0.000097 rename("ssh/ssh-reconfigure", "/home/vlefevre/bin/ssh-reconfigure") = 0
52026     15.057725 lseek(0, 0, SEEK_CUR) = -1 ESPIPE (Illegal seek)
52026      0.000189 close(0)            = 0
52026      0.000175 close(1)            = 0
52026      0.000093 munmap(0x7fd5a610b000, 4096) = 0
52026      0.000079 close(2)            = 0
52026      0.000106 exit_group(0)       = ?
52026      0.000307 +++ exited with 0 +++

The NFS server isn't particularly loaded.

Has anyone seen a problem like that?
Where does it come from?
And why does it affect only Debian 8 machines? A new bug?

--
Vincent Lefèvre <[hidden email]> - Web: <https://www.vinc17.net/>
100% accessible validated (X)HTML - Blog: <https://www.vinc17.net/blog/>
Work: CR INRIA - computer arithmetic / AriC project (LIP, ENS-Lyon)

Reply | Threaded
Open this post in threaded view
|

Re: NFS rename sometimes hangs for 15 seconds after upgrade to Debian 8

Vincent Lefevre-10
On 2015-09-22 15:19:10 +0200, Vincent Lefevre wrote:
> At my lab, with NFS accounts, some machines have been upgraded to
> Debian 8, and I get regular hangs on these machines, while they
> never occurred before on the same machine and still never occur
> on a machine that is still under Debian 7.
[...]

I've done a bug report:

  https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=799838

with a simple way to reproduce the bug:

  https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=799838#10

i.e. it suffices to read the target on some machine, then move a file
to the target on another (Debian 8) machine.

--
Vincent Lefèvre <[hidden email]> - Web: <https://www.vinc17.net/>
100% accessible validated (X)HTML - Blog: <https://www.vinc17.net/blog/>
Work: CR INRIA - computer arithmetic / AriC project (LIP, ENS-Lyon)

Reply | Threaded
Open this post in threaded view
|

Re: NFS rename sometimes hangs for 15 seconds after upgrade to Debian 8

Peter Ludikovsky
In reply to this post by Vincent Lefevre-10
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Hello,

I tried to replicate your problem, but couldn't.

When mounted as NFS3 over UDP:
- ----
root@lab1# echo foo > file1
root@lab1# cp file1 file2

root@lab2# cat file2
foo

root@lab1# time mv file1 file2
real 0m0.004s
user 0m0.000s
sys 0m0.000s
- ----

Same with NFS4 over TCP:
root@lab1# time mv file1 file2
real 0m0.005s
user 0m0.000s
sys 0m0.000s

Any special mount options or export options you might be using?

Regards,
/peter

Am 22.09.2015 um 15:19 schrieb Vincent Lefevre:

> At my lab, with NFS accounts, some machines have been upgraded to
> Debian 8, and I get regular hangs on these machines, while they
> never occurred before on the same machine and still never occur on
> a machine that is still under Debian 7.
>
> This happens with one of my scripts, which does a lot of "mv", and
> more often when the target has just been modified from another
> machine.
>
> A strace showed that it is the "rename" system call that hangs:
>
> 52026      0.000208 execve("/bin/mv", ["mv", "-v",
> "ssh/ssh-reconfigure", "/home/vlefevre/bin/ssh-reconfigu"...], [/*
> 112 vars */]) = 0 [...] 52026      0.000097
> rename("ssh/ssh-reconfigure",
> "/home/vlefevre/bin/ssh-reconfigure") = 0 52026     15.057725
> lseek(0, 0, SEEK_CUR) = -1 ESPIPE (Illegal seek) 52026
> 0.000189 close(0)            = 0 52026 0.000175 close(1)
> = 0 52026      0.000093 munmap(0x7fd5a610b000, 4096) = 0 52026
> 0.000079 close(2) = 0 52026      0.000106 exit_group(0)       = ?
> 52026      0.000307 +++ exited with 0 +++
>
> The NFS server isn't particularly loaded.
>
> Has anyone seen a problem like that? Where does it come from? And
> why does it affect only Debian 8 machines? A new bug?
>
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.22 (MingW32)

iQIcBAEBAgAGBQJWAsuuAAoJEM+6Ng5pbtyZGhMP/2yC/CgWhb5+yvA5jb/pd3sW
FwNw1mvH+FQikKVvvPgy5AMO3d2rQafh/fXQlfZRnC0Hu3jXn99K60mV//88KqGm
N+axMi5octq8s2zhF/K0/axsiPHAhgo9ggdlcdXE0ekQvgw5gzXJ6hDutnaFTaun
gay6HK6Ujp0JuvCxUQzq8uiv5VTPJiT0ftyBAF02dFqO0yBgGhDAt80NUtMGRwZB
nuya8pD7EhZH7x0vabnv3935+X0V1bpQhyKHWAbzCMY4Uce2wt0wtR9FEJVOyj4v
gQ7WHULo53fYOLdKw5REhgtEDqIFDz6MRsOup28gv7sZBW7/Ps3IbOi06/ovuR3W
WWZhYys+F/5Nmt8tKD7XWZfYQ7TnUN35Z1PS0xCK8+cEUWxSbI2OftyVBfI+iRCn
W3n0N2cvxyaRfWGwLs5ROO7/g+5xeNH5WiziaNZa5QfFae8j2J8lUIyG8aSZ/Axe
n0/HfwfA5vp9IIPVSOgsw+xmjkzCtIOtL+2cWBu5nTixD301hFRSuZCbUsz5f0Sq
tDUDzyYKYsUXJ5+YvBktgZ1dKD6nR3TuJdRl70gPJo6RuCxhvfRqHy47qrSOw6qL
HfKUZYnlgOMV1ufXLTI9sTq/YcDmJzKD+LDJUlumF88le58w5bq8KeR8uusGySSF
wQjgvRlwVkzvDsxm9b0z
=CCwk
-----END PGP SIGNATURE-----

Reply | Threaded
Open this post in threaded view
|

Re: NFS rename sometimes hangs for 15 seconds after upgrade to Debian 8

Vincent Lefevre-10
On 2015-09-23 17:56:30 +0200, Peter Ludikovsky wrote:

> When mounted as NFS3 over UDP:
> - ----
> root@lab1# echo foo > file1
> root@lab1# cp file1 file2
>
> root@lab2# cat file2
> foo
>
> root@lab1# time mv file1 file2
> real 0m0.004s
> user 0m0.000s
> sys 0m0.000s
> - ----
>
> Same with NFS4 over TCP:
> root@lab1# time mv file1 file2
> real 0m0.005s
> user 0m0.000s
> sys 0m0.000s
>
> Any special mount options or export options you might be using?

Here this is NFS4 over TCP:

filer.lip.ens-lyon.fr:/export/home/vlefevre on /home/vlefevre type nfs4 (rw,nosuid,relatime,vers=4.0,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,port=0,timeo=600,retrans=2,sec=sys,clientaddr=140.77.14.11,local_lock=none,addr=140.77.14.5)

Is there any other information I could get?

--
Vincent Lefèvre <[hidden email]> - Web: <https://www.vinc17.net/>
100% accessible validated (X)HTML - Blog: <https://www.vinc17.net/blog/>
Work: CR INRIA - computer arithmetic / AriC project (LIP, ENS-Lyon)

Reply | Threaded
Open this post in threaded view
|

Re: NFS rename sometimes hangs for 15 seconds after upgrade to Debian 8

Peter Ludikovsky
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Am 24.09.2015 um 11:46 schrieb Vincent Lefevre:

> On 2015-09-23 17:56:30 +0200, Peter Ludikovsky wrote:
>> When mounted as NFS3 over UDP: - ---- root@lab1# echo foo >
>> file1 root@lab1# cp file1 file2
>>
>> root@lab2# cat file2 foo
>>
>> root@lab1# time mv file1 file2 real 0m0.004s user 0m0.000s sys
>> 0m0.000s - ----
>>
>> Same with NFS4 over TCP: root@lab1# time mv file1 file2 real
>> 0m0.005s user 0m0.000s sys 0m0.000s
>>
>> Any special mount options or export options you might be using?
>
> Here this is NFS4 over TCP:
>
> filer.lip.ens-lyon.fr:/export/home/vlefevre on /home/vlefevre type
>  nfs4
> (rw,nosuid,relatime,vers=4.0,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,port=0,timeo=600,retrans=2,sec=sys,clientaddr=140.77.14.11,local_lock=none,addr=140.77.14.5)
>
>
>
>
Is there any other information I could get?
>

The most obvious thing: remove the public visible IPs and DNS names.

Other than that, what mount command did you use? Are you mounting the
share yourself, or is this an fstab or autofs mount? Do you have any
control over the NFS server, and if so what are its settings?

I did manage to replicate your issue. However, it disappears as soon
as I sync before the mv, as well as when both the rsize and wsize
parameters are reduced to 65536, suggesting this has something to do
with a local cache and/or lock.

Apparently the issue also disappears as soon as both sides have noac
added to the mount options.

Regards,
/peter
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.22 (MingW32)

iQIcBAEBAgAGBQJWA+6pAAoJEM+6Ng5pbtyZvjEQALFrhCJHmF/JYmAvXncmbBdz
PYvSHTbAN9vNzCc4vqzkf4CmhgIVNSrP0WsPE/AHo2BNgpQQIQzfiiTgLGRBJnqT
jDctQE/bJYMhk1aw08b/c2l6a7Ykc9R1rXu516ftrBKe9f5fcxFLndjIrtgGP01K
3Tah6v8epDd2t3MDzhABZdn4QB5oWesgcnISJOBsrds8+vfYOKjEYiCNGm7wlduV
8968HXrCWJxNrkyhMmHTpPopFxzk1LZD270zxkUiidA4kDxjcADSCJhP6+Bivv1H
h+QhzNZ8NUEYBam+PuL3LhlBFalgvEZGfRF8gULEbjX7umA4b2tLWhgQAraYmiDO
arfCrUPkcqv4oB9mldrL1PfhKLzMtz4bQfg3/8u7gBBWnpkFq7JhE9CMQ7oddBBQ
rosT9sXsCRneYXfINJ57czof2kca3+5p/nZB7wgANKwD0nFWhFObduZJRnxqNXes
jdKNRrGg4oy0HJAMWXUvr/0Oy5iqoWwtGNhx4FHr2W4Lgie8g+YOciLsInsdytCu
x5x2EHcIOpVUBkya+xHxD02kk2loWKNh/sIV0RdgKabJ9YhY7srxh6uiPm/SNnvJ
67mpH7ekg5FOFc6njndeq1nXBzp/CX29GM8AnC5no5/Zmqf1qLiXxlTJFvYzZpNS
g4oRynIeq5Zd7sD2JoaW
=FojV
-----END PGP SIGNATURE-----

Reply | Threaded
Open this post in threaded view
|

Re: NFS rename sometimes hangs for 15 seconds after upgrade to Debian 8

Vincent Lefevre-10
On 2015-09-24 14:38:01 +0200, Peter Ludikovsky wrote:
> Other than that, what mount command did you use? Are you mounting the
> share yourself, or is this an fstab or autofs mount?

autofs

The only option used in /etc/auto.master is the obvious -o nosuid.

But the options seem to be the same on Debian 7 clients.

> Do you have any control over the NFS server, and if so what are its
> settings?

I could ask the admin, if they are useful for the bug.

> I did manage to replicate your issue. However, it disappears as soon
> as I sync before the mv, as well as when both the rsize and wsize
> parameters are reduced to 65536, suggesting this has something to do
> with a local cache and/or lock.
>
> Apparently the issue also disappears as soon as both sides have noac
> added to the mount options.

Did you have to change the server settings to replicate the problem?

Note that if I do a "touch" on the target before the mv, the problem
doesn't occur.

--
Vincent Lefèvre <[hidden email]> - Web: <https://www.vinc17.net/>
100% accessible validated (X)HTML - Blog: <https://www.vinc17.net/blog/>
Work: CR INRIA - computer arithmetic / AriC project (LIP, ENS-Lyon)

Reply | Threaded
Open this post in threaded view
|

Re: NFS rename sometimes hangs for 15 seconds after upgrade to Debian 8

Peter Ludikovsky
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1



Am 25.09.2015 um 09:32 schrieb Vincent Lefevre:

> On 2015-09-24 14:38:01 +0200, Peter Ludikovsky wrote:
>> Other than that, what mount command did you use? Are you mounting
>> the share yourself, or is this an fstab or autofs mount?
>
> autofs
>
> The only option used in /etc/auto.master is the obvious -o nosuid.
>
> But the options seem to be the same on Debian 7 clients.
>
>> Do you have any control over the NFS server, and if so what are
>> its settings?
>
> I could ask the admin, if they are useful for the bug.
>
>> I did manage to replicate your issue. However, it disappears as
>> soon as I sync before the mv, as well as when both the rsize and
>> wsize parameters are reduced to 65536, suggesting this has
>> something to do with a local cache and/or lock.
>>
>> Apparently the issue also disappears as soon as both sides have
>> noac added to the mount options.
>
> Did you have to change the server settings to replicate the
> problem?
>
> Note that if I do a "touch" on the target before the mv, the
> problem doesn't occur.
>

No, I didn't have to change anything on the server side. noac is a
mount setting for NFS, which basically tells the client not to cache
any file/directory attributes, but to ask the server each time.

My guess is:
Due to the rather large wsize/rsize, the clients create a rather large
attribute cache. As a result, when you cat a file on the second
machine it updates the atime in the cache, but doesn't yet transfer
that information to the server. When you do the mv on the first
machine, the client tries to get the current attributes on the files
involved, which prompts the server to wait until the second client
runs into some kind of timeout and updates the attributes. Only then
is the second file "unlocked" to be overwritten.

Further proof of that: when setting noatime as a mount option, instead
of noac, there's also no delay on the mv.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.22 (MingW32)

iQIcBAEBAgAGBQJWBQGeAAoJEM+6Ng5pbtyZel8QAItKpN7wTP527aF06YGYymwQ
Ema5V6cQ8dfYHT6dr2YKqCdG+I9TZeg4xeX5i6b8jtrbKqTHuOFK95SfN/dt3FTb
6clEgXndEZP022Mko7C7IGVBsQQIgnc57POywsglcBW3Vkq7FhBualg10S+KnhUw
XFeHhW9SFNwY6sjM+r77rvzilMjEnbry/ik8gSzEB9Y/Zak92DzHFVB/mQshgZKG
uDuwI3ZY9pRsW8QfxfX4/Ueo6IwtrP3K9eBObifbpiuFzfj8h13+hA4rCp+JYfhb
+2lvlngzQH6pArk/cXt/7RkIi7Rr4Baxyn4ZR5viO2HcDBtFszsvWJkwHMl6JO5v
7YeNL0ZuVQiHXZ5XjNKiVnMuyu0cBuzgYA/vOqwkMDbJj5GRZ9g2Mu3BIxHjCI5l
xqGJ2Lj3RJpzk5K0y0BYurxxEA2Fr4L2r3OqrChHXzYdU/+sCJ1FRvU+tHK++Vdn
CgGxnrWn7nzBo3l1MCmj1KeZgEVzCkBJhJ66AWlNL4EUYEDYowolaNOur1dsT9Ci
dxYxWru8YyifEW3zNlBkBJXqu4s2YkTY5J7pS4apYX4jvhrKpFkqtlu/gvTYzLrc
PU9W/12VsmBQqLhfIo/1We0iRX/wHNyMSHgI/vvWsuwtlgjcM4ECwNK/M96WLncr
0CuIVjGKpxXxn6FOC4Ey
=DNAx
-----END PGP SIGNATURE-----

Reply | Threaded
Open this post in threaded view
|

Re: NFS rename sometimes hangs for 15 seconds after upgrade to Debian 8

Vincent Lefevre-10
On 2015-09-25 10:11:10 +0200, Peter Ludikovsky wrote:

> My guess is:
> Due to the rather large wsize/rsize, the clients create a rather large
> attribute cache. As a result, when you cat a file on the second
> machine it updates the atime in the cache, but doesn't yet transfer
> that information to the server. When you do the mv on the first
> machine, the client tries to get the current attributes on the files
> involved, which prompts the server to wait until the second client
> runs into some kind of timeout and updates the attributes. Only then
> is the second file "unlocked" to be overwritten.
>
> Further proof of that: when setting noatime as a mount option, instead
> of noac, there's also no delay on the mv.

Thanks for the explanations. But the second client, though it doesn't
update the attributes immediately, seems to do it quickly when asked
by the server, otherwise the problem would also be visible on Debian 7
machines. I think that on Debian 8, when the rename cannot be done
immediately, the client retries after 15 seconds instead of retrying
earlier.

Is this related to the following patch?

http://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/commit/?id=8478eaa16e701ecfe054b62ec764bc1291b79e19

--
Vincent Lefèvre <[hidden email]> - Web: <https://www.vinc17.net/>
100% accessible validated (X)HTML - Blog: <https://www.vinc17.net/blog/>
Work: CR INRIA - computer arithmetic / AriC project (LIP, ENS-Lyon)

Reply | Threaded
Open this post in threaded view
|

Re: NFS rename sometimes hangs for 15 seconds after upgrade to Debian 8

Peter Ludikovsky
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1



Am 25.09.2015 um 14:04 schrieb Vincent Lefevre:

> On 2015-09-25 10:11:10 +0200, Peter Ludikovsky wrote:
>> My guess is: Due to the rather large wsize/rsize, the clients
>> create a rather large attribute cache. As a result, when you cat
>> a file on the second machine it updates the atime in the cache,
>> but doesn't yet transfer that information to the server. When you
>> do the mv on the first machine, the client tries to get the
>> current attributes on the files involved, which prompts the
>> server to wait until the second client runs into some kind of
>> timeout and updates the attributes. Only then is the second file
>> "unlocked" to be overwritten.
>>
>> Further proof of that: when setting noatime as a mount option,
>> instead of noac, there's also no delay on the mv.
>
> Thanks for the explanations. But the second client, though it
> doesn't update the attributes immediately, seems to do it quickly
> when asked by the server, otherwise the problem would also be
> visible on Debian 7 machines. I think that on Debian 8, when the
> rename cannot be done immediately, the client retries after 15
> seconds instead of retrying earlier.
>
> Is this related to the following patch?
>
> http://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/commit/?id=8478eaa16e701ecfe054b62ec764bc1291b79e19
>
>
Can't say if it's related. I have, however, managed to capture the
related network traffic (see attachments).

The big difference happens at packets 58/54 (Deb7/Deb8). For Deb7, the
RENAME call is immediately answered by an NFS4_OK, whereas for Deb8 as
the client it's an NFS4ERR_DELAY. I haven't seen any reason on the
client communication that would explain that, however this is far
beyond my knowledge of the NFS internals.

Maybe attach that information to your bug report as a point for
investigation.

Regards
/peter
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.22 (MingW32)

iQIcBAEBAgAGBQJWCQizAAoJEM+6Ng5pbtyZxBUQAIp/OIHN4SM1N6DnS1GjN31Q
8i0eDFw3PNd/LoFBuQ8CUPK7U1ts8II2z0hBqCSFVj8VGGnxOVgkqYPEh2+Ovz4G
1SkeDmlo79POIwfXIhL8ORG/0svr2bbZ81L4ciFAQyF2xiRLq8Y+fNTgS5qNAjm2
xSFwnno7rXyKkT/U1oufjoCD65u263UpBCYgDeNs/pgbay+q3CEmYJOerd10cue7
fkgzysdRTYKQ9G9LqPsWPspMo+TMxzr415ln0bYcBmN/R0mQx4+lPzZsacSX4Hjh
FO5y8j0xyJKUE9BS3dCDW7me9SMQl2fbCoS9eDTs5LocwnvhggVVbtVc1iAHL8te
S06Fq8c2Pj5WNQlCP1cY42drJnqYOZdTirk2aAGAdklbRP1CBAvecld2r0vdAHFv
2tvSeyIYNVR5eGDT4tbpx94Dzy/MOM66d1Cxlu5lBk/pPOw1/gHwgxCIeHAXXV1d
5i1OYenFN988JQbr8kpyBwBVBuCK4IiGnOkZRECIdTS1VV6YzlpMH8LeRvKs8OiO
t09O8LBoHvA9Zgy7Fz8c3mDlHhcRqJXRH7ArspMM519eFaXaxIzA0o1vQfHUzDB/
yPfVsnX0oCRF9VwfpIMOCg0RJBb/XZB3Iiotou+Uj8QOrcyY0Hw3NBOy1kQjtvUh
bGysI2qLKAcggJ3P7k4n
=3edq
-----END PGP SIGNATURE-----

nfs_deb7.pcap (18K) Download Attachment
nfs_deb8.pcap (18K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: NFS rename sometimes hangs for 15 seconds after upgrade to Debian 8

Vincent Lefevre-10
On 2015-09-28 11:30:27 +0200, Peter Ludikovsky wrote:
> Maybe attach that information to your bug report as a point for
> investigation.

Thanks. I have pasted the message contents and added a link to it
(in the mailing-list archives).

--
Vincent Lefèvre <[hidden email]> - Web: <https://www.vinc17.net/>
100% accessible validated (X)HTML - Blog: <https://www.vinc17.net/blog/>
Work: CR INRIA - computer arithmetic / AriC project (LIP, ENS-Lyon)

Reply | Threaded
Open this post in threaded view
|

Re: NFS rename sometimes hangs for 15 seconds after upgrade to Debian 8

Mike Kupfer
In reply to this post by Peter Ludikovsky
Peter Ludikovsky wrote:

> The big difference happens at packets 58/54 (Deb7/Deb8). For Deb7, the
> RENAME call is immediately answered by an NFS4_OK, whereas for Deb8 as
> the client it's an NFS4ERR_DELAY. I haven't seen any reason on the
> client communication that would explain that, however this is far
> beyond my knowledge of the NFS internals.

The DELAY response in the deb8 trace is likely so that the server can
ask the .3 client to return its delegation, which it does at packets
56-57.  (The .3 client obtained its delegation at packet 39 as part of
the reply to the OPEN op.)

In the deb7 trace, the other client is .4, not .3.  It does not get a
delegation when it opens file2 (see packet 39), so the server can
process the RENAME immediately.

mike

Reply | Threaded
Open this post in threaded view
|

Re: NFS rename sometimes hangs for 15 seconds after upgrade to Debian 8

Peter Ludikovsky
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1



Am 28.09.2015 um 18:59 schrieb Mike Kupfer:

> Peter Ludikovsky wrote:
>
>> The big difference happens at packets 58/54 (Deb7/Deb8). For
>> Deb7, the RENAME call is immediately answered by an NFS4_OK,
>> whereas for Deb8 as the client it's an NFS4ERR_DELAY. I haven't
>> seen any reason on the client communication that would explain
>> that, however this is far beyond my knowledge of the NFS
>> internals.
>
> The DELAY response in the deb8 trace is likely so that the server
> can ask the .3 client to return its delegation, which it does at
> packets 56-57.  (The .3 client obtained its delegation at packet 39
> as part of the reply to the OPEN op.)
>
> In the deb7 trace, the other client is .4, not .3.  It does not get
> a delegation when it opens file2 (see packet 39), so the server
> can process the RENAME immediately.
>
> mike
>

Hang on, the two traces haven't been made at the same time, but rather
separate. As in:

* mount the share on 2 machines
* create the files on one
* cat it on the other
* time mv again on the first
* umount the share

However, the captures omit the mount/umount part, since that doesn't
appear to affect the outcome.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.22 (MingW32)

iQIcBAEBAgAGBQJWCXahAAoJEM+6Ng5pbtyZJ1IP/3QJi8sJPPfGvLBxwKK8h2lx
ywnVGRzZhM90ILND6JjIBXmlgKEdGf6fHxdu9cjLDYri3CzfL/oM2T/X3CPmCn3s
23T8x2P6Yp3jGhR8HgxV/HWlUAt3wKpzfkYiQFGHVmjR78nEE61Do027p/Z2M31E
nBy8yNUoINjY0RGfiHVm0zzU+CrPG/CVUmWNFTB+Z9xkz96svqdudttoa5HMEON+
QHzNYtyw2dsg9EI2VxxDe/Uv1MJz7CIy41cTsQCZYaQOhBw+ywExNRww6IZcDg4R
QpLNv2Vo2NvQpuENQyqgEgx22opTuKbNLFoCN2DUDWPXhjRN1sqVuWgjWvX3+7X7
et3EqddZRZosjSOSI1sTMfWixQUIvimg9qnlovBh4qzZXmsyPUKC6852qhoHk/IN
JyHggEff/u3TeMmJqnyHgI1Gys08kXyCql3DGMyhIii4xJ2bLK//JvyhuGe6JzOn
Zo5UQt1nr+M5t0xwCKX0HvpnSbKhC04HDYxaTZ/DAgT9t3yOMFstM0vn8m0M4z2C
W3ry8itJf5PP9bWCeI3XbgsZkwrcg57oK4VfXE3hGQjzZQ+Bkina1vLQcDRywcrx
7ENbUQH7QbzmheM/ezzw6Vz+AdRvWwus1CSJ0fqeBTdSU+lDGUKQpLVbtY1yQfvL
XK5RL1TmemDp5VkrRDPT
=+m/o
-----END PGP SIGNATURE-----

Reply | Threaded
Open this post in threaded view
|

Re: NFS rename sometimes hangs for 15 seconds after upgrade to Debian 8

Mike Kupfer
Peter Ludikovsky wrote:

> Am 28.09.2015 um 18:59 schrieb Mike Kupfer:

> > In the deb7 trace, the other client is .4, not .3.  It does not get
> > a delegation when it opens file2 (see packet 39), so the server
> > can process the RENAME immediately.

> Hang on, the two traces haven't been made at the same time, but rather
> separate. As in:
>
> * mount the share on 2 machines
> * create the files on one
> * cat it on the other
> * time mv again on the first
> * umount the share

Okay, but I don't understand why you mention that.  In the deb7 trace,
the "other" client is not granted a delegation.  In the deb8 trace, the
"other" client *is* granted a delegation.  And it's almost certainly the
presence of this delegation that causes the server to stall the RENAME
request.

I looked at the traces some more.  In the deb7 trace, the open of file2
by the .4 client is followed by an OPEN_CONFIRM handshake.  This means
this is the first open that the .4 client is doing using that particular
open_owner (one of the state objects that NFSv4 introduced).  The
callback information in the SETCLIENTID call (packet 34) looks sane, so
I think this is why the server did not grant a delegation in the deb7
trace case.  That is, subsequent OPEN requests from the .4 client might
be granted delegations.  (I say "might" because the server is free to
use various heuristics in deciding when to grant a delegation, and I
don't know what logic the Linux v4 server uses.)

I think it would be best to see packet traces from Vincent, if that's
possible.  (The trace should include traffic between the server and both
clients.)  I can think of a couple possible reasons why his Deb7 client
behaves better than the Deb8 clients, and having traces from his rig
would reduce the guesswork.

mike

Reply | Threaded
Open this post in threaded view
|

Re: NFS rename sometimes hangs for 15 seconds after upgrade to Debian 8

Vincent Lefevre-10
On 2015-09-28 20:12:50 -0700, Mike Kupfer wrote:
> I think it would be best to see packet traces from Vincent, if that's
> possible.  (The trace should include traffic between the server and both
> clients.)  I can think of a couple possible reasons why his Deb7 client
> behaves better than the Deb8 clients, and having traces from his rig
> would reduce the guesswork.

Is there a way to get traces as a normal user?
Otherwise I'll have to ask the sysadmin...

--
Vincent Lefèvre <[hidden email]> - Web: <https://www.vinc17.net/>
100% accessible validated (X)HTML - Blog: <https://www.vinc17.net/blog/>
Work: CR INRIA - computer arithmetic / AriC project (LIP, ENS-Lyon)

Reply | Threaded
Open this post in threaded view
|

Re: NFS rename sometimes hangs for 15 seconds after upgrade to Debian 8

Mike Kupfer
Vincent Lefevre wrote:

> Is there a way to get traces as a normal user?

Not that I know of.

mike

Reply | Threaded
Open this post in threaded view
|

Re: NFS rename sometimes hangs for 15 seconds after upgrade to Debian 8

Stephan Seitz-5
In reply to this post by Vincent Lefevre-10
On Thu, Oct 01, 2015 at 01:54:07PM +0200, Vincent Lefevre wrote:
>Is there a way to get traces as a normal user?
>Otherwise I'll have to ask the sysadmin...

Yes. If you do a „dpkg-reconfigure wireshark-common” you’ll get ask if
normal user should be allowed to trace. If you say yes then a new group
called wireshark will be created. Everyone who is a member of this group
can now use wireshark or tshark.

This should work in Debian 7 and 8.

Many greetings,

        Stephan

--
| Stephan Seitz          E-Mail: [hidden email] |
| Public Keys: http://fsing.rootsland.net/~stse/keys.html |

smime.p7s (5K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: NFS rename sometimes hangs for 15 seconds after upgrade to Debian 8

Vincent Lefevre-10
On 2015-10-01 19:43:03 +0200, Stephan Seitz wrote:
> On Thu, Oct 01, 2015 at 01:54:07PM +0200, Vincent Lefevre wrote:
> >Is there a way to get traces as a normal user?
> >Otherwise I'll have to ask the sysadmin...
>
> Yes. If you do a „dpkg-reconfigure wireshark-common” you’ll get ask
> if normal user should be allowed to trace. If you say yes then a new
> group called wireshark will be created. Everyone who is a member of
> this group can now use wireshark or tshark.

This doesn't help because a normal user can't do that, and even if the
sysadmin adds a normal user to the wireshark group, this would allow
this user to see the packets for the processes of the other users.

--
Vincent Lefèvre <[hidden email]> - Web: <https://www.vinc17.net/>
100% accessible validated (X)HTML - Blog: <https://www.vinc17.net/blog/>
Work: CR INRIA - computer arithmetic / AriC project (LIP, ENS-Lyon)