Very slow NFS/Samba performance

[Ian Collins via illumos-zfs]

I'm seeing unexpectedly poor NFS and samba read/write performance on a
well specified SmartOS server.

If I run

rsync --progress -a -e ssh bigFile box:/nfsShare/public/

I see around 70MB/s

however if I try

rsync --progress -a bigFile /net/box/nfsShare/public/

I see < 2 MB/s

The box has 376GB of RAM, 2x200G Intel 3700 SSD logs and 2x512GB Samsung
SSD cache drives. The pool is healthy and about 75% full.

Setting sync=disabled on the nfsShare/public filesystem makes no difference.

Similar performance is seen with smb (Samba) shares.

Any debugging hints welcome!

Thanks,

[Rich via illumos-zfs]

What about sync=always?

There's a few underspecified parameters here - you're describing one
box, but there are presumably at least two involved. I'd presume
you're describing the box receiving the file, but then what's the
source?

What's the network between the two, and what do the two SVG files
resulting from generating a FlameGraph[1] with something like:

dtrace -x stackframes=100 -n 'profile-997 /arg0/ { @[stack()] =
count(); } tick-60s { exit(0); }' -o FOO.stacks
./stackcollapse.pl FOO.stacks > FOO.folded
egrep -v '^unix`thread_start;unix`idle' FOO.folded > FOO_mod.folded
./flamegraph.pl --title "FOO" FOO.folded > FOO.svg
./flamegraph.pl --title "FOO_mod" FOO_mod.folded > FOO_mod.svg

in a checkout of the FlameGraph repo, while you're running the copy
over NFS versus CIFS versus locally look like? [2]

- Rich

[1] - http://www.brendangregg.com/FlameGraphs/cpuflamegraphs.html
[2] - It's possible you're bottlenecking in userland, and that
incantation is for instrumenting where you're spending time in the
kernel, but I'd really doubt that you'd bottleneck in userland prior
to saturating 100 Mbit unless something hilarious is going on...

On Wed, Oct 15, 2014 at 9:56 PM, Ian Collins via illumos-zfs

I'm seeing unexpectedly poor NFS and samba read/write performance on a well
specified SmartOS server.
If I run
rsync --progress -a -e ssh bigFile box:/nfsShare/public/
I see around 70MB/s
however if I try
rsync --progress -a bigFile /net/box/nfsShare/public/
I see < 2 MB/s
The box has 376GB of RAM, 2x200G Intel 3700 SSD logs and 2x512GB Samsung SSD
cache drives. The pool is healthy and about 75% full.
Setting sync=disabled on the nfsShare/public filesystem makes no difference.
Similar performance is seen with smb (Samba) shares.
Any debugging hints welcome!
Thanks,
--
Ian.
-------------------------------------------
illumos-zfs
Archives: https://www.listbox.com/member/archive/182191/=now
https://www.listbox.com/member/archive/rss/182191/24536931-5d25148d
https://www.listbox.com/member/?&
Powered by Listbox: http://www.listbox.com

[Ian Collins via illumos-zfs]

Post by Rich via illumos-zfs
What about sync=always?
There's a few underspecified parameters here - you're describing one
box, but there are presumably at least two involved. I'd presume
you're describing the box receiving the file, but then what's the
source?

No, I'm describing the performance seen form all of the client systems
using the server.

Post by Rich via illumos-zfs
What's the network between the two, and what do the two SVG files

I'm graphing various actions at the moment. The box is very busy, so
it's quite hard to isolate things...

[Ian Collins via illumos-zfs]

Post by Ian Collins via illumos-zfs
I'm seeing unexpectedly poor NFS and samba read/write performance on a
well specified SmartOS server.
If I run
rsync --progress -a -e ssh bigFile box:/nfsShare/public/
I see around 70MB/s
however if I try
rsync --progress -a bigFile /net/box/nfsShare/public/
I see < 2 MB/s
The box has 376GB of RAM, 2x200G Intel 3700 SSD logs and 2x512GB Samsung
SSD cache drives. The pool is healthy and about 75% full.
Setting sync=disabled on the nfsShare/public filesystem makes no difference.
Similar performance is seen with smb (Samba) shares.
Any debugging hints welcome!

Another couple of data points:

The pool is showing a very high busy percentage, but the individual
drives aren't.

Logs:

r/s w/s Mr/s Mw/s wait actv wsvc_t asvc_t %w %b device
0.0 543.6 0.0 12.7 0.3 0.0 0.5 0.1 17 4 c2t0d0
0.0 543.8 0.0 12.6 0.3 0.0 0.6 0.1 19 4 c2t1d0

Typical drive a pool totals:

r/s w/s Mr/s Mw/s wait actv wsvc_t asvc_t %w %b device
8.0 63.2 0.5 1.9 0.0 0.2 0.0 2.7 0 12
c0t5000C500585574E3d0
314.0 2234.5 12.3 73.8 22.7 5.7 8.9 2.2 19 85 zones

Note the 85% busy!

NFS v3 write is now very poor, while it used to be very good:

NFSv3 read/write distributions (us):

read
value ------------- Distribution ------------- count
2 | 0
4 | 7
8 |@@@@@ 148
16 |@@@@@@@@@@@@@@@@@@ 554
32 |@@@@@@@@@@@@@ 396
64 |@ 35
128 | 10
256 |@ 34
512 |@ 23
1024 | 1
2048 | 5
4096 |@ 18
8192 | 14
16384 | 6
32768 | 7
65536 | 3
131072 | 0

write
value ------------- Distribution ------------- count
64 | 0
128 | 58
256 |@ 108
512 | 72
1024 | 10
2048 | 4
4096 | 11
8192 |@@ 253
16384 |@@@@@@@ 1048
32768 |@@@@@@@@@@@@@@@@@@@@ 2850
65536 |@@@@@@@@@ 1346
131072 | 57
262144 | 3
524288 | 0

[Ian Collins via illumos-zfs]

Post by Ian Collins via illumos-zfs

Post by Ian Collins via illumos-zfs
I'm seeing unexpectedly poor NFS and samba read/write performance on a
well specified SmartOS server.
If I run
rsync --progress -a -e ssh bigFile box:/nfsShare/public/
I see around 70MB/s
however if I try
rsync --progress -a bigFile /net/box/nfsShare/public/
I see < 2 MB/s
The box has 376GB of RAM, 2x200G Intel 3700 SSD logs and 2x512GB Samsung
SSD cache drives. The pool is healthy and about 75% full.
Setting sync=disabled on the nfsShare/public filesystem makes no difference.
Similar performance is seen with smb (Samba) shares.
Any debugging hints welcome!

The pool is showing a very high busy percentage, but the individual
drives aren't.
r/s w/s Mr/s Mw/s wait actv wsvc_t asvc_t %w %b device
0.0 543.6 0.0 12.7 0.3 0.0 0.5 0.1 17 4 c2t0d0
0.0 543.8 0.0 12.6 0.3 0.0 0.6 0.1 19 4 c2t1d0
r/s w/s Mr/s Mw/s wait actv wsvc_t asvc_t %w %b device
8.0 63.2 0.5 1.9 0.0 0.2 0.0 2.7 0 12
c0t5000C500585574E3d0
314.0 2234.5 12.3 73.8 22.7 5.7 8.9 2.2 19 85 zones
Note the 85% busy!

I did discover one smbd process doing a very high number of very small
(79 byte) writes. When this process stopped, NFS performance picked up
again. The strange thing is this would have been running on the old
host which had 1/4 the RAM, 1/4 the cores and half the spindles.

[Ian Collins via illumos-zfs]

Post by Ian Collins via illumos-zfs
I'm seeing unexpectedly poor NFS and samba read/write performance on a
well specified SmartOS server.
If I run
rsync --progress -a -e ssh bigFile box:/nfsShare/public/
I see around 70MB/s
however if I try
rsync --progress -a bigFile /net/box/nfsShare/public/
I see < 2 MB/s

It turns out after some basic dtrace probing that the problem was samba
logging set too high. The smbd processes were generating ~100K writes a
second...

Am I correct in assuming one reason for low drive %b and high pool %b
was single threaded write coalescing?

I assume this would explain why (synchronous) NFS writes where impacted,
but not large sequential writes from scp.

[Keith Wesolowski via illumos-zfs]

Post by Ian Collins via illumos-zfs
Am I correct in assuming one reason for low drive %b and high pool %b
was single threaded write coalescing?

I don't think so. In our investigations in the past, we found that this
number (pool %busy) has essentially the same semantics as each device;
i.e., if there is even a single outstanding request against the pool as
a whole, the pool is "busy". On a pool with a single vdev, that number
is (possibly) meaningful. On any other pool, it isn't, and the more
vdevs you have, the less useful it is. In practice it seems to be very
easy to have a 100% busy pool with underlying devices that are very
lightly utilised. As such, I tend to ignore the pool %busy figure
entirely. It doesn't seem (necessarily) to have anything to do with
write-coalescing or threadedness. If you want to indict coalescing, it
seems you need to look deeper.