Enumerating async_destroy operations in progress

[Eric Sproul]

Hi,
I'm confronted with an issue where I suspect that too many
async_destroy operations are going on at once, causing saturation of
I/O resources and knock-on effects up the stack to clients. Is there
a way to enumerate the destroy operations that are currently in
progress?

A bit of preliminary dtrace has shown this kernel stack, but only
once, regardless of how many destroys are going on (maybe multiple
destroys in the same txg are coalesced):

genunix`ldi_strategy+0x53
zfs`vdev_disk_io_start+0xcc
zfs`zio_vdev_io_start+0xab
zfs`zio_execute+0x88
zfs`zio_nowait+0x21
zfs`vdev_mirror_io_start+0xcd
zfs`zio_vdev_io_start+0xab
zfs`zio_execute+0x88
zfs`zio_nowait+0x21
zfs`vdev_mirror_io_start+0xcd
zfs`zio_vdev_io_start+0x250
zfs`zio_execute+0x88
zfs`zio_nowait+0x21
zfs`arc_read+0x519
zfs`traverse_prefetch_metadata+0x9f
zfs`traverse_visitbp+0x38b
zfs`traverse_dnode+0x8b
zfs`traverse_visitbp+0x607
zfs`traverse_impl+0x159
zfs`traverse_dataset_destroyed+0x49
1

Perhaps there is some way to get a list of destroy operations via mdb?

Any pointers or advice would be appreciated.
Thanks,
Eric

[Matthew Ahrens]

Post by Eric Sproul
Hi,
I'm confronted with an issue where I suspect that too many
async_destroy operations are going on at once,

The async destroy operations aren't processed all at once; zfs only works
on one at a time. So having one in progress would have the same
performance impact as having queued up many destroys.

Post by Eric Sproul
causing saturation of
I/O resources and knock-on effects up the stack to clients. Is there
a way to enumerate the destroy operations that are currently in
progress?

You could get the destroy queue via mdb or zdb, but unfortunately I don't
think we keep any human-readable description of the destroy (e.g. name of
the filesystem that was destroyed). You can see which filesystems and
snapshots were destroyed recently with "zpool history [-i]". You can see
how much data remains to be destroyed with "zpool get freeing".

You may be able to improve client write performance by reducing
zfs_scan_min_time_ms (try halving or quartering it).

On very recent bits, you may be able to improve client read performance by
reducing zfs_vdev_async_read_max_active (from 3 to 1).

--matt

Post by Eric Sproul
A bit of preliminary dtrace has shown this kernel stack, but only
once, regardless of how many destroys are going on (maybe multiple
genunix`ldi_strategy+0x53
zfs`vdev_disk_io_start+0xcc
zfs`zio_vdev_io_start+0xab
zfs`zio_execute+0x88
zfs`zio_nowait+0x21
zfs`vdev_mirror_io_start+0xcd
zfs`zio_vdev_io_start+0xab
zfs`zio_execute+0x88
zfs`zio_nowait+0x21
zfs`vdev_mirror_io_start+0xcd
zfs`zio_vdev_io_start+0x250
zfs`zio_execute+0x88
zfs`zio_nowait+0x21
zfs`arc_read+0x519
zfs`traverse_prefetch_metadata+0x9f
zfs`traverse_visitbp+0x38b
zfs`traverse_dnode+0x8b
zfs`traverse_visitbp+0x607
zfs`traverse_impl+0x159
zfs`traverse_dataset_destroyed+0x49
1
Perhaps there is some way to get a list of destroy operations via mdb?
Any pointers or advice would be appreciated.
Thanks,
Eric
-------------------------------------------
illumos-zfs
Archives: https://www.listbox.com/member/archive/182191/=now
https://www.listbox.com/member/archive/rss/182191/21635000-ebd1d460
https://www.listbox.com/member/?&
Powered by Listbox: http://www.listbox.com

-------------------------------------------
illumos-zfs
Archives: https://www.listbox.com/member/archive/182191/=now
RSS Feed: https://www.listbox.com/member/archive/rss/182191/23047029-187a0c8d
Modify Your Subscription: https://www.listbox.com/member/?member_id=23047029&id_secret=23047029-2e85923f
Powered by Listbox: http://www.listbox.com

[Eric Sproul]

Post by Matthew Ahrens

Post by Eric Sproul
Hi,
I'm confronted with an issue where I suspect that too many
async_destroy operations are going on at once,

The async destroy operations aren't processed all at once; zfs only works
on one at a time. So having one in progress would have the same
performance impact as having queued up many destroys.

That is good to know, and that means we don't need to worry about firing
off too many at once.

You could get the destroy queue via mdb or zdb, but unfortunately I don't

Post by Matthew Ahrens
think we keep any human-readable description of the destroy (e.g. name of
the filesystem that was destroyed). You can see which filesystems and
snapshots were destroyed recently with "zpool history [-i]". You can see
how much data remains to be destroyed with "zpool get freeing".

Shame on me for not reading the man page more carefully. :) That's very
cool.


You may be able to improve client write performance by reducing

Post by Matthew Ahrens
zfs_scan_min_time_ms (try halving or quartering it).

Thanks, I'll look into that. After I sent this mail, I found out that
dedup was also being used, and that the dedup ratio was quite low, around
1.2x. I believe this is a far more likely culprit for slow performance
during destroys than the destroy activity itself.

Post by Matthew Ahrens
On very recent bits, you may be able to improve client read performance by
reducing zfs_vdev_async_read_max_active (from 3 to 1).

We're not there yet, but I am looking forward to seeing how the new write
throttle performs. :)

Thanks Matt!
Eric



-------------------------------------------
illumos-zfs
Archives: https://www.listbox.com/member/archive/182191/=now
RSS Feed: https://www.listbox.com/member/archive/rss/182191/23047029-187a0c8d
Modify Your Subscription: https://www.listbox.com/member/?member_id=23047029&id_secret=23047029-2e85923f
Powered by Listbox: http://www.listbox.com