ZFS Improve N-way mirror performance

It builds upon the ZFSOnLinux changes. The ZFSOnLinux modifications
should be trivial to pull into Illumos. Porting the FreeBSD enhancements
to other platforms might be a little more difficult, but it can be done.

-------------------------------------------
illumos-zfs
Archives: https://www.listbox.com/member/archive/182191/=now
RSS Feed: https://www.listbox.com/member/archive/rss/182191/23047029-187a0c8d
Modify Your Subscription: https://www.listbox.com/member/?member_id=23047029&id_secret=23047029-2e85923f
Powered by Listbox: http://www.listbox.com

Alexander Motin

2013-10-25 15:28:45 UTC

Post by Richard Yao

Post by Bart Coddens
I saw this merge today on freebsd, it originated on zfsonlinux, could
this be possible on illumos ?
http://anzwix.com/a/FreeBSD/ImproveZFSNwayMirrorReadPerformanceByUsingLoadAn

It is quite different from ZoL changes in implemented logic. You may see
that results are quite different too. From FreeBSD-specific parts there
mostly mechanism to detect SSDs v/s HDDs. That in indeed
platform-specific part, but any platform that wants to do that kind of
advanced scheduling should probably have it.

--
Alexander Motin

Adam Leventhal

2013-10-25 16:24:43 UTC

Very impressive results. This would be a great addition to illumos.

Adam

Post by Bart Coddens
Hi all,
I saw this merge today on freebsd, it originated on zfsonlinux, could this
be possible on illumos ?
http://anzwix.com/a/FreeBSD/ImproveZFSNwayMirrorReadPerformanceByUsingLoadAn
It looks like a very interesting idea to me
Best Regards,
Bart Coddens
-------------------------------------------
illumos-zfs
Archives: https://www.listbox.com/member/archive/182191/=now
https://www.listbox.com/member/archive/rss/182191/22008099-95c33fdc
https://www.listbox.com/member/?&
Powered by Listbox: http://www.listbox.com

--
Adam Leventhal
CTO, Delphix
http://blog.delphix.com/ahl

Steven Hartland

2013-10-25 16:45:23 UTC

----- Original Message -----

Post by Adam Leventhal
Very impressive results. This would be a great addition to illumos.
Adam

I'll look at getting a webrev for illumos done once I've clear
some other backlogged changes I have pending.

If someone wants to give it a shot in the mean time feel free.

The core ZFS changes should be pretty easy to port only the
FreeBSD disk interface layer stuff which faciliate the
non-rotating optimsiation would need to be reworked.

Regards
Steve

================================================
This e.mail is private and confidential between Multiplay (UK) Ltd. and the person or entity to whom it is addressed. In the event of misdirection, the recipient is prohibited from using, copying, printing or otherwise disseminating it or any information contained in it.

In the event of misdirection, illegible or incomplete transmission please telephone +44 845 868 1337
or return the E.mail to ***@multiplay.co.uk.

Matthew Ahrens

2013-10-25 16:52:48 UTC

Yes, this would be great to get into illumos. It would be useful even
without the (more difficult to port) SSD detection logic.

--matt

Post by Bart Coddens
Hi all,
I saw this merge today on freebsd, it originated on zfsonlinux, could this
be possible on illumos ?
http://anzwix.com/a/FreeBSD/**ImproveZFSNwayMirrorReadPerfor**
manceByUsingLoadAn<http://anzwix.com/a/FreeBSD/ImproveZFSNwayMirrorReadPerformanceByUsingLoadAn>
It looks like a very interesting idea to me
Best Regards,
Bart Coddens
------------------------------**-------------
illumos-zfs
Archives: https://www.listbox.com/**member/archive/182191/=now<https://www.listbox.com/member/archive/182191/=now>
RSS Feed: https://www.listbox.com/**member/archive/rss/182191/**
21635000-ebd1d460<https://www.listbox.com/member/archive/rss/182191/21635000-ebd1d460>
Modify Your Subscription: https://www.listbox.com/**
member/?&id_**secret=21635000-73dc201a<https://www.listbox.com/member/?&>
Powered by Listbox: http://www.listbox.com

-------------------------------------------
illumos-zfs
Archives: https://www.listbox.com/member/archive/182191/=now
RSS Feed: https://www.listbox.com/member/archive/rss/182191/23047029-187a0c8d
Modify Your Subscription: https://www.listbox.com/member/?member_id=23047029&id_secret=23047029-2e85923f
Powered by Listbox: http://www.listbox.com

Jim Klimov

2013-10-25 17:14:08 UTC

Post by Matthew Ahrens
Yes, this would be great to get into illumos. It would be useful even
without the (more difficult to port) SSD detection logic.

Aren't there simple ways like the rotation-speed attribute which is
zero for non-rotating media?

Can some of the "old-timers" please clarify: I thought that the old SVM
mirroring (i.e. as used with UFS), and Linux MDADM, and as I thought
nearly any implementation of mirroring, took into consideration such
parameters as the recent IO location (which may have meant shorter
trips for the mechanical magnetic head, at least before the days that
disks began to queue stuff themselves). Not so sure about load though.

That is, the way I was taught about it "in principle", writes to legacy
mirrors always committed (or at least initiated) IOs to the "primary"
disk first, then the subsequent ones (hence the trust that the first
disk in such array is the golden standard in case of discrepancies),
and in absence of writes, random reads were served from different
disks, based on locations "closest" to the head for a particular disk
and thus minimizing latency the best they could.

Now I wonder... was that BS (or at least just-theoretical wishful
thinking about an ideal mirror), or were some RAID systems actually
engineered according to this principle? If yes, how come it was not
done in ZFS from the beginning? :)

Thanks,
//Jim Klimov

Keith Wesolowski

2013-10-25 17:32:47 UTC

Post by Matthew Ahrens
Yes, this would be great to get into illumos. It would be useful even
without the (more difficult to port) SSD detection logic.

Aren't there simple ways like the rotation-speed attribute which is
zero for non-rotating media?

Sam Zaydel

2013-10-25 18:49:00 UTC

It looks like the effect on Reads is significant, but what do these changes
do to mixed Read/Write workloads, especially when IO is mostly random?

On Fri, Oct 25, 2013 at 10:32 AM, Keith Wesolowski <

Post by Matthew Ahrens
Yes, this would be great to get into illumos. It would be useful even
without the (more difficult to port) SSD detection logic.

Aren't there simple ways like the rotation-speed attribute which is
zero for non-rotating media?

FWIW, we ship a version of I believe Garrett's old work here, and it
works well for us.
See
https://github.com/joyent/illumos-joyent/commit/281d692636513b749e11b75b758cd009114b2216.
There are some consumers in smartos-live.
-------------------------------------------
illumos-zfs
Archives: https://www.listbox.com/member/archive/182191/=now
https://www.listbox.com/member/archive/rss/182191/24342081-7731472e
https://www.listbox.com/member/?&
Powered by Listbox: http://www.listbox.com

--
Join the geek side, we have Ï!

Please feel free to connect with me on LinkedIn.
http://www.linkedin.com/in/samzaydel

-------------------------------------------
illumos-zfs
Archives: https://www.listbox.com/member/archive/182191/=now
RSS Feed: https://www.listbox.com/member/archive/rss/182191/23047029-187a0c8d
Modify Your Subscription: https://www.listbox.com/member/?member_id=23047029&id_secret=23047029-2e85923f
Powered by Listbox: http://www.listbox.com

Keith Wesolowski

2013-10-25 19:08:16 UTC

Post by Sam Zaydel
It looks like the effect on Reads is significant, but what do these changes
do to mixed Read/Write workloads, especially when IO is mostly random?

The change I'm pointing out is just detection. It has no effect on I/O.

Garrett D'Amore

2013-10-25 18:50:28 UTC

The DKIOCSOLIDSTATE work definitely looks like mine. :-)

At the time I was using it to tune the zfs_maxpending per vdev. But as
that tunable is now gone thanks to the rewrite of the write-throttle logic,
I guess it isn't quite as important.

I'd still like to see the DKIOCSOLIDSTATE work go back upstream though.
Ultimately I think this may have other benefits -- for example, it makes
sense to disable the disksort algorithm for such devices -- sorting only
has the effect of making accesses go out of order, and introducing
variability into an I/O stream that would otherwise probably be more
deterministic.

On Fri, Oct 25, 2013 at 10:32 AM, Keith Wesolowski <

Post by Matthew Ahrens
Yes, this would be great to get into illumos. It would be useful even
without the (more difficult to port) SSD detection logic.

Aren't there simple ways like the rotation-speed attribute which is
zero for non-rotating media?

FWIW, we ship a version of I believe Garrett's old work here, and it
works well for us.
See
https://github.com/joyent/illumos-joyent/commit/281d692636513b749e11b75b758cd009114b2216.
There are some consumers in smartos-live.
-------------------------------------------
illumos-zfs
Archives: https://www.listbox.com/member/archive/182191/=now
https://www.listbox.com/member/archive/rss/182191/22035932-85c5d227
https://www.listbox.com/member/?&
Powered by Listbox: http://www.listbox.com

Steven Hartland

2013-10-25 22:12:52 UTC

----- Original Message -----
From: "Garrett D'Amore" <***@damore.org>
To: <***@lists.illumos.org>
Sent: Friday, October 25, 2013 7:50 PM
Subject: Re: [zfs] ZFS Improve N-way mirror performance

Post by Garrett D'Amore
The DKIOCSOLIDSTATE work definitely looks like mine. :-)
At the time I was using it to tune the zfs_maxpending per vdev. But as
that tunable is now gone thanks to the rewrite of the write-throttle logic,
I guess it isn't quite as important.
I'd still like to see the DKIOCSOLIDSTATE work go back upstream though.
Ultimately I think this may have other benefits -- for example, it makes
sense to disable the disksort algorithm for such devices -- sorting only
has the effect of making accesses go out of order, and introducing
variability into an I/O stream that would otherwise probably be more
deterministic.

Richard Elling

2013-10-25 19:29:51 UTC

Yes, I watched Garrett hack this out on a flight from Moscow to LAX. The SSD detection in
illumos is pretty reliable already.

The bigger question is: how badly do we want device-specific information to propagate
into the ZFS logic? Historically, this hasn't worked out well for file systems. I think we should
be careful to not make assumptions about "disks," whether they rotate or not. Rather, we
need to understand queues, caches, and the costs associated with them. In other words,
the problem space is much larger than just HDDs vs Flash SSDs.
-- richard

Post by Matthew Ahrens
Yes, this would be great to get into illumos. It would be useful even
without the (more difficult to port) SSD detection logic.

Aren't there simple ways like the rotation-speed attribute which is
zero for non-rotating media?

FWIW, we ship a version of I believe Garrett's old work here, and it
works well for us.
See
https://github.com/joyent/illumos-joyent/commit/281d692636513b749e11b75b758cd009114b2216. There are some consumers in smartos-live.
-------------------------------------------
illumos-zfs
Archives: https://www.listbox.com/member/archive/182191/=now
RSS Feed: https://www.listbox.com/member/archive/rss/182191/22820713-4fad4b89
Modify Your Subscription: https://www.listbox.com/member/?&
Powered by Listbox: http://www.listbox.com

Steven Hartland

2013-10-25 22:20:26 UTC

While the logic about just queuing depth provided a good improvement,
adding the rotation penalty logic resulted in a noticable performance
improvement in my mixed device testing. So this sort of information
about the devices definitely provides benefits.

We could take the logic further and building a huristic information
about a devices performance over time, which would enable us to
provide better algorithums for performing device operations but this
would only really benefit mixed device performance enviroment which
I'm not sure is that prevelent.

What do people thing?

Regards
Steve
----- Original Message -----
From: "Richard Elling" <***@gmail.com>
To: <***@lists.illumos.org>
Sent: Friday, October 25, 2013 8:29 PM
Subject: Re: [zfs] ZFS Improve N-way mirror performance

Yes, I watched Garrett hack this out on a flight from Moscow to LAX. The SSD detection in
illumos is pretty reliable already.

The bigger question is: how badly do we want device-specific information to propagate
into the ZFS logic? Historically, this hasn't worked out well for file systems. I think we should
be careful to not make assumptions about "disks," whether they rotate or not. Rather, we
need to understand queues, caches, and the costs associated with them. In other words,
the problem space is much larger than just HDDs vs Flash SSDs.
-- richard

Post by Matthew Ahrens
Yes, this would be great to get into illumos. It would be useful even
without the (more difficult to port) SSD detection logic.

Aren't there simple ways like the rotation-speed attribute which is
zero for non-rotating media?

FWIW, we ship a version of I believe Garrett's old work here, and it
works well for us.
See
https://github.com/joyent/illumos-joyent/commit/281d692636513b749e11b75b758cd009114b2216. There are some consumers in
smartos-live.
-------------------------------------------
illumos-zfs
Archives: https://www.listbox.com/member/archive/182191/=now
RSS Feed: https://www.listbox.com/member/archive/rss/182191/22820713-4fad4b89
Modify Your Subscription: https://www.listbox.com/member/?&
Powered by Listbox: http://www.listbox.com

--

***@RichardElling.com
+1-760-896-4422

-------------------------------------------
illumos-zfs
Archives: https://www.listbox.com/member/archive/182191/=now
RSS Feed: https://www.listbox.com/member/archive/rss/182191/24401717-fdfe502b
Modify Your Subscription: https://www.listbox.com/member/?&
Powered by Listbox: http://www.listbox.com

================================================
This e.mail is private and confidential between Multiplay (UK) Ltd. and the person or entity to whom it is addressed. In the event of misdirection, the recipient is prohibited from using, copying, printing or otherwise disseminating it or any information contained in it.

In the event of misdirection, illegible or incomplete transmission please telephone +44 845 868 1337
or return the E.mail to ***@multiplay.co.uk.

Karl Wagner

2013-10-25 22:29:03 UTC

Only my personal opinion, but a mixed environment is something I have
considered in non-zfs systems. For example, mdraid has a write mostly
setting, which would be perfect for an ssd/HDD setup. You get the (read)
speed of an ssd without wasting money on a second one as a mirror.

I never considered it for zfs as I know (in it's current form) it wouldn't
have the desired benefit.

On a side note, I think a write mostly seeing would come in handy too.

Post by Steven Hartland
While the logic about just queuing depth provided a good improvement,
adding the rotation penalty logic resulted in a noticable performance
improvement in my mixed device testing. So this sort of information
about the devices definitely provides benefits.
We could take the logic further and building a huristic information
about a devices performance over time, which would enable us to
provide better algorithums for performing device operations but this
would only really benefit mixed device performance enviroment which
I'm not sure is that prevelent.
What do people thing?
Regards
Steve
----- Original Message ----- From: "Richard Elling" <
Sent: Friday, October 25, 2013 8:29 PM
Subject: Re: [zfs] ZFS Improve N-way mirror performance
Yes, I watched Garrett hack this out on a flight from Moscow to LAX. The SSD detection in
illumos is pretty reliable already.
The bigger question is: how badly do we want device-specific information to propagate
into the ZFS logic? Historically, this hasn't worked out well for file
systems. I think we should
be careful to not make assumptions about "disks," whether they rotate or not. Rather, we
need to understand queues, caches, and the costs associated with them. In other words,
the problem space is much larger than just HDDs vs Flash SSDs.
-- richard
On Oct 25, 2013, at 10:32 AM, Keith Wesolowski <

Post by Matthew Ahrens
Yes, this would be great to get into illumos. It would be useful even
without the (more difficult to port) SSD detection logic.

Aren't there simple ways like the rotation-speed attribute which is
zero for non-rotating media?

FWIW, we ship a version of I believe Garrett's old work here, and it
works well for us.
See
https://github.com/joyent/**illumos-joyent/commit/**
281d692636513b749e11b75b758cd0**09114b2216<https://github.com/joyent/illumos-joyent/commit/281d692636513b749e11b75b758cd009114b2216>.
There are some consumers in smartos-live.
------------------------------**-------------
illumos-zfs
Archives: https://www.listbox.com/**member/archive/182191/=now<https://www.listbox.com/member/archive/182191/=now>
RSS Feed: https://www.listbox.com/**member/archive/rss/182191/**
22820713-4fad4b89<https://www.listbox.com/member/archive/rss/182191/22820713-4fad4b89>
Modify Your Subscription: https://www.listbox.com/**member/?&<https://www.listbox.com/member/?&>
Powered by Listbox: http://www.listbox.com

--
+1-760-896-4422
------------------------------**-------------
illumos-zfs
Archives: https://www.listbox.com/**member/archive/182191/=now<https://www.listbox.com/member/archive/182191/=now>
RSS Feed: https://www.listbox.com/**member/archive/rss/182191/**
24401717-fdfe502b<https://www.listbox.com/member/archive/rss/182191/24401717-fdfe502b>
Modify Your Subscription: https://www.listbox.com/**member/?&<https://www.listbox.com/member/?&>
Powered by Listbox: http://www.listbox.com
==============================**==================
This e.mail is private and confidential between Multiplay (UK) Ltd. and
the person or entity to whom it is addressed. In the event of misdirection,
the recipient is prohibited from using, copying, printing or otherwise
disseminating it or any information contained in it.
In the event of misdirection, illegible or incomplete transmission please
telephone +44 845 868 1337
------------------------------**-------------
illumos-zfs
Archives: https://www.listbox.com/**member/archive/182191/=now<https://www.listbox.com/member/archive/182191/=now>
RSS Feed: https://www.listbox.com/**member/archive/rss/182191/**
24409195-16edb367<https://www.listbox.com/member/archive/rss/182191/24409195-16edb367>
Modify Your Subscription: https://www.listbox.com/**
member/?&id_**secret=24409195-ef4deb50<https://www.listbox.com/member/?&>
Powered by Listbox: http://www.listbox.com

Richard Elling

2013-10-25 23:33:56 UTC

Yes, this is what I expect. It is unlikely that queue depth is the right variable
to base a decision upon -- some devices are faster with more queued I/Os.

Post by Steven Hartland
We could take the logic further and building a huristic information
about a devices performance over time, which would enable us to
provide better algorithums for performing device operations but this
would only really benefit mixed device performance enviroment which
I'm not sure is that prevelent.

Yes. Also note that writes and reads have asymmetric response times. This is
true for HDDs (writes tend to complete faster than reads) and Flash SSDs (writes
tend to complete slower than reads) but isn't true for RAM SSDs where writes
and reads complete quickly and predictably. Obviously, caches, transport latency,
and RAID controllers further complicate the predictions.

Post by Steven Hartland
What do people thing?

Plenty of fertile ground for research ;-)
-- richard

Post by Steven Hartland
Regards
Steve
Sent: Friday, October 25, 2013 8:29 PM
Subject: Re: [zfs] ZFS Improve N-way mirror performance
Yes, I watched Garrett hack this out on a flight from Moscow to LAX. The SSD detection in
illumos is pretty reliable already.
The bigger question is: how badly do we want device-specific information to propagate
into the ZFS logic? Historically, this hasn't worked out well for file systems. I think we should
be careful to not make assumptions about "disks," whether they rotate or not. Rather, we
need to understand queues, caches, and the costs associated with them. In other words,
the problem space is much larger than just HDDs vs Flash SSDs.
-- richard

Post by Matthew Ahrens
Yes, this would be great to get into illumos. It would be useful even
without the (more difficult to port) SSD detection logic.

Aren't there simple ways like the rotation-speed attribute which is
zero for non-rotating media?

FWIW, we ship a version of I believe Garrett's old work here, and it
works well for us.
See
https://github.com/joyent/illumos-joyent/commit/281d692636513b749e11b75b758cd009114b2216. There are some consumers in smartos-live.
-------------------------------------------
illumos-zfs
Archives: https://www.listbox.com/member/archive/182191/=now
RSS Feed: https://www.listbox.com/member/archive/rss/182191/22820713-4fad4b89
Modify Your Subscription: https://www.listbox.com/member/?&
Powered by Listbox: http://www.listbox.com

--
+1-760-896-4422
-------------------------------------------
illumos-zfs
Archives: https://www.listbox.com/member/archive/182191/=now
RSS Feed: https://www.listbox.com/member/archive/rss/182191/24401717-fdfe502b
Modify Your Subscription: https://www.listbox.com/member/?&
Powered by Listbox: http://www.listbox.com
================================================
This e.mail is private and confidential between Multiplay (UK) Ltd. and the person or entity to whom it is addressed. In the event of misdirection, the recipient is prohibited from using, copying, printing or otherwise disseminating it or any information contained in it.
In the event of misdirection, illegible or incomplete transmission please telephone +44 845 868 1337
-------------------------------------------
illumos-zfs
Archives: https://www.listbox.com/member/archive/182191/=now
RSS Feed: https://www.listbox.com/member/archive/rss/182191/22820713-4fad4b89
Modify Your Subscription: https://www.listbox.com/member/?&
Powered by Listbox: http://www.listbox.com

Paul Kraus

2013-10-28 18:22:50 UTC

Maybe I'm being naive here (I am much more an admin than a developer), but a system that tunes itself and is NOT based on specific device configuration would, in my opinion, be the most useful. It could adapt automatically to new technology, changes (improvements) to existing technology, and even changes as devices age. Best of all, it would not need an admin to specify a special parameter to enable the improvement.

So I like the idea of gathering statistics about devices and apply changes to behavior to improve performance. The downside is that performance will not be very well defined as the system adapts over time, but a baseline should be achievable.

--
Paul Kraus
Deputy Technical Director, LoneStarCon 3
Sound Coordinator, Schenectady Light Opera Company

Paul Kraus

2013-10-27 19:24:54 UTC

Post by Jim Klimov
That is, the way I was taught about it "in principle", writes to legacy
mirrors always committed (or at least initiated) IOs to the "primary"
disk first, then the subsequent ones (hence the trust that the first
disk in such array is the golden standard in case of discrepancies),
and in absence of writes, random reads were served from different
disks, based on locations "closest" to the head for a particular disk
and thus minimizing latency the best they could.

Sun's software disk volume technology let you configure read behavior. IIRC the options were:

1. read from primary until it fails
2. round robin from all drives
3. geometric (which did sort of what you said)

My understanding of geometric was that it divided the logical volume into N sections, where N == the number of disks in the mirror, then assigned reads for one particular section to one drive. This had the effect (for read heavy loads) in a 3 way mirror (for example) of reading from the first 1/3 of the 1st drive, the second 1/3 of the 2nd drive, and the last 1/3 of the 3rd drive. Reducing seek times as you were seeking over only 1/3 of the entire disk.

--
Paul Kraus
Deputy Technical Director, LoneStarCon 3
Sound Coordinator, Schenectady Light Opera Company

Jim Klimov

2013-10-25 18:47:26 UTC

Post by Matthew Ahrens
Yes, this would be great to get into illumos. It would be useful even
without the (more difficult to port) SSD detection logic.

On second thought, perhaps you're right. Let's say that our max-pending
queue was 5 requests per device. All components in the mirror (2 HDDs
and 1 SSD in the example above) get the five requests equally. However,
the SSD has no rotational latency and completes faster - and thus is
available for any new incoming request (per "Load of the vdevs (number
of outstanding I/O requests)" newly introduced metric). And while this
surprisingly fast device processes the new batch(es) of read IOs, the
HDDs would return from their first five tasks.

Indeed, this can help even without explicit knowledge of SSD vs. HDD...

My 2c,
//Jim

Matthew Ahrens

2013-10-25 20:22:30 UTC

Post by Matthew Ahrens
Yes, this would be great to get into illumos. It would be useful even
without the (more difficult to port) SSD detection logic.

On second thought, perhaps you're right. Let's say that our max-pending
queue was 5 requests per device. All components in the mirror (2 HDDs
and 1 SSD in the example above) get the five requests equally. However,
the SSD has no rotational latency and completes faster - and thus is
available for any new incoming request (per "Load of the vdevs (number
of outstanding I/O requests)" newly introduced metric). And while this
surprisingly fast device processes the new batch(es) of read IOs, the
HDDs would return from their first five tasks.
Indeed, this can help even without explicit knowledge of SSD vs. HDD...

Yes, exactly.

--matt

-------------------------------------------
illumos-zfs
Archives: https://www.listbox.com/member/archive/182191/=now
RSS Feed: https://www.listbox.com/member/archive/rss/182191/23047029-187a0c8d
Modify Your Subscription: https://www.listbox.com/member/?member_id=23047029&id_secret=23047029-2e85923f
Powered by Listbox: http://www.listbox.com

Saso Kiselkov

2013-10-25 22:13:41 UTC

I have a port of this on Illumos, but I didn't have the time to test it
yet. I'll upload it some time in the next two weeks or so for review.

Cheers,

--
Saso

j***@cos.ru

2013-10-28 07:21:57 UTC

Â I wonder if instead of a "HDD vs SSD" flag there should be some lower-to-earth ones, such as lack of offset-related latency or presence of wear due to writes, etc - things we have special handling for? These two bits for example happen to intersect in ssd's, but wear is not a problem for ddrdrives. I think this would help with optimal use of some present and future storage devices.

Typos courtesy of my Samsung Mobile

-------- ÐÑÑÐŸÐŽÐœÐŸÐµ ÑÐŸÐŸÐ±ÑÐµÐœÐžÐµ --------
ÐÑ: Steven Hartland <***@multiplay.co.uk>
ÐÐ°ÑÐ°: 2013.10.26 0:12 (GMT+01:00)
ÐÐŸÐŒÑ: ***@lists.illumos.org
Ð¢ÐµÐŒÐ°: Re: [zfs] ZFS Improve N-way mirror performance

----- Original Message -----
From: "Garrett D'Amore" <***@damore.org>
To: <***@lists.illumos.org>
Sent: Friday, October 25, 2013 7:50 PM
Subject: Re: [zfs] ZFS Improve N-way mirror performance

Post by Garrett D'Amore
The DKIOCSOLIDSTATE work definitely looks like mine. :-)
At the time I was using it to tune the zfs_maxpending per vdev.Â Â But as
that tunable is now gone thanks to the rewrite of the write-throttle logic,
I guess it isn't quite as important.
I'd still like to see the DKIOCSOLIDSTATE work go back upstream though.
Ultimately I think this may have other benefits -- for example, it makes
sense to disable the disksort algorithm for such devices -- sorting only
has the effect of making accesses go out of order, and introducing
variability into an I/O stream that would otherwise probably be more
deterministic.

Kristoffer Sheather @ CloudCentral

2013-10-28 07:26:28 UTC

How about tagging devices with properties in a pre-defined schema in a fashion like ZFS datasets.

zpool set <poolname>/<devicename> type=ssd|hdd|ram
zpool set <poolname>/<devicename> property1=xyz
zpool set <poolname>/<devicename> propertyN=123

Then the ZFS IO system can optimise IO access dependant upon these completely optional properties. This would also work for a self-learning system which is able to detect the device properties and set same properties automagically.

----------------------------------------
From: "***@cos.ru" <***@cos.ru>
Sent: Monday, October 28, 2013 6:22 PM
To: ***@lists.illumos.org
Subject: Re: [zfs] ZFS Improve N-way mirror performance

I wonder if instead of a "HDD vs SSD" flag there should be some lower-to-earth ones, such as lack of offset-related latency or presence of wear due to writes, etc - things we have special handling for? These two bits for example happen to intersect in ssd's, but wear is not a problem for ddrdrives. I think this would help with optimal use of some present and future storage devices.

Typos courtesy of my Samsung Mobile

-------- ???????? ????????? --------
??: Steven Hartland <***@multiplay.co.uk>
????: 2013.10.26 0:12 (GMT+01:00)
????: ***@lists.illumos.org
????: Re: [zfs] ZFS Improve N-way mirror performance

----- Original Message -----
From: "Garrett D'Amore" <***@damore.org>
To: <***@lists.illumos.org>
Sent: Friday, October 25, 2013 7:50 PM
Subject: Re: [zfs] ZFS Improve N-way mirror performance

Steven Hartland

2013-10-28 10:19:52 UTC

You don't want the admin to have to do this, it needs to be
automatically determined from the device IMO.

Regards
Steve
----- Original Message -----
From: "Kristoffer Sheather @ CloudCentral" <***@cloudcentral.com.au>
To: <***@lists.illumos.org>
Sent: Monday, October 28, 2013 7:26 AM
Subject: Re: [zfs] ZFS Improve N-way mirror performance

How about tagging devices with properties in a pre-defined schema in a fashion like ZFS datasets.

zpool set <poolname>/<devicename> type=ssd|hdd|ram
zpool set <poolname>/<devicename> property1=xyz
zpool set <poolname>/<devicename> propertyN=123

Then the ZFS IO system can optimise IO access dependant upon these completely optional properties. This would also work for a
self-learning system which is able to detect the device properties and set same properties automagically.

----------------------------------------
From: "***@cos.ru" <***@cos.ru>
Sent: Monday, October 28, 2013 6:22 PM
To: ***@lists.illumos.org
Subject: Re: [zfs] ZFS Improve N-way mirror performance

I wonder if instead of a "HDD vs SSD" flag there should be some lower-to-earth ones, such as lack of offset-related latency or
presence of wear due to writes, etc - things we have special handling for? These two bits for example happen to intersect in
ssd's, but wear is not a problem for ddrdrives. I think this would help with optimal use of some present and future storage
devices.

Typos courtesy of my Samsung Mobile

-------- ???????? ????????? --------
??: Steven Hartland <***@multiplay.co.uk>
????: 2013.10.26 0:12 (GMT+01:00)
????: ***@lists.illumos.org
????: Re: [zfs] ZFS Improve N-way mirror performance

----- Original Message -----
From: "Garrett D'Amore" <***@damore.org>
To: <***@lists.illumos.org>
Sent: Friday, October 25, 2013 7:50 PM
Subject: Re: [zfs] ZFS Improve N-way mirror performance

I did exactly that in FreeBSD a while back and it had a great
benefit for high I/O request throughput.

I'd say DKIOCSOLIDSTATE information could easily be directly used as
either a basis of how to add the rotational check or even as a
replacement as far as rotational check is concerned as it essentially
represents the same detail about the disk.

Regards
Steve

================================================
This e.mail is private and confidential between Multiplay (UK) Ltd. and the person or entity to whom it is addressed. In the event
of misdirection, the recipient is prohibited from using, copying, printing or otherwise disseminating it or any information
contained in it.

In the event of misdirection, illegible or incomplete transmission please telephone +44 845 868 1337
or return the E.mail to ***@multiplay.co.uk.

-------------------------------------------
illumos-zfs
Archives: https://www.listbox.com/member/archive/182191/=now
RSS Feed: https://www.listbox.com/member/archive/rss/182191/22497542-d75cd9d9
Modify Your Subscription: https://www.listbox.com/member/?&
Powered by Listbox: http://www.listbox.com

illumos-zfs | Archives | Modify Your Subscription

-------------------------------------------
illumos-zfs
Archives: https://www.listbox.com/member/archive/182191/=now
RSS Feed: https://www.listbox.com/member/archive/rss/182191/24401717-fdfe502b
Modify Your Subscription: https://www.listbox.com/member/?&
Powered by Listbox: http://www.listbox.com

================================================
This e.mail is private and confidential between Multiplay (UK) Ltd. and the person or entity to whom it is addressed. In the event of misdirection, the recipient is prohibited from using, copying, printing or otherwise disseminating it or any information contained in it.

In the event of misdirection, illegible or incomplete transmission please telephone +44 845 868 1337
or return the E.mail to ***@multiplay.co.uk.

Garrett D'Amore

2013-10-28 17:55:34 UTC

No. Because the determination is not based on *pool* but on vdev.

Post by Kristoffer Sheather @ CloudCentral
How about tagging devices with properties in a pre-defined schema in a
fashion like ZFS datasets.
zpool set <poolname>/<devicename> type=ssd|hdd|ram
zpool set <poolname>/<devicename> property1=xyz
zpool set <poolname>/<devicename> propertyN=123
Then the ZFS IO system can optimise IO access dependant upon these
completely optional properties. This would also work for a self-learning
system which is able to detect the device properties and set same
properties automagically.
------------------------------
*Sent*: Monday, October 28, 2013 6:22 PM
*Subject*: Re: [zfs] ZFS Improve N-way mirror performance
I wonder if instead of a "HDD vs SSD" flag there should be some
lower-to-earth ones, such as lack of offset-related latency or presence of
wear due to writes, etc - things we have special handling for? These two
bits for example happen to intersect in ssd's, but wear is not a problem
for ddrdrives. I think this would help with optimal use of some present and
future storage devices.
Typos courtesy of my Samsung Mobile
-------- ???????? ????????? --------
????: 2013.10.26 0:12 (GMT+01:00)
????: Re: [zfs] ZFS Improve N-way mirror performance
----- Original Message -----
Sent: Friday, October 25, 2013 7:50 PM
Subject: Re: [zfs] ZFS Improve N-way mirror performance

logic,

Post by Garrett D'Amore
I guess it isn't quite as important.
I'd still like to see the DKIOCSOLIDSTATE work go back upstream though.
Ultimately I think this may have other benefits -- for example, it makes
sense to disable the disksort algorithm for such devices -- sorting only
has the effect of making accesses go out of order, and introducing
variability into an I/O stream that would otherwise probably be more
deterministic.

Kristoffer Sheather @ CloudCentral

2013-10-28 10:22:07 UTC

You can do either, certainly it would be simplest to implement manually
first, then autonomically later. Either way the same schema could be used.

----------------------------------------
From: "Steven Hartland" <***@multiplay.co.uk>
Sent: Monday, October 28, 2013 9:20 PM
To: ***@lists.illumos.org
Subject: Re: [zfs] ZFS Improve N-way mirror performance

You don't want the admin to have to do this, it needs to be
automatically determined from the device IMO.

Regards
Steve
----- Original Message -----
From: "Kristoffer Sheather @ CloudCentral"
<***@cloudcentral.com.au>
To: <***@lists.illumos.org>
Sent: Monday, October 28, 2013 7:26 AM
Subject: Re: [zfs] ZFS Improve N-way mirror performance

How about tagging devices with properties in a pre-defined schema in a
fashion like ZFS datasets.

zpool set <poolname>/<devicename> type=ssd|hdd|ram
zpool set <poolname>/<devicename> property1=xyz
zpool set <poolname>/<devicename> propertyN=123

Then the ZFS IO system can optimise IO access dependant upon these
completely optional properties. This would also work for a
self-learning system which is able to detect the device properties and set
same properties automagically.

----------------------------------------
From: "***@cos.ru" <***@cos.ru>
Sent: Monday, October 28, 2013 6:22 PM
To: ***@lists.illumos.org
Subject: Re: [zfs] ZFS Improve N-way mirror performance

I wonder if instead of a "HDD vs SSD" flag there should be some
lower-to-earth ones, such as lack of offset-related latency or
presence of wear due to writes, etc - things we have special handling for?
These two bits for example happen to intersect in
ssd's, but wear is not a problem for ddrdrives. I think this would help
with optimal use of some present and future storage
devices.

Typos courtesy of my Samsung Mobile

-------- ???????? ????????? --------
??: Steven Hartland <***@multiplay.co.uk>
????: 2013.10.26 0:12 (GMT+01:00)
????: ***@lists.illumos.org
????: Re: [zfs] ZFS Improve N-way mirror performance

----- Original Message -----
From: "Garrett D'Amore" <***@damore.org>
To: <***@lists.illumos.org>
Sent: Friday, October 25, 2013 7:50 PM
Subject: Re: [zfs] ZFS Improve N-way mirror performance

logic,

I did exactly that in FreeBSD a while back and it had a great
benefit for high I/O request throughput.

I'd say DKIOCSOLIDSTATE information could easily be directly used as
either a basis of how to add the rotational check or even as a
replacement as far as rotational check is concerned as it essentially
represents the same detail about the disk.

Regards
Steve

================================================
This e.mail is private and confidential between Multiplay (UK) Ltd. and the
person or entity to whom it is addressed. In the event
of misdirection, the recipient is prohibited from using, copying, printing
or otherwise disseminating it or any information
contained in it.

In the event of misdirection, illegible or incomplete transmission please
telephone +44 845 868 1337
or return the E.mail to ***@multiplay.co.uk.

-------------------------------------------
illumos-zfs
Archives: https://www.listbox.com/member/archive/182191/=now
RSS Feed:
https://www.listbox.com/member/archive/rss/182191/22497542-d75cd9d9
Modify Your Subscription: https://www.listbox.com/member/?&
Powered by Listbox: http://www.listbox.com

illumos-zfs | Archives | Modify Your Subscription

-------------------------------------------
illumos-zfs
Archives: https://www.listbox.com/member/archive/182191/=now
RSS Feed:
https://www.listbox.com/member/archive/rss/182191/24401717-fdfe502b
Modify Your Subscription: https://www.listbox.com/member/?&
Powered by Listbox: http://www.listbox.com

================================================
This e.mail is private and confidential between Multiplay (UK) Ltd. and the
person or entity to whom it is addressed. In the event of misdirection, the
recipient is prohibited from using, copying, printing or otherwise
disseminating it or any information contained in it.

In the event of misdirection, illegible or incomplete transmission please
telephone +44 845 868 1337
or return the E.mail to ***@multiplay.co.uk.

-------------------------------------------
illumos-zfs
Archives: https://www.listbox.com/member/archive/182191/=now
RSS Feed:
https://www.listbox.com/member/archive/rss/182191/23629987-2afa167a
Modify Your Subscription:
https://www.listbox.com/member/?&
a8
Powered by Listbox: http://www.listbox.com

-------------------------------------------
illumos-zfs
Archives: https://www.listbox.com/member/archive/182191/=now
RSS Feed: https://www.listbox.com/member/archive/rss/182191/23047029-187a0c8d
Modify Your Subscription: https://www.listbox.com/member/?member_id=23047029&id_secret=23047029-2e85923f
Powered by Listbox: http://www.listbox.com

Kristoffer Sheather @ CloudCentral

2013-10-28 18:25:40 UTC

Agreed, hence why I had <pool>/<devicename>.

----------------------------------------
From: "Garrett D'Amore" <***@damore.org>
Sent: Tuesday, October 29, 2013 4:56 AM
To: "***@lists.illumos.org" <***@lists.illumos.org>
Subject: Re: [zfs] ZFS Improve N-way mirror performance

No. Because the determination is not based on *pool* but on vdev.

On Mon, Oct 28, 2013 at 12:26 AM, Kristoffer Sheather @ CloudCentral
<***@cloudcentral.com.au> wrote:
How about tagging devices with properties in a pre-defined schema in a
fashion like ZFS datasets.

zpool set <poolname>/<devicename> type=ssd|hdd|ram
zpool set <poolname>/<devicename> property1=xyz
zpool set <poolname>/<devicename> propertyN=123

Then the ZFS IO system can optimise IO access dependant upon these
completely optional properties. This would also work for a self-learning
system which is able to detect the device properties and set same
properties automagically.

----------------------------------------
From: "***@cos.ru" <***@cos.ru>
Sent: Monday, October 28, 2013 6:22 PM
To: ***@lists.illumos.org
Subject: Re: [zfs] ZFS Improve N-way mirror performance

I wonder if instead of a "HDD vs SSD" flag there should be some
lower-to-earth ones, such as lack of offset-related latency or presence of
wear due to writes, etc - things we have special handling for? These two
bits for example happen to intersect in ssd's, but wear is not a problem
for ddrdrives. I think this would help with optimal use of some present and
future storage devices.

Typos courtesy of my Samsung Mobile

-------- ???????? ????????? --------
??: Steven Hartland <***@multiplay.co.uk>
????: 2013.10.26 0:12 (GMT+01:00)
????: ***@lists.illumos.org
????: Re: [zfs] ZFS Improve N-way mirror performance

----- Original Message -----
From: "Garrett D'Amore" <***@damore.org>
To: <***@lists.illumos.org>
Sent: Friday, October 25, 2013 7:50 PM
Subject: Re: [zfs] ZFS Improve N-way mirror performance

logic,

makes

Post by Garrett D'Amore
sense to disable the disksort algorithm for such devices -- sorting only
has the effect of making accesses go out of order, and introducing
variability into an I/O stream that would otherwise probably be more
deterministic.

Garrett D'Amore

2013-10-28 18:32:21 UTC