Post by Andrew GallowayTo answer your subject question: in a word, yes.
It is exceptionally rare to run into a use-case that validly needs
separate pools, but wouldn't also then be better done with each pool
running on a different box. I can't recall the last time I actively
suggested such a thing to a customer. I am pretty keen on one pool per
system.
I disagree. For example, an Oracle database might store transaction logs
on an all flash pool, but leave other data in a more conventional spinning
media pool. Unless all your consumers on a given box are of the same class
and have identical storage needs, then separate pools are definitely
reasonable.
Not only is this in and of itself actually not a common use-case, but it is
a great example of a specific requirement that proves the general rule. You
may indeed find it wise to create a separate, faster disk (or SSD) pool to
handle the logs while putting the rest onto a separate slower pool. You
might also put that pool on another box, however, if you're of such size
where such a thing is even worth the effort and expense in the first place,
though I admit not always. A good specific example to prove the general
rule. You'd also still only be left with 2 real data pools on the system,
and hopefully with only one doing a lot of the work; if the other is super
busy, I go back to 'multiple systems'. Not 10, for example.
Post by Andrew GallowayIn a bigger box with multitenancy, using separate storage pools can give
you a different fault isolation boundary, allowing some services to stay
up, while others tank. Again, this is more true when you can indicate you
have different data class types: for example a pool backing a Swift cluster
node needs no redundancy (Swift handles redundancy at the application
layer), whereas a pool backing for NFS needs a bit more care.
In almost every case where I see this cited as a reason for breaking the
pool, I again go to 'and on to another system, too'. Nothing wrong with
having lots of pools -- each on its own system. :) -- We have a working
example here of a university customer with many pools on one storage box,
built in this manner for I believe this specific reason, and to this day (a
multi-year issue) it still causes headaches. Would not do again. :)
Post by Andrew GallowayAllowing later expansion is another reason to do multipools. While a
stripe of mirrors is easy to manage, upgrading such a strip is harder the
larger the stripe. And, there may be performance implications from
extremely wide stripes. And RAID-Z is effectively non-upgradeable, but
adding more RAID-Z pools is a cheap and easy way to quickly add storage.
Sure, but not really advisable. What is academically plausible and even
technically sane sounding is not always what production finds to be a good
idea. Growing a storage system with a 24-disk raidz2 pool of 3 x 8 disk
vdevs by adding a whole new pool of 6 disks in a raidz1, and then later
again in another pool with 12 disks in a raidz3, and then agai -- yeah. No.
Putting on my zfs sysadmin hat, I say this is insane. If you /must/ grow
that pool, add more 8-disk raidz2 vdevs to the existing pool, with the
understanding it may take nigh on forever to actually balance properly -
but hey, at least there's a chance it might.. a new pool never will. Plus,
if I need to grow, there's some chance I need to grow an existing dataset..
one that might be too big to fit on a smaller new pool.
Post by Andrew GallowayThis becomes all the more important in the face of SANs, where the number
of spindles available to a single chassis may be enormous.
Often, single chassis with enormous number of disks is a mistake. Scale
out. Not up. The main area where this is not necessarily true is archival
-- where multiple pools would likely be completely pointless.
Post by Andrew GallowayIn addition to the comments so far and the things you witnessed, most zfs
tunables are (regrettably) global, and cannot be applied on a per-pool
basis, further impacting the supportability and efficiency of multi-pool
boxes. There are many reasons not to multi-pool on the same box, and very
few reasons to do so.
Some of the tunables are indeed global, but not all of them. And there is
work underway to further allow more fine grained tuning.
I know, and that will go a long way to increasing the list of use-cases
that may be OK to apply multiple pools on a single system to work with.
However, what is today and what will be in a year are not the same thing. :)
Post by Andrew GallowayI think you have a very narrow view of how storage is used, and I would be
cautious about applying this particular advice without a careful
understanding of the problem(s) you are trying to solve. For a simple
small configuration a single pool is often (perhaps even usually) best.
Its far from true that this advice should be taken universally though.
At the risk of responding to what appears to be something close to an ad
hominem attack, I like to think I have an extremely broad view of how
storage is used, as you should know, as you know what I do and where, and
with the sort of clients I am commonly engaged. I have been on many
hundreds, if not thousands, of /production/ ZFS SAN appliances of all
shapes, sizes and colors.
I can find you lots of exceptions for any general rule, but that doesn't
mean the general rule isn't generally true, and generally speaking,
multiple pools is something to avoid until you're reasonably assured your
use-case requires them and that they're not insane to do. Such specific
exceptions certainly exist. They're definitely exceptions, however.
Post by Andrew GallowayNow, that said, the configuration described below (6 pools of two disks
mirrored) seems
iffy. I suspect that this is not an ideal configuration,
but I won't say for certain without understanding more carefully the
underlying reasons for that configuration.
I think I've made it clear I think 'iffy' is a little weak a description
for such a thing. I can think of no reason to split at such a small scale,
especially when the split pools are comprised of identical drives, hit by
identical clients (as Attila already described). :)
Post by Andrew GallowayOne thing that is true, using small pools means that a bad (high latency)
disk is going to be more tragic for I/O then in a larger pool, because in
the larger pool ZFS can distribute I/Os based on actual device readiness,
and its more likely that it will be able to find an idle disk if it has 12
to choose from instead of just 2. The read/write pattern matters as well,
btw.
Agreed. Another of what I'm sure if we spent time going over it is probably
a fairly extensive list of reasons to go for single instead of multiple
pools within a single storage box, with some limited exceptions (that,
again, just prove the general rule).
Post by Andrew Galloway- Garrett
- Andrew
Post by Attila NagyHi,
I've had two, completely identical machines, both with 12 SATA disks.
They act as NFS servers, for once written (WORM type) files.
Machine A had 6 zpools, each of them made of two disk mirrors (zpool
create fs1 mirror da0 da1, zpool create fs2 mirror da2 da3 ...), while
machine B had 1 zpool, made of two disk mirrors (zpool create fs mirror da0
da1 mirror da2 da3 mirror da4 da5 ...) and 6 file systems created on it to
match the directory layout.
Each machine got a similar (real, not simulated) load. Each of the file
systems were evenly (as far as the application can do this) loaded, there
were no significantly more or less busy disk pairs in the separate zpool
case on machine A.
Yet, machine A with 6 separate zpools struggled under the load, while
machine B with 1 zpool could easily satisfy the requests.
I'm talking here about 40% average disk usage on machine A (50-70 IOPS
avg on each devices) and well under 5(!) (10-15 IOPS per device) on machine
B.
The difference was in write, the read IOPS numbers were on par.
Also, every zfs operations (zpool list, status etc) took ages to run on
machine A, while ran fast on machine B.
The OS is FreeBSD stable/9, r255573.
Any ideas what causes this? Is this a ZFS issue at all, or a FreeBSD one?
ps: I'm off list, so please keep me CC-ed.
------------------------------**-------------
illumos-zfs
Archives: https://www.listbox.com/**member/archive/182191/=now<https://www.listbox.com/member/archive/182191/=now>
RSS Feed: https://www.listbox.com/**member/archive/rss/182191/**
24484421-62d25f20<https://www.listbox.com/member/archive/rss/182191/24484421-62d25f20>
Modify Your Subscription: https://www.listbox.com/**member/?&id_**
secret=24484421-04fe8ef2 <https://www.listbox.com/member/?&>
Powered by Listbox: http://www.listbox.com
*illumos-zfs* | Archives<https://www.listbox.com/member/archive/182191/=now>
<https://www.listbox.com/member/archive/rss/182191/22035932-85c5d227> |
Modify <https://www.listbox.com/member/?&> Your Subscription
<http://www.listbox.com/>
*illumos-zfs* | Archives<https://www.listbox.com/member/archive/182191/=now>
<https://www.listbox.com/member/archive/rss/182191/24484421-62d25f20> |
Modify<https://www.listbox.com/member/?&>Your Subscription
<http://www.listbox.com>
-------------------------------------------
illumos-zfs
Archives: https://www.listbox.com/member/archive/182191/=now
RSS Feed: https://www.listbox.com/member/archive/rss/182191/23047029-187a0c8d
Modify Your Subscription: https://www.listbox.com/member/?member_id=23047029&id_secret=23047029-2e85923f
Powered by Listbox: http://www.listbox.com
