sensible pool layout?

Discussion:

Eugen Leitl

2013-10-13 08:12:35 UTC

I've got a napp-it/OmniOS zfs appliance with the following
pool layout (rpool is 2x 80 GByte Intel SSD, tank0
is 8x Seagate 1 TByte consumer drives, 1x C300 SSD).
8 GByte RAM (non-ECC), dual-core low end AMD.

Scrub (tank0) is at about 200+ MByte/s. Usage
is read-mostly, plus being a daily backup target
in a SOHO setting.

Makes sense, what would you do differently?

***@oozfs:~# zpool status rpool
pool: rpool
state: ONLINE
scan: resilvered 6.14G in 0h2m with 0 errors on Sat Oct 12 21:28:38 2013
config:

NAME STATE READ WRITE CKSUM
rpool ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
c1t1d0s0 ONLINE 0 0 0
c1t2d0s0 ONLINE 0 0 0

errors: No known data errors
***@oozfs:~# zpool status tank0
pool: tank0
state: ONLINE
scan: scrub repaired 0 in 5h43m with 0 errors on Sun Oct 13 05:18:05 2013
config:

NAME STATE READ WRITE CKSUM
tank0 ONLINE 0 0 0
raidz3-0 ONLINE 0 0 0
c4t5000C500098BE9DDd0 ONLINE 0 0 0
c4t5000C50009C72C48d0 ONLINE 0 0 0
c4t5000C50009C73968d0 ONLINE 0 0 0
c4t5000C5000FD2E794d0 ONLINE 0 0 0
c4t5000C5000FD37075d0 ONLINE 0 0 0
c4t5000C5000FD39D53d0 ONLINE 0 0 0
c4t5000C5000FD3BC10d0 ONLINE 0 0 0
c4t5000C5000FD3E8A7d0 ONLINE 0 0 0
logs
c1t3d0 ONLINE 0 0 0

errors: No known data errors

Ian Collins

2013-10-13 08:39:47 UTC

Permalink

Post by Eugen Leitl
I've got a napp-it/OmniOS zfs appliance with the following
pool layout (rpool is 2x 80 GByte Intel SSD, tank0
is 8x Seagate 1 TByte consumer drives, 1x C300 SSD).
8 GByte RAM (non-ECC), dual-core low end AMD.
Scrub (tank0) is at about 200+ MByte/s. Usage
is read-mostly, plus being a daily backup target
in a SOHO setting.
Makes sense, what would you do differently?
pool: rpool
state: ONLINE
scan: resilvered 6.14G in 0h2m with 0 errors on Sat Oct 12 21:28:38 2013
NAME STATE READ WRITE CKSUM
rpool ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
c1t1d0s0 ONLINE 0 0 0
c1t2d0s0 ONLINE 0 0 0
errors: No known data errors
pool: tank0
state: ONLINE
scan: scrub repaired 0 in 5h43m with 0 errors on Sun Oct 13 05:18:05 2013
NAME STATE READ WRITE CKSUM
tank0 ONLINE 0 0 0
raidz3-0 ONLINE 0 0 0
c4t5000C500098BE9DDd0 ONLINE 0 0 0
c4t5000C50009C72C48d0 ONLINE 0 0 0
c4t5000C50009C73968d0 ONLINE 0 0 0
c4t5000C5000FD2E794d0 ONLINE 0 0 0
c4t5000C5000FD37075d0 ONLINE 0 0 0
c4t5000C5000FD39D53d0 ONLINE 0 0 0
c4t5000C5000FD3BC10d0 ONLINE 0 0 0
c4t5000C5000FD3E8A7d0 ONLINE 0 0 0
logs
c1t3d0 ONLINE 0 0 0
errors: No known data errors

With ITB drives, I would probably have used raidz2, but theres's no harm
in using raidz3.

--
Ian.

Eugen Leitl

2013-10-13 08:57:46 UTC

Permalink

Post by Ian Collins
With ITB drives, I would probably have used raidz2, but theres's no
harm in using raidz3.

Schlacta, Christ

2013-10-13 16:42:31 UTC

Permalink

I might suggest replacing your untrusted drives with trusted drives I've at
a time as budget permits.
I'd also suggest upgrading your ram to ecc. Especially since this is a
backup server.
Finally, since your root drives are so over sized, I'd recommend carving a
few GiB out of those for mirrored log, and use the single ssd as cache.

Post by Eugen Leitl

Post by Ian Collins
With ITB drives, I would probably have used raidz2, but theres's no
harm in using raidz3.

With nearline SATA or SAS I would have gone for raidz2.
These are survivor drives, mostly refurbished from a
quite notorius line of Seagate consumer drives.
I trust them about as far as I can throw them.
-------------------------------------------
illumos-zfs
Archives: https://www.listbox.com/member/archive/182191/=now
https://www.listbox.com/member/archive/rss/182191/23054485-60ad043a
https://www.listbox.com/member/?&
Powered by Listbox: http://www.listbox.com

-------------------------------------------
illumos-zfs
Archives: https://www.listbox.com/member/archive/182191/=now
RSS Feed: https://www.listbox.com/member/archive/rss/182191/23047029-187a0c8d
Modify Your Subscription: https://www.listbox.com/member/?member_id=23047029&id_secret=23047029-2e85923f
Powered by Listbox: http://www.listbox.com

Keith Wesolowski

2013-10-21 16:17:53 UTC

Permalink

Throw away the C300, or use it as L2ARC instead. This device will lose
your data in the event of a power failure. It also has a pathetic 72 TB
of total write endurance (sequential) rating, so unless by "read-mostly"
you really mean 99% read, it will probably fail in just a few months. I
suggest using DTrace or even iostat to collect some data here so that
you will be able to estimate the expected lifetime of this device in
your specific application. Regardless of endurance, though, it's unsafe
for use as a slog.

Eugen Leitl

2013-10-21 16:41:22 UTC

Permalink

Post by Keith Wesolowski

Throw away the C300, or use it as L2ARC instead. This device will lose

Thanks, I will do that.

Post by Keith Wesolowski
your data in the event of a power failure. It also has a pathetic 72 TB

My drop-in replacement would have been M500 (which does have buffer
capacitors IIRC), but I see it's also only 72 TBytes write endurance.
A P400M is out of budget, is there anything which would be more
reliable than a C300? Is there something like an Intel X25-E 32 GByte
(4 PBytes write endurance over 3 years) but without the price tag?
Or should I just give up on the slog entirely?

Post by Keith Wesolowski
of total write endurance (sequential) rating, so unless by "read-mostly"
you really mean 99% read, it will probably fail in just a few months. I

It is actually something like that, but I'd rather not screw up
that pool by a failing slog.

Post by Keith Wesolowski
suggest using DTrace or even iostat to collect some data here so that
you will be able to estimate the expected lifetime of this device in
your specific application. Regardless of endurance, though, it's unsafe
for use as a slog.

Keith Wesolowski

2013-10-21 17:46:48 UTC

Permalink

Post by Eugen Leitl
My drop-in replacement would have been M500 (which does have buffer
capacitors IIRC), but I see it's also only 72 TBytes write endurance.
A P400M is out of budget, is there anything which would be more
reliable than a C300? Is there something like an Intel X25-E 32 GByte
(4 PBytes write endurance over 3 years) but without the price tag?
Or should I just give up on the slog entirely?

The cheapest safe slog I'm aware of is the Intel 330. It's quite slow,
however. You may find better options in your own research, as mine was
based on slightly less budget-constrained parameters. As Richard
suggests, you may not benefit from a slog at all.

Post by Eugen Leitl

Post by Keith Wesolowski
of total write endurance (sequential) rating, so unless by "read-mostly"
you really mean 99% read, it will probably fail in just a few months. I

It is actually something like that, but I'd rather not screw up
that pool by a failing slog.

It's best to measure. Given how little you're writing, maybe this won't
be an issue. That's why we have tools!

Timothy Coalson

2013-10-21 17:59:29 UTC

Permalink

On Mon, Oct 21, 2013 at 12:46 PM, Keith Wesolowski <

Post by Keith Wesolowski

Did you mean 320? Intel's site doesn't appear to claim power loss
protection on the 330.

Tim

-------------------------------------------
illumos-zfs
Archives: https://www.listbox.com/member/archive/182191/=now
RSS Feed: https://www.listbox.com/member/archive/rss/182191/23047029-187a0c8d
Modify Your Subscription: https://www.listbox.com/member/?member_id=23047029&id_secret=23047029-2e85923f
Powered by Listbox: http://www.listbox.com

Keith Wesolowski

2013-10-21 18:08:26 UTC

Permalink

Post by Timothy Coalson
Did you mean 320? Intel's site doesn't appear to claim power loss
protection on the 330.

Sorry, yes. I've used the 320 for this, but have no information on the
330.

Garrett D'Amore

2013-10-21 20:04:49 UTC

Permalink

The Intel SSD 320 is no longer manufactured. Additionally, I will caution folks that I've recently seen a rather bizarre erratum which supposedly is fixed in recent firmware, but I've seen it with the latest firmware:

The drive reports is serial number as BAD_CTX, and a total capacity of 8 MB, when it encounters flaky power. Perhaps a reset on the bus for example.

I've only seen one occurrence of this, but it was on a drive in production, here's the details:

http://downloadmirror.intel.com/18363/eng/Intel_SSD_Firmware_Update_Tool_Release_Notes_2%200%206.pdf

HOWEVER:

The Intel DC S3700 looks like a nice option 10 drive writes per day for 5 years (about 1.8PB) , and power loss protection, and a pretty high write IOPs number. They're more expensive than your usual cheapie drives (about $500), but you're only looking at one for an SLOG, and its loads less expensive than other options like a ZeusRAM.

If I was going to buy a drive for an SLOG, I'd be looking hard at this option.

- Garrett

Post by Keith Wesolowski

Post by Timothy Coalson
Did you mean 320? Intel's site doesn't appear to claim power loss
protection on the 330.

Sorry, yes. I've used the 320 for this, but have no information on the
330.
-------------------------------------------
illumos-zfs
Archives: https://www.listbox.com/member/archive/182191/=now
RSS Feed: https://www.listbox.com/member/archive/rss/182191/22035932-85c5d227
Modify Your Subscription: https://www.listbox.com/member/?&
Powered by Listbox: http://www.listbox.com

jason matthews

2013-10-22 18:43:49 UTC

Permalink

Post by Garrett D'Amore
The Intel DC S3700 looks like a nice option 10 drive writes per day for 5 years (about 1.8PB) , and power loss protection, and a pretty high write IOPs number. They're more expensive than your usual cheapie drives (about $500), but you're only looking at one for an SLOG, and its loads less expensive than other options like a ZeusRAM.
If I was going to buy a drive for an SLOG, I'd be looking hard at this option.

i am testing with with eight of 800gb version of these drives now, courtesy of dell. i can get filebench (5 instances, 16 threads/instance on 4x e5-4650L) to push them 5% beyond their rated 20k 8k random writes/second (zfs turns random writes into sequential writes, but i digress). they are nice drives for sure. i am still working out how to eliminate the dramatic performance roll off when using these devices as primary storage beyond two stripes.

The Intel 910 with four stripes out runs four stripes of 800gb drives on LSI 9207-8i HBAs. The difference was 56k writes for the 910 and 48k for the four DC S3700s.

two thoughts on this most organizations don't need hundreds of gigabyte of ZIL, the ddrdrive x1 is orders of magnitude faster.

here is my preliminary data on drive performance. based on the compression results, filebench may not be a good tool as the data appears to be highly compressible.

# of stripes /mirrors compression trial1 trial 2 trial 3 average avg/span lost iops/ span MB/s CPU Utilization %
1 off 21,232 21,067 21,092 21,130 21,130 0 169 40
2 off 38,852 40,000 38,797 39,216 19,608 -3044 314 40
3 off 49,242 48,577 49,614 49,144 16,381 -14247 393 40
4 off 46,185 44,454 55,014 48,551 28,174 -35970 388 40

1 lz4 84,876 85,011 85,035 84,974 84974 0 679 100
2 lz4 84,564 84,453 84,019 84,345 42,173 -85603 674 100
3 lz4 83,996 84,462 84,730 84,396 84,396 -170,526 675 100
4 lz4 75,239 75,456 75,380 75,358 18,840 -264,538 589 100
1 lzjb 75,907 75,920 75,903 75,910 75,910 0 593 100
2 lzjb 80,387 80,576 80,383 80,449 40,224 -71371.33333 100
3 lzjb 81,212 81,493 81,328 81,344 27,115 -146385.6667 636 100
4 lzjb 85,080 84,848 85,485 85,138 21,284 -218,502 663 100

j.

-------------------------------------------
illumos-zfs
Archives: https://www.listbox.com/member/archive/182191/=now
RSS Feed: https://www.listbox.com/member/archive/rss/182191/23047029-187a0c8d
Modify Your Subscription: https://www.listbox.com/member/?member_id=23047029&id_secret=23047029-2e85923f
Powered by Listbox: http://www.listbox.com

Ian Collins

2013-10-23 04:12:56 UTC

Permalink

The Intel DC S3700 looks like a nice option… 10 drive writes per day
for 5 years (about 1.8PB) , and power loss protection, and a pretty
high write IOPs number. They're more expensive than your usual
cheapie drives (about $500), but you're only looking at one for an
SLOG, and its loads less expensive than other options like a ZeusRAM.
If I was going to buy a drive for an SLOG, I'd be looking hard at this option.

I have been using these for log devices for a while (the 100GB drive is
well under $300) and they do have very good write IOPs.

--
Ian.

Jim Klimov

2013-10-21 20:28:33 UTC

Permalink

Post by Keith Wesolowski

I was looking at Intel DC 3700 series (note - not DC3500 which seem
to have half the cost and much lower "promised" reliability), but
then I found Seagate SSD 600 Pro series and Seagate Enterprise SSD
which are even cheaper but promise all the nice features as well,
and beat the Intel's in many tests.

The two seem to be roughly equivalent in specs and tests, both with
capacitors and explicitly promised power-loss protection, and only
one lineup was available in our shops. I took a couple for a spin as
a rpool/zil/l2arc combo (on N54L), and would welcome ready-to-try
dtrace scripts and other tools to test these and report to the list :)

Also note that these series are available in different sizes, like
100Gb and 120Gb, which are reportedly the same hardware with different
reservations. Smaller ones have both much higher performance and
total-written-volume reliability. Alas, 100Gb ones are not sold here,
so I took 120Gb ones and formatted only 100Gb into partitions leaving
20Gb logically empty - hopefully, this would achieve a similar effect.

Somehow, of the 4*4Tb Deskstars two were DOA and one was very slow,
with only one of four performing similarly to expectations. Now we're
into replacing them with some other 4Tb model... perhaps WD Red's?

//Jim

Schlacta, Christ

2013-10-21 21:16:14 UTC

Permalink

I had a similar experience with 3tb wd reds. I returned then to the store
as doa and got replacements. Now I have 6 drives working perfectly.

Post by Jim Klimov

Post by Eugen Leitl
My drop-in replacement would have been M500 (which does have buffer

Post by Eugen Leitl
capacitors IIRC), but I see it's also only 72 TBytes write endurance.
A P400M is out of budget, is there anything which would be more
reliable than a C300? Is there something like an Intel X25-E 32 GByte
(4 PBytes write endurance over 3 years) but without the price tag?
Or should I just give up on the slog entirely?

I was looking at Intel DC 3700 series (note - not DC3500 which seem
to have half the cost and much lower "promised" reliability), but
then I found Seagate SSD 600 Pro series and Seagate Enterprise SSD
which are even cheaper but promise all the nice features as well,
and beat the Intel's in many tests.
The two seem to be roughly equivalent in specs and tests, both with
capacitors and explicitly promised power-loss protection, and only
one lineup was available in our shops. I took a couple for a spin as
a rpool/zil/l2arc combo (on N54L), and would welcome ready-to-try
dtrace scripts and other tools to test these and report to the list :)
Also note that these series are available in different sizes, like
100Gb and 120Gb, which are reportedly the same hardware with different
reservations. Smaller ones have both much higher performance and
total-written-volume reliability. Alas, 100Gb ones are not sold here,
so I took 120Gb ones and formatted only 100Gb into partitions leaving
20Gb logically empty - hopefully, this would achieve a similar effect.
Somehow, of the 4*4Tb Deskstars two were DOA and one was very slow,
with only one of four performing similarly to expectations. Now we're
into replacing them with some other 4Tb model... perhaps WD Red's?
//Jim
------------------------------**-------------
illumos-zfs
Archives: https://www.listbox.com/**member/archive/182191/=now<https://www.listbox.com/member/archive/182191/=now>
RSS Feed: https://www.listbox.com/**member/archive/rss/182191/**
23054485-60ad043a<https://www.listbox.com/member/archive/rss/182191/23054485-60ad043a>
Modify Your Subscription: https://www.listbox.com/**
member/?&id_**secret=23054485-335460f5<https://www.listbox.com/member/?&>
Powered by Listbox: http://www.listbox.com

Ian Collins

2013-10-22 00:26:23 UTC

Permalink

Post by Jim Klimov

Post by Keith Wesolowski

As a log device?

--
Ian.

Jason Matthews

2013-10-22 04:06:07 UTC

Permalink

Post by Ian Collins

Post by Jim Klimov
I was looking at Intel DC 3700 series (note - not DC3500 which seem
to have half the cost and much lower "promised" reliability), but
then I found Seagate SSD 600 Pro series and Seagate Enterprise SSD
which are even cheaper but promise all the nice features as well,
and beat the Intel's in many tests.

As a log device?

if you have this kind of budget why not use a memory based card like x1 or whatever replaced it.

i have 40 or so ddrdrive x1 drives and they work well and orders of magnitude faster than mlc flash.

j.

Ian Collins

2013-10-22 07:25:40 UTC

Permalink

Post by Jason Matthews

Post by Ian Collins

As a log device?

if you have this kind of budget why not use a memory based card like x1 or whatever replaced it.

If I had that kind of budget!

Post by Jason Matthews
i have 40 or so ddrdrive x1 drives and they work well and orders of magnitude faster than mlc flash.

I guess it isn't easy to compare form the datasheet, the 4K random write
figures they quote are on par with the 3700s.

--
Ian.

Richard Elling

2013-10-21 17:31:47 UTC

Permalink

Is your backup a sync write workload? If not, don't worry about the slog.
-- richard

--

***@RichardElling.com
+1-760-896-4422

-------------------------------------------
illumos-zfs
Archives: https://www.listbox.com/member/archive/182191/=now
RSS Feed: https://www.listbox.com/member/archive/rss/182191/23047029-187a0c8d
Modify Your Subscription: https://www.listbox.com/member/?member_id=23047029&id_secret=23047029-2e85923f
Powered by Listbox: http://www.listbox.com

jason matthews

2013-10-21 22:11:35 UTC

Permalink

this can work for you.

you can get by on raidz but raidz2 would be better if you have the drive slots to spare. i have an archive storage system built on forty-eight (going on to ninety-six) 2tb consumer drives plus six spares. It is totally doable - but i recommend ECC memory. One bad address in a DIMM could ruin your whole storage pool.

i assume you want to use the c300 as a L2ARC device. that will be fine. i use m4 as L2ARC with l2arc_noprefetch=0 which puts more stress on the SSDs. In a pool that writes 1.4TB/day, with two years of use, the percentage of wear reported via SMART is about 4% (three such M4s exist in the pool) - so that is 12% normalized. I like to operate on a three year refresh cycle so 12% is under utilized.

My experience with the M4/C300 is that well exceed their rated 3000 write cycle rating. I have disks that were used as primary storage for the monitoring system that now report up to 7000 write cycles for the whole disk. that said, garbage collection on the M4/C300 is horrible so it should have not a primary role in your environment - i am phasing them out for Intel DC S3700s. L2ARC seems has a good fit as far as duty cycle goes so the M4/C300 seems to keep up.

my i/o pattern will be different than yours, so your mileage will vary. you should not use the c300 as ZIL - it is not a suitable device.

j.

-------------------------------------------
illumos-zfs
Archives: https://www.listbox.com/member/archive/182191/=now
RSS Feed: https://www.listbox.com/member/archive/rss/182191/23047029-187a0c8d
Modify Your Subscription: https://www.listbox.com/member/?member_id=23047029&id_secret=23047029-2e85923f
Powered by Listbox: http://www.listbox.com