Boot from degraded rpool

[Jim Klimov]

Kind of following up on that old system of mine (old OI release
with custom illumos-gate from early 2012, in particular), I found
that when I mirrored from old disk to new one and then rebooted,
and physically removed one of the drives, the system failed to
mount_vfs_root (apparently related to the NOTICE about degraded
pool).

Booting from a live media, importing the rpool and detaching the
now-missing drive fixed the issue.

However (if this situation is still in place): isn't it kind of
counter-productive to allow mirroring to increase availability
of the system and then forbid it to boot from broken mirrors -
so that in case of a dead disk, the system would be unavailable
for as long as it takes somebody to get to it physically?

Or is there some command-line toggle (or a /etc/system switch)
for the kernel to allow such boot-ups?

//Jim

[Jim Bolmgren-jim99]

Hi Jim,

I'm just getting started with ZFS so mostly my comments are from reading. From my personal experience of only a couple weeks I can't tell you what is going wrong but in reading everything I have read suggests the memory has to be ECC so that night be an issue. One thing I know from experience when mirroring hard drives is if the master has problems with broken files it will mirror the same issues over to the slave. I experienced boot issues with an old Windows server like that once. Also just because an old drive spins up doesn't mean the platter is good so you might be getting errors because the surface of the platter is dying. Another potential problem I had once was the hard drive controller was slowly failing so it was writing bad files intermittently onto a good drive. Sorry for all the doom and gloom and I hope eventually you find the answer. Maybe as a test you should rebuild the OS on the newer drive and don't do any mirroring to see if any issues go away.

Jim

----- Original Message -----
From: "Jim Klimov" <***@cos.ru>
To: ***@lists.illumos.org
Sent: Saturday, December 21, 2013 9:38:29 AM
Subject: [zfs] Boot from degraded rpool

Kind of following up on that old system of mine (old OI release
with custom illumos-gate from early 2012, in particular), I found
that when I mirrored from old disk to new one and then rebooted,
and physically removed one of the drives, the system failed to
mount_vfs_root (apparently related to the NOTICE about degraded
pool).

Booting from a live media, importing the rpool and detaching the
now-missing drive fixed the issue.

However (if this situation is still in place): isn't it kind of
counter-productive to allow mirroring to increase availability
of the system and then forbid it to boot from broken mirrors -
so that in case of a dead disk, the system would be unavailable
for as long as it takes somebody to get to it physically?

Or is there some command-line toggle (or a /etc/system switch)
for the kernel to allow such boot-ups?

//Jim


-------------------------------------------
illumos-zfs
Archives: https://www.listbox.com/member/archive/182191/=now
RSS Feed: https://www.listbox.com/member/archive/rss/182191/25363661-336ccbd6
Modify Your Subscription: https://www.listbox.com/member/?&
Powered by Listbox: http://www.listbox.com

[Tim Cook]

Fairly certain this is just an issue with your failmode. If you set the
failmode to continue I would expect it to boot as normal (assuming you've
got up to date grub on both drives).

*failmode*=*wait* | *continue* | *panic*
Controls the system behavior in the event of catastrophic pool fail-
ure. This condition is typically a result of a loss of connectivity
to the underlying storage device(s) or a failure of all devices
within the pool. The behavior of such an event is determined as fol-
lows:

*wait* Blocks all I/O access until the device connectivity is recov-
ered and the errors are cleared. This is the default behav-
ior.

*continue*
Returns *EIO* to any new write I/O requests but allows reads to
any of the remaining healthy devices. Any write requests that
have yet to be committed to disk would be blocked.

*panic* Prints out a message to the console and generates a system
crash dump.



--Tim

Post by Jim Bolmgren-jim99
Hi Jim,
I'm just getting started with ZFS so mostly my comments are from reading.
From my personal experience of only a couple weeks I can't tell you what
is going wrong but in reading everything I have read suggests the memory
has to be ECC so that night be an issue. One thing I know from experience
when mirroring hard drives is if the master has problems with broken files
it will mirror the same issues over to the slave. I experienced boot issues
with an old Windows server like that once. Also just because an old drive
spins up doesn't mean the platter is good so you might be getting errors
because the surface of the platter is dying. Another potential problem I
had once was the hard drive controller was slowly failing so it was writing
bad files intermittently onto a good drive. Sorry for all the doom and
gloom and I hope eventually you find the answer. Maybe as a test you should
rebuild the OS on the newer drive and don't do any mirroring to see if any
issues go away.
Jim
----- Original Message -----
Sent: Saturday, December 21, 2013 9:38:29 AM
Subject: [zfs] Boot from degraded rpool
Kind of following up on that old system of mine (old OI release
with custom illumos-gate from early 2012, in particular), I found
that when I mirrored from old disk to new one and then rebooted,
and physically removed one of the drives, the system failed to
mount_vfs_root (apparently related to the NOTICE about degraded
pool).
Booting from a live media, importing the rpool and detaching the
now-missing drive fixed the issue.
However (if this situation is still in place): isn't it kind of
counter-productive to allow mirroring to increase availability
of the system and then forbid it to boot from broken mirrors -
so that in case of a dead disk, the system would be unavailable
for as long as it takes somebody to get to it physically?
Or is there some command-line toggle (or a /etc/system switch)
for the kernel to allow such boot-ups?
//Jim
-------------------------------------------
illumos-zfs
Archives: https://www.listbox.com/member/archive/182191/=now
https://www.listbox.com/member/archive/rss/182191/25363661-336ccbd6
Modify Your Subscription: https://www.listbox.com/member/?&
Powered by Listbox: http://www.listbox.com
-------------------------------------------
illumos-zfs
Archives: https://www.listbox.com/member/archive/182191/=now
https://www.listbox.com/member/archive/rss/182191/21996034-de72a2b0
https://www.listbox.com/member/?&
Powered by Listbox: http://www.listbox.com

-------------------------------------------
illumos-zfs
Archives: https://www.listbox.com/member/archive/182191/=now
RSS Feed: https://www.listbox.com/member/archive/rss/182191/23047029-187a0c8d
Modify Your Subscription: https://www.listbox.com/member/?member_id=23047029&id_secret=23047029-2e85923f
Powered by Listbox: http://www.listbox.com

[Saso Kiselkov]

Post by Tim Cook
Fairly certain this is just an issue with your failmode. If you set the
failmode to continue I would expect it to boot as normal (assuming
you've got up to date grub on both drives).
*failmode*=*wait* | *continue* | *panic*
Controls the system behavior in the event of catastrophic pool fail-
ure. This condition is typically a result of a loss of connectivity
to the underlying storage device(s) or a failure of all devices
within the pool. The behavior of such an event is determined as fol-
*wait* Blocks all I/O access until the device connectivity is recov-
ered and the errors are cleared. This is the default behav-
ior.
*continue*
Returns /EIO/ to any new write I/O requests but allows reads to
any of the remaining healthy devices. Any write requests that
have yet to be committed to disk would be blocked.
*panic* Prints out a message to the console and generates a system
crash dump.

Failmode pertains to a pool which has suffered data loss (e.g. loss of a
top-level vdev), not a recoverable state (e.g. loss of one side of a
mirror).

[Tim Cook]

Post by Tim Cook

Post by Tim Cook
Fairly certain this is just an issue with your failmode. If you set the
failmode to continue I would expect it to boot as normal (assuming
you've got up to date grub on both drives).
*failmode*=*wait* | *continue* | *panic*
Controls the system behavior in the event of catastrophic pool

fail-

Post by Tim Cook
ure. This condition is typically a result of a loss of

connectivity

Post by Tim Cook
to the underlying storage device(s) or a failure of all devices
within the pool. The behavior of such an event is determined as

fol-

Post by Tim Cook
*wait* Blocks all I/O access until the device connectivity is

recov-

Post by Tim Cook
ered and the errors are cleared. This is the default

behav-

Post by Tim Cook
ior.
*continue*
Returns /EIO/ to any new write I/O requests but allows

reads to

Post by Tim Cook
any of the remaining healthy devices. Any write requests

that

Post by Tim Cook
have yet to be committed to disk would be blocked.
*panic* Prints out a message to the console and generates

a system

Post by Tim Cook
crash dump.

Failmode pertains to a pool which has suffered data loss (e.g. loss of a
top-level vdev), not a recoverable state (e.g. loss of one side of a
mirror).
--
Saso

That's not how the property ACTUALLY worked in older versions of OI, I have
no idea if it's since been fixed (*Looks like supposedly has). I literally
had the exact same symptoms he had with ANY drive loss. My pool was
raid-z2 and the box hung indefinitely on drive failure.

*CR #6322646 is the one I believe addressed this last year. It absolutely
wasn't fixed with the version he's on.


--Tim



-------------------------------------------
illumos-zfs
Archives: https://www.listbox.com/member/archive/182191/=now
RSS Feed: https://www.listbox.com/member/archive/rss/182191/23047029-187a0c8d
Modify Your Subscription: https://www.listbox.com/member/?member_id=23047029&id_secret=23047029-2e85923f
Powered by Listbox: http://www.listbox.com

[Saso Kiselkov]

Post by Tim Cook

Post by Tim Cook
Fairly certain this is just an issue with your failmode. If you

set the

Post by Tim Cook
failmode to continue I would expect it to boot as normal (assuming
you've got up to date grub on both drives).
*failmode*=*wait* | *continue* | *panic*
Controls the system behavior in the event of catastrophic

pool fail-

Post by Tim Cook
ure. This condition is typically a result of a loss of

connectivity

Post by Tim Cook
to the underlying storage device(s) or a failure of all devices
within the pool. The behavior of such an event is

determined as fol-

Post by Tim Cook
*wait* Blocks all I/O access until the device connectivity

is recov-

Post by Tim Cook
ered and the errors are cleared. This is the

default behav-

Post by Tim Cook
ior.
*continue*
Returns /EIO/ to any new write I/O requests but

allows reads to

Post by Tim Cook
any of the remaining healthy devices. Any write

requests that

Post by Tim Cook
have yet to be committed to disk would be blocked.
*panic* Prints out a message to the console and

generates a system

Post by Tim Cook
crash dump.

Failmode pertains to a pool which has suffered data loss (e.g. loss of a
top-level vdev), not a recoverable state (e.g. loss of one side of a
mirror).
--
Saso
That's not how the property ACTUALLY worked in older versions of OI, I
have no idea if it's since been fixed (*Looks like supposedly has). I
literally had the exact same symptoms he had with ANY drive loss. My
pool was raid-z2 and the box hung indefinitely on drive failure.

Looking at the code history for anything having to do with failmode, the
last changes of substance to it seem to have taken place somewhere
around 2007-2008, years before there even was an OI.

Post by Tim Cook
*CR #6322646 is the one I believe addressed this last year. It
absolutely wasn't fixed with the version he's on.

This commit is from 2007 and its the one that introduced failmode.

[Jim Klimov]

Hello all, thanks for comments.

Post by Jim Bolmgren-jim99
One thing I know from experience when mirroring hard drives is if
the master has problems with broken files it will mirror the same
issues over to the slave. I experienced boot issues with an old
Windows server like that once.

* To Jim: ZFS mirroring is better than others (including Linux md
and motherboard-builtin software RAID) in exactly this regard: there
is no single trusted "master disk". There are checksums of every
block, so that errors are known and not replicated. And can be fixed
if another copy is available and thanks to checksums - known to be good.

ECC is not a "requirement" but an essential recommendation. For example
on this box without ECC support, I can not be certain if any problems
with RAM exist at all and/or are the reason for errors I see.



* To Saso: you outlined the same procedure I followed on that box.
I did not re-test this yet with a newer release, so am ready to write
this off as a glitch in older software which may have been fixed since.
At least it is nice to know that it probably does not need fixing
anymore :)

* The failmode, just in case, is "continue":

# zpool get failmode rpool
NAME PROPERTY VALUE SOURCE
rpool failmode continue local


* fmdump took quite a while to get all the reports from back in
2011 onto my terminal, but most of that is checksum/data errors.
The few "scsi" errors listed below started after I issued smartctl
commands to check the disks and are not seen in the earlier days.

Dec 20 12:31:52.1691 ereport.fs.zfs.data
0xc4d4e4587c700001
Dec 20 12:31:52.1691 ereport.fs.zfs.checksum
0xc4d4e4587c700001
Dec 20 12:31:52.1691 ereport.fs.zfs.checksum
0xc4d4e4587c700001
Dec 20 12:31:52.1691 ereport.fs.zfs.checksum
0xc4d4e4587c700001
Dec 20 12:31:52.1691 ereport.fs.zfs.checksum
0xc4d4e4587c700001
Dec 20 12:31:52.1691 ereport.fs.zfs.checksum
0xc4d4e4587c700001
Dec 20 12:31:52.1691 ereport.fs.zfs.checksum
0xc4d4e4587c700001
Dec 20 12:31:52.1691 ereport.fs.zfs.checksum
0xc4d4e4587c700001
Dec 20 12:31:52.1691 ereport.fs.zfs.checksum
0xc4d4e4587c700001
Dec 20 12:31:52.1691 ereport.fs.zfs.checksum
0xc4d4e4587c700001
Dec 20 12:31:52.1691 ereport.fs.zfs.checksum
0xc4d4e4587c700001
Dec 20 12:31:52.1691 ereport.fs.zfs.checksum
0xc4d4e4587c700001
Dec 20 12:31:52.1691 ereport.fs.zfs.checksum
0xc4d4e4587c700001
Dec 20 12:32:00.0861 ereport.fs.zfs.data
0xc4f26304b1d00401
Dec 20 12:32:00.0861 ereport.fs.zfs.checksum
0xc4f26304b1d00401
Dec 20 12:32:00.0861 ereport.fs.zfs.checksum
0xc4f26304b1d00401
Dec 20 12:32:00.0861 ereport.fs.zfs.checksum
0xc4f26304b1d00401
Dec 20 12:32:00.0861 ereport.fs.zfs.checksum
0xc4f26304b1d00401
Dec 20 12:32:00.0861 ereport.fs.zfs.checksum
0xc4f26304b1d00401
Dec 20 12:32:00.0861 ereport.fs.zfs.checksum
0xc4f26304b1d00401
Dec 20 12:32:00.0861 ereport.fs.zfs.checksum
0xc4f26304b1d00401
Dec 20 12:32:00.0861 ereport.fs.zfs.checksum
0xc4f26304b1d00401
Dec 20 12:32:00.0861 ereport.fs.zfs.checksum
0xc4f26304b1d00401
Dec 20 12:32:00.0861 ereport.fs.zfs.checksum
0xc4f26304b1d00401
Dec 20 12:32:00.0861 ereport.fs.zfs.checksum
0xc4f26304b1d00401
Dec 20 12:32:00.0861 ereport.fs.zfs.checksum
0xc4f26304b1d00401
Dec 20 12:38:44.8779 ereport.fs.zfs.data
0xcad6708e44300001
Dec 20 12:38:44.8779 ereport.fs.zfs.checksum
0xcad6708e44300001
Dec 20 12:38:44.8779 ereport.fs.zfs.checksum
0xcad6708e44300001
Dec 20 12:38:44.8779 ereport.fs.zfs.checksum
0xcad6708e44300001
Dec 20 12:38:44.8779 ereport.fs.zfs.checksum
0xcad6708e44300001
Dec 20 12:38:44.8779 ereport.fs.zfs.checksum
0xcad6708e44300001
Dec 20 12:38:44.8779 ereport.fs.zfs.checksum
0xcad6708e44300001
Dec 20 12:38:44.8781 ereport.fs.zfs.checksum
0xcad6708e44300001
Dec 20 12:38:44.8779 ereport.fs.zfs.checksum
0xcad6708e44300001
Dec 20 12:38:44.8779 ereport.fs.zfs.checksum
0xcad6708e44300001
Dec 20 12:38:44.8779 ereport.fs.zfs.checksum
0xcad6708e44300001
Dec 20 12:38:44.8779 ereport.fs.zfs.checksum
0xcad6708e44300001
Dec 20 12:38:44.8779 ereport.fs.zfs.checksum
0xcad6708e44300001
Dec 20 15:02:07.5464 ereport.fs.zfs.data
0x4807967781b00401
Dec 20 15:02:07.5464 ereport.fs.zfs.checksum
0x4807967781b00401
Dec 20 15:02:07.5464 ereport.fs.zfs.checksum
0x4807967781b00401
Dec 20 15:02:07.5464 ereport.fs.zfs.checksum
0x4807967781b00401
Dec 20 15:02:07.5464 ereport.fs.zfs.checksum
0x4807967781b00401
Dec 20 15:02:07.5464 ereport.fs.zfs.checksum
0x4807967781b00401
Dec 20 15:02:07.5464 ereport.fs.zfs.checksum
0x4807967781b00401
Dec 20 15:02:07.5464 ereport.fs.zfs.checksum
0x4807967781b00401
Dec 20 15:02:07.5464 ereport.fs.zfs.checksum
0x4807967781b00401
Dec 20 15:02:07.5464 ereport.fs.zfs.checksum
0x4807967781b00401
Dec 20 15:02:07.5464 ereport.fs.zfs.checksum
0x4807967781b00401
Dec 20 15:02:07.5464 ereport.fs.zfs.checksum
0x4807967781b00401
Dec 20 15:02:07.5464 ereport.fs.zfs.checksum
0x4807967781b00401
Dec 20 15:25:26.6338 ereport.fs.zfs.data
0x5c63e44016700401
Dec 20 15:25:26.6338 ereport.fs.zfs.checksum
0x5c63e44016700401
Dec 20 15:25:26.6338 ereport.fs.zfs.checksum
0x5c63e44016700401
Dec 20 15:25:26.6338 ereport.fs.zfs.checksum
0x5c63e44016700401
Dec 20 15:25:26.6338 ereport.fs.zfs.checksum
0x5c63e44016700401
Dec 20 15:25:26.6338 ereport.fs.zfs.checksum
0x5c63e44016700401
Dec 20 15:25:26.6338 ereport.fs.zfs.checksum
0x5c63e44016700401
Dec 20 15:25:26.6338 ereport.fs.zfs.checksum
0x5c63e44016700401
Dec 20 15:25:26.6338 ereport.fs.zfs.checksum
0x5c63e44016700401
Dec 20 15:25:26.6338 ereport.fs.zfs.checksum
0x5c63e44016700401
Dec 20 15:25:26.6338 ereport.fs.zfs.checksum
0x5c63e44016700401
Dec 20 15:25:26.6338 ereport.fs.zfs.checksum
0x5c63e44016700401
Dec 20 15:25:26.6338 ereport.fs.zfs.checksum
0x5c63e44016700401
Dec 20 15:45:25.0675 ereport.fs.zfs.data
0x6dd4a8c6a6100401
Dec 20 15:45:25.0675 ereport.fs.zfs.checksum
0x6dd4a8c6a6100401
Dec 20 15:45:25.0675 ereport.fs.zfs.checksum
0x6dd4a8c6a6100401
Dec 20 15:45:25.0675 ereport.fs.zfs.checksum
0x6dd4a8c6a6100401
Dec 20 15:45:25.0675 ereport.fs.zfs.checksum
0x6dd4a8c6a6100401
Dec 20 15:45:25.0675 ereport.fs.zfs.checksum
0x6dd4a8c6a6100401
Dec 20 15:45:25.0675 ereport.fs.zfs.checksum
0x6dd4a8c6a6100401
Dec 20 15:45:25.0675 ereport.fs.zfs.checksum
0x6dd4a8c6a6100401
Dec 20 15:45:25.0675 ereport.fs.zfs.checksum
0x6dd4a8c6a6100401
Dec 20 15:45:25.0675 ereport.fs.zfs.checksum
0x6dd4a8c6a6100401
Dec 20 15:45:25.0675 ereport.fs.zfs.checksum
0x6dd4a8c6a6100401
Dec 20 15:45:25.0675 ereport.fs.zfs.checksum
0x6dd4a8c6a6100401
Dec 20 15:45:25.0675 ereport.fs.zfs.checksum
0x6dd4a8c6a6100401
Dec 21 00:06:33.1621 ereport.fs.zfs.checksum
0x23668f4121100401
Dec 21 00:29:25.4337 ereport.fs.zfs.checksum
0x375ef60acb400001
Dec 21 11:30:27.9301 ereport.fs.zfs.data
0x7890c7825ea00001
Dec 21 11:30:27.9301 ereport.fs.zfs.checksum
0x7890c7825ea00001
Dec 21 11:30:27.9301 ereport.fs.zfs.checksum
0x7890c7825ea00001
Dec 21 11:30:27.9301 ereport.fs.zfs.checksum
0x7890c7825ea00001
Dec 21 11:30:27.9301 ereport.fs.zfs.checksum
0x7890c7825ea00001
Dec 21 11:30:27.9301 ereport.fs.zfs.checksum
0x7890c7825ea00001
Dec 21 11:30:27.9301 ereport.fs.zfs.checksum
0x7890c7825ea00001
Dec 21 11:30:27.9301 ereport.fs.zfs.checksum
0x7890c7825ea00001
Dec 21 11:30:27.9301 ereport.fs.zfs.checksum
0x7890c7825ea00001
Dec 21 11:30:27.9301 ereport.fs.zfs.checksum
0x7890c7825ea00001
Dec 21 11:30:27.9301 ereport.fs.zfs.checksum
0x7890c7825ea00001
Dec 21 11:30:27.9301 ereport.fs.zfs.checksum
0x7890c7825ea00001
Dec 21 11:30:27.9301 ereport.fs.zfs.checksum
0x7890c7825ea00001
Dec 21 12:19:54.9335 ereport.fs.zfs.checksum
0xa3be5b89cf600401
Dec 21 12:21:05.7922 ereport.fs.zfs.checksum
0xa4c6578815c00001
Dec 21 12:22:25.0750 ereport.fs.zfs.checksum
0xa5edb5ccbd400401
Dec 21 12:22:54.1633 ereport.fs.zfs.checksum
0xa65a142aaec00001
Dec 21 13:43:10.4746 ereport.io.scsi.cmd.disk.dev.rqs.derr
0x10438d031a600001
Dec 21 13:43:16.1539 ereport.io.scsi.cmd.disk.dev.rqs.derr
0x1058b539fe100001
Dec 21 13:44:22.0787 ereport.io.scsi.cmd.disk.dev.rqs.derr
0x114e4bf547400001
Dec 21 13:44:26.9645 ereport.io.scsi.cmd.disk.dev.rqs.derr
0x11607f7778300401
Dec 21 13:44:29.2503 ereport.io.scsi.cmd.disk.dev.rqs.derr
0x1169036a8de00401
Dec 21 13:44:31.0954 ereport.io.scsi.cmd.disk.dev.rqs.derr
0x116fe2f35d100401
Dec 21 13:44:33.1227 ereport.io.scsi.cmd.disk.dev.rqs.derr
0x11777061a2c00401
Dec 21 13:54:29.4339 ereport.io.scsi.cmd.disk.dev.rqs.derr
0x1a25947164300401
Dec 21 13:54:33.6320 ereport.io.scsi.cmd.disk.dev.rqs.derr
0x1a353810fbe00001
Dec 21 13:54:35.5247 ereport.io.scsi.cmd.disk.dev.rqs.derr
0x1a3c450c52400001
Dec 21 13:54:37.3388 ereport.io.scsi.cmd.disk.dev.rqs.derr
0x1a43071f9f900401
Dec 21 13:54:39.4122 ereport.io.scsi.cmd.disk.dev.rqs.derr
0x1a4ac073bbd00001
Dec 21 14:32:21.3726 ereport.io.scsi.cmd.disk.dev.rqs.derr
0x3b358e413d000001
Dec 21 14:32:26.9309 ereport.io.scsi.cmd.disk.dev.rqs.derr
0x3b4a4330d3300401
Dec 21 14:32:28.6441 ereport.io.scsi.cmd.disk.dev.rqs.derr
0x3b50a5187d200401
Dec 21 14:32:31.0320 ereport.io.scsi.cmd.disk.dev.rqs.derr
0x3b598a653d500401
Dec 21 14:32:32.8426 ereport.io.scsi.cmd.disk.dev.rqs.derr
0x3b60493cbcc00401
Dec 21 14:33:32.8827 ereport.fs.zfs.checksum
0x3c3ff5586b600001
Dec 21 14:35:46.3749 ereport.fs.zfs.checksum
0x3e31442abf300001
Dec 21 14:35:46.3749 ereport.fs.zfs.checksum
0x3e31442abf300001
Dec 21 14:35:46.4594 ereport.fs.zfs.checksum
0x3e3194c997a00401
Dec 21 14:35:46.4594 ereport.fs.zfs.checksum
0x3e3194c997a00401
Dec 21 14:35:50.6820 ereport.fs.zfs.checksum
0x3e414fe468000401
Dec 21 14:35:51.1294 ereport.fs.zfs.checksum
0x3e42fa84d6500001
Dec 21 14:35:51.1294 ereport.fs.zfs.checksum
0x3e42fa84d6500001
Dec 21 14:35:54.3318 ereport.fs.zfs.checksum
0x3e4ee8a786b00001
Dec 21 14:35:54.3318 ereport.fs.zfs.checksum
0x3e4ee8a786b00001
Dec 21 14:42:11.9818 ereport.fs.zfs.checksum
0x43cdd06b51e00401
Dec 21 14:42:11.9818 ereport.fs.zfs.checksum
0x43cdd06b51e00401
Dec 21 14:42:12.9112 ereport.fs.zfs.checksum
0x43d146c7b3800401
Dec 21 14:42:12.9112 ereport.fs.zfs.checksum
0x43d146c7b3800401
Dec 21 14:49:18.4714 ereport.io.scsi.cmd.disk.dev.rqs.derr
0x4a02ad6a0f500401
Dec 21 14:49:22.7455 ereport.io.scsi.cmd.disk.dev.rqs.derr
0x4a1299a8da200401
Dec 21 14:49:23.4798 ereport.io.scsi.cmd.disk.dev.rqs.derr
0x4a1555f048900401
Dec 21 14:49:25.2605 ereport.io.scsi.cmd.disk.dev.rqs.derr
0x4a1bf84174c00401
Dec 21 14:49:26.8081 ereport.io.scsi.cmd.disk.dev.rqs.derr
0x4a21bc41f8e00401
Dec 21 15:06:37.8081 ereport.fs.zfs.checksum
0x5922a89687100001
Dec 21 15:06:37.8081 ereport.fs.zfs.checksum
0x5922a89687100001
Dec 21 16:09:16.0315 ereport.io.scsi.cmd.disk.dev.rqs.derr
0x8fd3964b04c00001
Dec 21 16:09:19.6828 ereport.io.scsi.cmd.disk.dev.rqs.derr
0x8fe1309461b00001
Dec 21 16:09:20.8254 ereport.io.scsi.cmd.disk.dev.rqs.derr
0x8fe5724086c00401
Dec 21 16:09:21.9860 ereport.io.scsi.cmd.disk.dev.rqs.derr
0x8fe9c5325a200401
Dec 21 16:09:23.1430 ereport.io.scsi.cmd.disk.dev.rqs.derr
0x8fee14a0d3c00001
Dec 21 16:09:41.5650 ereport.fs.zfs.checksum
0x9032b5ead3100401
Dec 21 16:09:41.5650 ereport.fs.zfs.checksum
0x9032b5ead3100401
Dec 21 16:11:33.0525 ereport.fs.zfs.checksum
0x91d20c40d8100001
Dec 21 16:11:33.0525 ereport.fs.zfs.checksum
0x91d20c40d8100001
Dec 21 16:11:33.1082 ereport.fs.zfs.checksum
0x91d24153a8200001
Dec 21 16:11:33.1082 ereport.fs.zfs.checksum
0x91d24153a8200001
Dec 21 16:11:36.6849 ereport.fs.zfs.checksum
0x91df9469f2900001
Dec 21 16:11:36.6849 ereport.fs.zfs.checksum
0x91df9469f2900001
Dec 21 16:11:37.1319 ereport.fs.zfs.checksum
0x91e13ebe1cf00401
Dec 21 16:11:37.1319 ereport.fs.zfs.checksum
0x91e13ebe1cf00401
Dec 21 16:11:41.7997 ereport.fs.zfs.checksum
0x91f2a28088800001
Dec 21 16:11:41.7997 ereport.fs.zfs.checksum
0x91f2a28088800001
Dec 21 16:20:54.8897 ereport.fs.zfs.checksum
0x99ff209881d00401
Dec 21 16:20:54.8897 ereport.fs.zfs.checksum
0x99ff209881d00401
Dec 21 16:20:55.8360 ereport.fs.zfs.checksum
0x9a02a72227d00001
Dec 21 16:20:55.8360 ereport.fs.zfs.checksum
0x9a02a72227d00001
Dec 21 16:40:10.5784 ereport.fs.zfs.checksum
0xaad08db88ff00401
Dec 21 16:40:10.5784 ereport.fs.zfs.checksum
0xaad08db88ff00401
Dec 21 17:39:51.2304 ereport.fs.zfs.checksum
0xdeec03d766600001
Dec 21 17:39:51.2304 ereport.fs.zfs.checksum
0xdeec03d766600001
Dec 21 17:39:59.9474 ereport.io.scsi.cmd.disk.dev.rqs.derr
0xdf0c7d41fac00401
Dec 21 17:40:04.5210 ereport.io.scsi.cmd.disk.dev.rqs.derr
0xdf1d871db4600001
Dec 21 17:40:05.6893 ereport.io.scsi.cmd.disk.dev.rqs.derr
0xdf21e14ba3a00001
Dec 21 17:40:06.8237 ereport.io.scsi.cmd.disk.dev.rqs.derr
0xdf261b31d9100001
Dec 21 17:40:07.9910 ereport.io.scsi.cmd.disk.dev.rqs.derr
0xdf2a747e62f00001
Dec 21 17:43:37.1941 ereport.fs.zfs.checksum
0xe235d2cf5dc00001
Dec 21 17:43:37.1941 ereport.fs.zfs.checksum
0xe235d2cf5dc00001
Dec 21 17:43:37.2973 ereport.fs.zfs.checksum
0xe2363535aa500401
Dec 21 17:43:37.2973 ereport.fs.zfs.checksum
0xe2363535aa500401
Dec 21 17:43:40.8574 ereport.fs.zfs.checksum
0xe243788409000401
Dec 21 17:43:40.8574 ereport.fs.zfs.checksum
0xe243788409000401
Dec 21 17:43:41.3875 ereport.fs.zfs.checksum
0xe245721750500401
Dec 21 17:43:41.3875 ereport.fs.zfs.checksum
0xe245721750500401
Dec 21 17:43:45.3929 ereport.fs.zfs.checksum
0xe2545e0f14e00401
Dec 21 17:43:45.3929 ereport.fs.zfs.checksum
0xe2545e0f14e00401
Dec 21 17:53:32.0557 ereport.fs.zfs.checksum
0xeaddee9f10c00401
Dec 21 17:53:32.0557 ereport.fs.zfs.checksum
0xeaddee9f10c00401
Dec 21 17:53:32.8619 ereport.fs.zfs.checksum
0xeae0ef883f600001
Dec 21 17:53:32.8619 ereport.fs.zfs.checksum
0xeae0ef883f600001
Dec 21 18:13:13.7183 ereport.fs.zfs.checksum
0xfc10204d46400401
Dec 21 18:13:13.7183 ereport.fs.zfs.checksum
0xfc10204d46400401
Dec 21 19:01:42.8304 ereport.io.scsi.cmd.disk.dev.rqs.derr
0x2665cd0890800401
Dec 21 19:01:47.0346 ereport.io.scsi.cmd.disk.dev.rqs.derr
0x2675769b61100401
Dec 21 19:01:51.8947 ereport.io.scsi.cmd.disk.dev.rqs.derr
0x268791a84a900001
Dec 21 19:01:54.7808 ereport.io.scsi.cmd.disk.dev.rqs.derr
0x2692523029400401
Dec 21 19:01:56.7315 ereport.io.scsi.cmd.disk.dev.rqs.derr
0x2699968233200001
Dec 21 19:40:54.1171 ereport.fs.zfs.checksum
0x07edff41b8d00401
Dec 21 19:45:45.8638 ereport.fs.zfs.checksum
0x0c2cd69bd5300001
Dec 21 19:45:45.9143 ereport.fs.zfs.checksum
0x0c2d06c29e100401
Dec 21 19:45:50.2857 ereport.fs.zfs.checksum
0x0c3d4faaf2900001
Dec 21 19:45:50.6402 ereport.fs.zfs.checksum
0x0c3ea1c8bb400001
Dec 21 19:52:20.5475 ereport.fs.zfs.checksum
0x11eb264cbfa00401
Dec 21 19:52:21.4633 ereport.fs.zfs.checksum
0x11ee8fb5ce300401
Dec 21 20:16:23.0050 ereport.fs.zfs.checksum
0x26e90e5c32f00001
Dec 22 00:00:08.7555 ereport.fs.zfs.checksum
0xea49a29745000401
Dec 22 00:04:17.6368 ereport.fs.zfs.checksum
0xede8d33a61100001
Dec 22 00:34:30.6962 ereport.fs.zfs.checksum
0x084b2b20b9000001
Dec 22 11:15:50.1069 ereport.fs.zfs.checksum
0x38431f4a12900401

//Jim

[Tim Cook]

Post by Jim Klimov
Hello all, thanks for comments.
One thing I know from experience when mirroring hard drives is if

Post by Jim Bolmgren-jim99
the master has problems with broken files it will mirror the same
issues over to the slave. I experienced boot issues with an old
Windows server like that once.

* To Jim: ZFS mirroring is better than others (including Linux md
and motherboard-builtin software RAID) in exactly this regard: there
is no single trusted "master disk". There are checksums of every
block, so that errors are known and not replicated. And can be fixed
if another copy is available and thanks to checksums - known to be good.
ECC is not a "requirement" but an essential recommendation. For example
on this box without ECC support, I can not be certain if any problems
with RAM exist at all and/or are the reason for errors I see.
* To Saso: you outlined the same procedure I followed on that box.
I did not re-test this yet with a newer release, so am ready to write
this off as a glitch in older software which may have been fixed since.
At least it is nice to know that it probably does not need fixing
anymore :)
# zpool get failmode rpool
NAME PROPERTY VALUE SOURCE
rpool failmode continue local

You had it at continue previously or just changed it? I'd be curious to
see if that fixes the issue on your current release if you re-test.

Post by Jim Klimov
* fmdump took quite a while to get all the reports from back in
2011 onto my terminal, but most of that is checksum/data errors.
The few "scsi" errors listed below started after I issued smartctl
commands to check the disks and are not seen in the earlier days.
Dec 20 12:31:52.1691 ereport.fs.zfs.data 0xc4d4e4587c700001
Dec 20 12:31:52.1691 ereport.fs.zfs.checksum 0xc4d4e4587c700001
Dec 20 12:31:52.1691 ereport.fs.zfs.checksum 0xc4d4e4587c700001
Dec 20 12:31:52.1691 ereport.fs.zfs.checksum 0xc4d4e4587c700001
Dec 20 12:31:52.1691 ereport.fs.zfs.checksum 0xc4d4e4587c700001
Dec 20 12:31:52.1691 ereport.fs.zfs.checksum 0xc4d4e4587c700001
Dec 20 12:31:52.1691 ereport.fs.zfs.checksum 0xc4d4e4587c700001
Dec 20 12:31:52.1691 ereport.fs.zfs.checksum 0xc4d4e4587c700001
Dec 20 12:31:52.1691 ereport.fs.zfs.checksum 0xc4d4e4587c700001
Dec 20 12:31:52.1691 ereport.fs.zfs.checksum 0xc4d4e4587c700001
Dec 20 12:31:52.1691 ereport.fs.zfs.checksum 0xc4d4e4587c700001
Dec 20 12:31:52.1691 ereport.fs.zfs.checksum 0xc4d4e4587c700001
Dec 20 12:31:52.1691 ereport.fs.zfs.checksum 0xc4d4e4587c700001
Dec 20 12:32:00.0861 ereport.fs.zfs.data 0xc4f26304b1d00401
Dec 20 12:32:00.0861 ereport.fs.zfs.checksum 0xc4f26304b1d00401
Dec 20 12:32:00.0861 ereport.fs.zfs.checksum 0xc4f26304b1d00401
Dec 20 12:32:00.0861 ereport.fs.zfs.checksum 0xc4f26304b1d00401
Dec 20 12:32:00.0861 ereport.fs.zfs.checksum 0xc4f26304b1d00401
Dec 20 12:32:00.0861 ereport.fs.zfs.checksum 0xc4f26304b1d00401
Dec 20 12:32:00.0861 ereport.fs.zfs.checksum 0xc4f26304b1d00401
Dec 20 12:32:00.0861 ereport.fs.zfs.checksum 0xc4f26304b1d00401
Dec 20 12:32:00.0861 ereport.fs.zfs.checksum 0xc4f26304b1d00401
Dec 20 12:32:00.0861 ereport.fs.zfs.checksum 0xc4f26304b1d00401
Dec 20 12:32:00.0861 ereport.fs.zfs.checksum 0xc4f26304b1d00401
Dec 20 12:32:00.0861 ereport.fs.zfs.checksum 0xc4f26304b1d00401
Dec 20 12:32:00.0861 ereport.fs.zfs.checksum 0xc4f26304b1d00401
Dec 20 12:38:44.8779 ereport.fs.zfs.data 0xcad6708e44300001
Dec 20 12:38:44.8779 ereport.fs.zfs.checksum 0xcad6708e44300001
Dec 20 12:38:44.8779 ereport.fs.zfs.checksum 0xcad6708e44300001
Dec 20 12:38:44.8779 ereport.fs.zfs.checksum 0xcad6708e44300001
Dec 20 12:38:44.8779 ereport.fs.zfs.checksum 0xcad6708e44300001
Dec 20 12:38:44.8779 ereport.fs.zfs.checksum 0xcad6708e44300001
Dec 20 12:38:44.8779 ereport.fs.zfs.checksum 0xcad6708e44300001
Dec 20 12:38:44.8781 ereport.fs.zfs.checksum 0xcad6708e44300001
Dec 20 12:38:44.8779 ereport.fs.zfs.checksum 0xcad6708e44300001
Dec 20 12:38:44.8779 ereport.fs.zfs.checksum 0xcad6708e44300001
Dec 20 12:38:44.8779 ereport.fs.zfs.checksum 0xcad6708e44300001
Dec 20 12:38:44.8779 ereport.fs.zfs.checksum 0xcad6708e44300001
Dec 20 12:38:44.8779 ereport.fs.zfs.checksum 0xcad6708e44300001
Dec 20 15:02:07.5464 ereport.fs.zfs.data 0x4807967781b00401
Dec 20 15:02:07.5464 ereport.fs.zfs.checksum 0x4807967781b00401
Dec 20 15:02:07.5464 ereport.fs.zfs.checksum 0x4807967781b00401
Dec 20 15:02:07.5464 ereport.fs.zfs.checksum 0x4807967781b00401
Dec 20 15:02:07.5464 ereport.fs.zfs.checksum 0x4807967781b00401
Dec 20 15:02:07.5464 ereport.fs.zfs.checksum 0x4807967781b00401
Dec 20 15:02:07.5464 ereport.fs.zfs.checksum 0x4807967781b00401
Dec 20 15:02:07.5464 ereport.fs.zfs.checksum 0x4807967781b00401
Dec 20 15:02:07.5464 ereport.fs.zfs.checksum 0x4807967781b00401
Dec 20 15:02:07.5464 ereport.fs.zfs.checksum 0x4807967781b00401
Dec 20 15:02:07.5464 ereport.fs.zfs.checksum 0x4807967781b00401
Dec 20 15:02:07.5464 ereport.fs.zfs.checksum 0x4807967781b00401
Dec 20 15:02:07.5464 ereport.fs.zfs.checksum 0x4807967781b00401
Dec 20 15:25:26.6338 ereport.fs.zfs.data 0x5c63e44016700401
Dec 20 15:25:26.6338 ereport.fs.zfs.checksum 0x5c63e44016700401
Dec 20 15:25:26.6338 ereport.fs.zfs.checksum 0x5c63e44016700401
Dec 20 15:25:26.6338 ereport.fs.zfs.checksum 0x5c63e44016700401
Dec 20 15:25:26.6338 ereport.fs.zfs.checksum 0x5c63e44016700401
Dec 20 15:25:26.6338 ereport.fs.zfs.checksum 0x5c63e44016700401
Dec 20 15:25:26.6338 ereport.fs.zfs.checksum 0x5c63e44016700401
Dec 20 15:25:26.6338 ereport.fs.zfs.checksum 0x5c63e44016700401
Dec 20 15:25:26.6338 ereport.fs.zfs.checksum 0x5c63e44016700401
Dec 20 15:25:26.6338 ereport.fs.zfs.checksum 0x5c63e44016700401
Dec 20 15:25:26.6338 ereport.fs.zfs.checksum 0x5c63e44016700401
Dec 20 15:25:26.6338 ereport.fs.zfs.checksum 0x5c63e44016700401
Dec 20 15:25:26.6338 ereport.fs.zfs.checksum 0x5c63e44016700401
Dec 20 15:45:25.0675 ereport.fs.zfs.data 0x6dd4a8c6a6100401
Dec 20 15:45:25.0675 ereport.fs.zfs.checksum 0x6dd4a8c6a6100401
Dec 20 15:45:25.0675 ereport.fs.zfs.checksum 0x6dd4a8c6a6100401
Dec 20 15:45:25.0675 ereport.fs.zfs.checksum 0x6dd4a8c6a6100401
Dec 20 15:45:25.0675 ereport.fs.zfs.checksum 0x6dd4a8c6a6100401
Dec 20 15:45:25.0675 ereport.fs.zfs.checksum 0x6dd4a8c6a6100401
Dec 20 15:45:25.0675 ereport.fs.zfs.checksum 0x6dd4a8c6a6100401
Dec 20 15:45:25.0675 ereport.fs.zfs.checksum 0x6dd4a8c6a6100401
Dec 20 15:45:25.0675 ereport.fs.zfs.checksum 0x6dd4a8c6a6100401
Dec 20 15:45:25.0675 ereport.fs.zfs.checksum 0x6dd4a8c6a6100401
Dec 20 15:45:25.0675 ereport.fs.zfs.checksum 0x6dd4a8c6a6100401
Dec 20 15:45:25.0675 ereport.fs.zfs.checksum 0x6dd4a8c6a6100401
Dec 20 15:45:25.0675 ereport.fs.zfs.checksum 0x6dd4a8c6a6100401
Dec 21 00:06:33.1621 ereport.fs.zfs.checksum 0x23668f4121100401
Dec 21 00:29:25.4337 ereport.fs.zfs.checksum 0x375ef60acb400001
Dec 21 11:30:27.9301 ereport.fs.zfs.data 0x7890c7825ea00001
Dec 21 11:30:27.9301 ereport.fs.zfs.checksum 0x7890c7825ea00001
Dec 21 11:30:27.9301 ereport.fs.zfs.checksum 0x7890c7825ea00001
Dec 21 11:30:27.9301 ereport.fs.zfs.checksum 0x7890c7825ea00001
Dec 21 11:30:27.9301 ereport.fs.zfs.checksum 0x7890c7825ea00001
Dec 21 11:30:27.9301 ereport.fs.zfs.checksum 0x7890c7825ea00001
Dec 21 11:30:27.9301 ereport.fs.zfs.checksum 0x7890c7825ea00001
Dec 21 11:30:27.9301 ereport.fs.zfs.checksum 0x7890c7825ea00001
Dec 21 11:30:27.9301 ereport.fs.zfs.checksum 0x7890c7825ea00001
Dec 21 11:30:27.9301 ereport.fs.zfs.checksum 0x7890c7825ea00001
Dec 21 11:30:27.9301 ereport.fs.zfs.checksum 0x7890c7825ea00001
Dec 21 11:30:27.9301 ereport.fs.zfs.checksum 0x7890c7825ea00001
Dec 21 11:30:27.9301 ereport.fs.zfs.checksum 0x7890c7825ea00001
Dec 21 12:19:54.9335 ereport.fs.zfs.checksum 0xa3be5b89cf600401
Dec 21 12:21:05.7922 ereport.fs.zfs.checksum 0xa4c6578815c00001
Dec 21 12:22:25.0750 ereport.fs.zfs.checksum 0xa5edb5ccbd400401
Dec 21 12:22:54.1633 ereport.fs.zfs.checksum 0xa65a142aaec00001
Dec 21 13:43:10.4746 ereport.io.scsi.cmd.disk.dev.rqs.derr
0x10438d031a600001
Dec 21 13:43:16.1539 ereport.io.scsi.cmd.disk.dev.rqs.derr
0x1058b539fe100001
Dec 21 13:44:22.0787 ereport.io.scsi.cmd.disk.dev.rqs.derr
0x114e4bf547400001
Dec 21 13:44:26.9645 ereport.io.scsi.cmd.disk.dev.rqs.derr
0x11607f7778300401
Dec 21 13:44:29.2503 ereport.io.scsi.cmd.disk.dev.rqs.derr
0x1169036a8de00401
Dec 21 13:44:31.0954 ereport.io.scsi.cmd.disk.dev.rqs.derr
0x116fe2f35d100401
Dec 21 13:44:33.1227 ereport.io.scsi.cmd.disk.dev.rqs.derr
0x11777061a2c00401
Dec 21 13:54:29.4339 ereport.io.scsi.cmd.disk.dev.rqs.derr
0x1a25947164300401
Dec 21 13:54:33.6320 ereport.io.scsi.cmd.disk.dev.rqs.derr
0x1a353810fbe00001
Dec 21 13:54:35.5247 ereport.io.scsi.cmd.disk.dev.rqs.derr
0x1a3c450c52400001
Dec 21 13:54:37.3388 ereport.io.scsi.cmd.disk.dev.rqs.derr
0x1a43071f9f900401
Dec 21 13:54:39.4122 ereport.io.scsi.cmd.disk.dev.rqs.derr
0x1a4ac073bbd00001
Dec 21 14:32:21.3726 ereport.io.scsi.cmd.disk.dev.rqs.derr
0x3b358e413d000001
Dec 21 14:32:26.9309 ereport.io.scsi.cmd.disk.dev.rqs.derr
0x3b4a4330d3300401
Dec 21 14:32:28.6441 ereport.io.scsi.cmd.disk.dev.rqs.derr
0x3b50a5187d200401
Dec 21 14:32:31.0320 ereport.io.scsi.cmd.disk.dev.rqs.derr
0x3b598a653d500401
Dec 21 14:32:32.8426 ereport.io.scsi.cmd.disk.dev.rqs.derr
0x3b60493cbcc00401
Dec 21 14:33:32.8827 ereport.fs.zfs.checksum 0x3c3ff5586b600001
Dec 21 14:35:46.3749 ereport.fs.zfs.checksum 0x3e31442abf300001
Dec 21 14:35:46.3749 ereport.fs.zfs.checksum 0x3e31442abf300001
Dec 21 14:35:46.4594 ereport.fs.zfs.checksum 0x3e3194c997a00401
Dec 21 14:35:46.4594 ereport.fs.zfs.checksum 0x3e3194c997a00401
Dec 21 14:35:50.6820 ereport.fs.zfs.checksum 0x3e414fe468000401
Dec 21 14:35:51.1294 ereport.fs.zfs.checksum 0x3e42fa84d6500001
Dec 21 14:35:51.1294 ereport.fs.zfs.checksum 0x3e42fa84d6500001
Dec 21 14:35:54.3318 ereport.fs.zfs.checksum 0x3e4ee8a786b00001
Dec 21 14:35:54.3318 ereport.fs.zfs.checksum 0x3e4ee8a786b00001
Dec 21 14:42:11.9818 ereport.fs.zfs.checksum 0x43cdd06b51e00401
Dec 21 14:42:11.9818 ereport.fs.zfs.checksum 0x43cdd06b51e00401
Dec 21 14:42:12.9112 ereport.fs.zfs.checksum 0x43d146c7b3800401
Dec 21 14:42:12.9112 ereport.fs.zfs.checksum 0x43d146c7b3800401
Dec 21 14:49:18.4714 ereport.io.scsi.cmd.disk.dev.rqs.derr
0x4a02ad6a0f500401
Dec 21 14:49:22.7455 ereport.io.scsi.cmd.disk.dev.rqs.derr
0x4a1299a8da200401
Dec 21 14:49:23.4798 ereport.io.scsi.cmd.disk.dev.rqs.derr
0x4a1555f048900401
Dec 21 14:49:25.2605 ereport.io.scsi.cmd.disk.dev.rqs.derr
0x4a1bf84174c00401
Dec 21 14:49:26.8081 ereport.io.scsi.cmd.disk.dev.rqs.derr
0x4a21bc41f8e00401
Dec 21 15:06:37.8081 ereport.fs.zfs.checksum 0x5922a89687100001
Dec 21 15:06:37.8081 ereport.fs.zfs.checksum 0x5922a89687100001
Dec 21 16:09:16.0315 ereport.io.scsi.cmd.disk.dev.rqs.derr
0x8fd3964b04c00001
Dec 21 16:09:19.6828 ereport.io.scsi.cmd.disk.dev.rqs.derr
0x8fe1309461b00001
Dec 21 16:09:20.8254 ereport.io.scsi.cmd.disk.dev.rqs.derr
0x8fe5724086c00401
Dec 21 16:09:21.9860 ereport.io.scsi.cmd.disk.dev.rqs.derr
0x8fe9c5325a200401
Dec 21 16:09:23.1430 ereport.io.scsi.cmd.disk.dev.rqs.derr
0x8fee14a0d3c00001
Dec 21 16:09:41.5650 ereport.fs.zfs.checksum 0x9032b5ead3100401
Dec 21 16:09:41.5650 ereport.fs.zfs.checksum 0x9032b5ead3100401
Dec 21 16:11:33.0525 ereport.fs.zfs.checksum 0x91d20c40d8100001
Dec 21 16:11:33.0525 ereport.fs.zfs.checksum 0x91d20c40d8100001
Dec 21 16:11:33.1082 ereport.fs.zfs.checksum 0x91d24153a8200001
Dec 21 16:11:33.1082 ereport.fs.zfs.checksum 0x91d24153a8200001
Dec 21 16:11:36.6849 ereport.fs.zfs.checksum 0x91df9469f2900001
Dec 21 16:11:36.6849 ereport.fs.zfs.checksum 0x91df9469f2900001
Dec 21 16:11:37.1319 ereport.fs.zfs.checksum 0x91e13ebe1cf00401
Dec 21 16:11:37.1319 ereport.fs.zfs.checksum 0x91e13ebe1cf00401
Dec 21 16:11:41.7997 ereport.fs.zfs.checksum 0x91f2a28088800001
Dec 21 16:11:41.7997 ereport.fs.zfs.checksum 0x91f2a28088800001
Dec 21 16:20:54.8897 ereport.fs.zfs.checksum 0x99ff209881d00401
Dec 21 16:20:54.8897 ereport.fs.zfs.checksum 0x99ff209881d00401
Dec 21 16:20:55.8360 ereport.fs.zfs.checksum 0x9a02a72227d00001
Dec 21 16:20:55.8360 ereport.fs.zfs.checksum 0x9a02a72227d00001
Dec 21 16:40:10.5784 ereport.fs.zfs.checksum 0xaad08db88ff00401
Dec 21 16:40:10.5784 ereport.fs.zfs.checksum 0xaad08db88ff00401
Dec 21 17:39:51.2304 ereport.fs.zfs.checksum 0xdeec03d766600001
Dec 21 17:39:51.2304 ereport.fs.zfs.checksum 0xdeec03d766600001
Dec 21 17:39:59.9474 ereport.io.scsi.cmd.disk.dev.rqs.derr
0xdf0c7d41fac00401
Dec 21 17:40:04.5210 ereport.io.scsi.cmd.disk.dev.rqs.derr
0xdf1d871db4600001
Dec 21 17:40:05.6893 ereport.io.scsi.cmd.disk.dev.rqs.derr
0xdf21e14ba3a00001
Dec 21 17:40:06.8237 ereport.io.scsi.cmd.disk.dev.rqs.derr
0xdf261b31d9100001
Dec 21 17:40:07.9910 ereport.io.scsi.cmd.disk.dev.rqs.derr
0xdf2a747e62f00001
Dec 21 17:43:37.1941 ereport.fs.zfs.checksum 0xe235d2cf5dc00001
Dec 21 17:43:37.1941 ereport.fs.zfs.checksum 0xe235d2cf5dc00001
Dec 21 17:43:37.2973 ereport.fs.zfs.checksum 0xe2363535aa500401
Dec 21 17:43:37.2973 ereport.fs.zfs.checksum 0xe2363535aa500401
Dec 21 17:43:40.8574 ereport.fs.zfs.checksum 0xe243788409000401
Dec 21 17:43:40.8574 ereport.fs.zfs.checksum 0xe243788409000401
Dec 21 17:43:41.3875 ereport.fs.zfs.checksum 0xe245721750500401
Dec 21 17:43:41.3875 ereport.fs.zfs.checksum 0xe245721750500401
Dec 21 17:43:45.3929 ereport.fs.zfs.checksum 0xe2545e0f14e00401
Dec 21 17:43:45.3929 ereport.fs.zfs.checksum 0xe2545e0f14e00401
Dec 21 17:53:32.0557 ereport.fs.zfs.checksum 0xeaddee9f10c00401
Dec 21 17:53:32.0557 ereport.fs.zfs.checksum 0xeaddee9f10c00401
Dec 21 17:53:32.8619 ereport.fs.zfs.checksum 0xeae0ef883f600001
Dec 21 17:53:32.8619 ereport.fs.zfs.checksum 0xeae0ef883f600001
Dec 21 18:13:13.7183 ereport.fs.zfs.checksum 0xfc10204d46400401
Dec 21 18:13:13.7183 ereport.fs.zfs.checksum 0xfc10204d46400401
Dec 21 19:01:42.8304 ereport.io.scsi.cmd.disk.dev.rqs.derr
0x2665cd0890800401
Dec 21 19:01:47.0346 ereport.io.scsi.cmd.disk.dev.rqs.derr
0x2675769b61100401
Dec 21 19:01:51.8947 ereport.io.scsi.cmd.disk.dev.rqs.derr
0x268791a84a900001
Dec 21 19:01:54.7808 ereport.io.scsi.cmd.disk.dev.rqs.derr
0x2692523029400401
Dec 21 19:01:56.7315 ereport.io.scsi.cmd.disk.dev.rqs.derr
0x2699968233200001
Dec 21 19:40:54.1171 ereport.fs.zfs.checksum 0x07edff41b8d00401
Dec 21 19:45:45.8638 ereport.fs.zfs.checksum 0x0c2cd69bd5300001
Dec 21 19:45:45.9143 ereport.fs.zfs.checksum 0x0c2d06c29e100401
Dec 21 19:45:50.2857 ereport.fs.zfs.checksum 0x0c3d4faaf2900001
Dec 21 19:45:50.6402 ereport.fs.zfs.checksum 0x0c3ea1c8bb400001
Dec 21 19:52:20.5475 ereport.fs.zfs.checksum 0x11eb264cbfa00401
Dec 21 19:52:21.4633 ereport.fs.zfs.checksum 0x11ee8fb5ce300401
Dec 21 20:16:23.0050 ereport.fs.zfs.checksum 0x26e90e5c32f00001
Dec 22 00:00:08.7555 ereport.fs.zfs.checksum 0xea49a29745000401
Dec 22 00:04:17.6368 ereport.fs.zfs.checksum 0xede8d33a61100001
Dec 22 00:34:30.6962 ereport.fs.zfs.checksum 0x084b2b20b9000001
Dec 22 11:15:50.1069 ereport.fs.zfs.checksum 0x38431f4a12900401
//Jim
-------------------------------------------
illumos-zfs
Archives: https://www.listbox.com/member/archive/182191/=now
RSS Feed: https://www.listbox.com/member/archive/rss/182191/
21996034-de72a2b0
Modify Your Subscription: https://www.listbox.com/
member/?&
Powered by Listbox: http://www.listbox.com

-------------------------------------------
illumos-zfs
Archives: https://www.listbox.com/member/archive/182191/=now
RSS Feed: https://www.listbox.com/member/archive/rss/182191/23047029-187a0c8d
Modify Your Subscription: https://www.listbox.com/member/?member_id=23047029&id_secret=23047029-2e85923f
Powered by Listbox: http://www.listbox.com

[Saso Kiselkov]

Post by Jim Klimov
Kind of following up on that old system of mine (old OI release
with custom illumos-gate from early 2012, in particular), I found
that when I mirrored from old disk to new one and then rebooted,
and physically removed one of the drives, the system failed to
mount_vfs_root (apparently related to the NOTICE about degraded
pool).
Booting from a live media, importing the rpool and detaching the
now-missing drive fixed the issue.
However (if this situation is still in place): isn't it kind of
counter-productive to allow mirroring to increase availability
of the system and then forbid it to boot from broken mirrors -
so that in case of a dead disk, the system would be unavailable
for as long as it takes somebody to get to it physically?
Or is there some command-line toggle (or a /etc/system switch)
for the kernel to allow such boot-ups?

Just tested this on a fresh oi_151a7 install inside of a VM and it
worked like a charm. Install onto a single disk, then add a second one,
attach, installgrub, resync, poweroff, delete first disk, boot. No
problems. I suspect that you missed at least one of these steps.

[j***@cos.ru]

I always have rpools at continue, no recent changes here


Typos courtesy of my Samsung Mobile

-------- ИсхПЎМПе сППбщеМОе --------
От: Tim Cook <***@cook.ms>
Дата: 2013.12.22 21:28 (GMT+04:00)
КПЌу: ***@lists.illumos.org
ТеЌа: Re: [zfs] Boot from degraded rpool



On Sun, Dec 22, 2013 at 1:32 AM, Jim Klimov <***@cos.ru> wrote:
Hello all, thanks for comments.

One thing I know from experience when mirroring hard drives is if

Post by Jim Bolmgren-jim99
the master has problems with broken files it will mirror the same
issues over to the slave. I experienced boot issues with an old

Windows server like that once.

* To Jim: ZFS mirroring is better than others (including Linux md
and motherboard-builtin software RAID) in exactly this regard: there
is no single trusted "master disk". There are checksums of every
block, so that errors are known and not replicated. And can be fixed
if another copy is available and thanks to checksums - known to be good.

ECC is not a "requirement" but an essential recommendation. For example
on this box without ECC support, I can not be certain if any problems
with RAM exist at all and/or are the reason for errors I see.



* To Saso: you outlined the same procedure I followed on that box.
I did not re-test this yet with a newer release, so am ready to write
this off as a glitch in older software which may have been fixed since.
At least it is nice to know that it probably does not need fixing
anymore :)

* The failmode, just in case, is "continue":

# zpool get failmode rpool
NAME   PROPERTY  VALUE     SOURCE
rpool  failmode  continue  local




You had it at continue previously or just changed it?  I'd be curious to see if that fixes the issue on your current release if you re-test.





 
* fmdump took quite a while to get all the reports from back in
2011 onto my terminal, but most of that is checksum/data errors.
The few "scsi" errors listed below started after I issued smartctl
commands to check the disks and are not seen in the earlier days.

Dec 20 12:31:52.1691 ereport.fs.zfs.data 0xc4d4e4587c700001
Dec 20 12:31:52.1691 ereport.fs.zfs.checksum 0xc4d4e4587c700001
Dec 20 12:31:52.1691 ereport.fs.zfs.checksum 0xc4d4e4587c700001
Dec 20 12:31:52.1691 ereport.fs.zfs.checksum 0xc4d4e4587c700001
Dec 20 12:31:52.1691 ereport.fs.zfs.checksum 0xc4d4e4587c700001
Dec 20 12:31:52.1691 ereport.fs.zfs.checksum 0xc4d4e4587c700001
Dec 20 12:31:52.1691 ereport.fs.zfs.checksum 0xc4d4e4587c700001
Dec 20 12:31:52.1691 ereport.fs.zfs.checksum 0xc4d4e4587c700001
Dec 20 12:31:52.1691 ereport.fs.zfs.checksum 0xc4d4e4587c700001
Dec 20 12:31:52.1691 ereport.fs.zfs.checksum 0xc4d4e4587c700001
Dec 20 12:31:52.1691 ereport.fs.zfs.checksum 0xc4d4e4587c700001
Dec 20 12:31:52.1691 ereport.fs.zfs.checksum 0xc4d4e4587c700001
Dec 20 12:31:52.1691 ereport.fs.zfs.checksum 0xc4d4e4587c700001
Dec 20 12:32:00.0861 ereport.fs.zfs.data 0xc4f26304b1d00401
Dec 20 12:32:00.0861 ereport.fs.zfs.checksum 0xc4f26304b1d00401
Dec 20 12:32:00.0861 ereport.fs.zfs.checksum 0xc4f26304b1d00401
Dec 20 12:32:00.0861 ereport.fs.zfs.checksum 0xc4f26304b1d00401
Dec 20 12:32:00.0861 ereport.fs.zfs.checksum 0xc4f26304b1d00401
Dec 20 12:32:00.0861 ereport.fs.zfs.checksum 0xc4f26304b1d00401
Dec 20 12:32:00.0861 ereport.fs.zfs.checksum 0xc4f26304b1d00401
Dec 20 12:32:00.0861 ereport.fs.zfs.checksum 0xc4f26304b1d00401
Dec 20 12:32:00.0861 ereport.fs.zfs.checksum 0xc4f26304b1d00401
Dec 20 12:32:00.0861 ereport.fs.zfs.checksum 0xc4f26304b1d00401
Dec 20 12:32:00.0861 ereport.fs.zfs.checksum 0xc4f26304b1d00401
Dec 20 12:32:00.0861 ereport.fs.zfs.checksum 0xc4f26304b1d00401
Dec 20 12:32:00.0861 ereport.fs.zfs.checksum 0xc4f26304b1d00401
Dec 20 12:38:44.8779 ereport.fs.zfs.data 0xcad6708e44300001
Dec 20 12:38:44.8779 ereport.fs.zfs.checksum 0xcad6708e44300001
Dec 20 12:38:44.8779 ereport.fs.zfs.checksum 0xcad6708e44300001
Dec 20 12:38:44.8779 ereport.fs.zfs.checksum 0xcad6708e44300001
Dec 20 12:38:44.8779 ereport.fs.zfs.checksum 0xcad6708e44300001
Dec 20 12:38:44.8779 ereport.fs.zfs.checksum 0xcad6708e44300001
Dec 20 12:38:44.8779 ereport.fs.zfs.checksum 0xcad6708e44300001
Dec 20 12:38:44.8781 ereport.fs.zfs.checksum 0xcad6708e44300001
Dec 20 12:38:44.8779 ereport.fs.zfs.checksum 0xcad6708e44300001
Dec 20 12:38:44.8779 ereport.fs.zfs.checksum 0xcad6708e44300001
Dec 20 12:38:44.8779 ereport.fs.zfs.checksum 0xcad6708e44300001
Dec 20 12:38:44.8779 ereport.fs.zfs.checksum 0xcad6708e44300001
Dec 20 12:38:44.8779 ereport.fs.zfs.checksum 0xcad6708e44300001
Dec 20 15:02:07.5464 ereport.fs.zfs.data 0x4807967781b00401
Dec 20 15:02:07.5464 ereport.fs.zfs.checksum 0x4807967781b00401
Dec 20 15:02:07.5464 ereport.fs.zfs.checksum 0x4807967781b00401
Dec 20 15:02:07.5464 ereport.fs.zfs.checksum 0x4807967781b00401
Dec 20 15:02:07.5464 ereport.fs.zfs.checksum 0x4807967781b00401
Dec 20 15:02:07.5464 ereport.fs.zfs.checksum 0x4807967781b00401
Dec 20 15:02:07.5464 ereport.fs.zfs.checksum 0x4807967781b00401
Dec 20 15:02:07.5464 ereport.fs.zfs.checksum 0x4807967781b00401
Dec 20 15:02:07.5464 ereport.fs.zfs.checksum 0x4807967781b00401
Dec 20 15:02:07.5464 ereport.fs.zfs.checksum 0x4807967781b00401
Dec 20 15:02:07.5464 ereport.fs.zfs.checksum 0x4807967781b00401
Dec 20 15:02:07.5464 ereport.fs.zfs.checksum 0x4807967781b00401
Dec 20 15:02:07.5464 ereport.fs.zfs.checksum 0x4807967781b00401
Dec 20 15:25:26.6338 ereport.fs.zfs.data 0x5c63e44016700401
Dec 20 15:25:26.6338 ereport.fs.zfs.checksum 0x5c63e44016700401
Dec 20 15:25:26.6338 ereport.fs.zfs.checksum 0x5c63e44016700401
Dec 20 15:25:26.6338 ereport.fs.zfs.checksum 0x5c63e44016700401
Dec 20 15:25:26.6338 ereport.fs.zfs.checksum 0x5c63e44016700401
Dec 20 15:25:26.6338 ereport.fs.zfs.checksum 0x5c63e44016700401
Dec 20 15:25:26.6338 ereport.fs.zfs.checksum 0x5c63e44016700401
Dec 20 15:25:26.6338 ereport.fs.zfs.checksum 0x5c63e44016700401
Dec 20 15:25:26.6338 ereport.fs.zfs.checksum 0x5c63e44016700401
Dec 20 15:25:26.6338 ereport.fs.zfs.checksum 0x5c63e44016700401
Dec 20 15:25:26.6338 ereport.fs.zfs.checksum 0x5c63e44016700401
Dec 20 15:25:26.6338 ereport.fs.zfs.checksum 0x5c63e44016700401
Dec 20 15:25:26.6338 ereport.fs.zfs.checksum 0x5c63e44016700401
Dec 20 15:45:25.0675 ereport.fs.zfs.data 0x6dd4a8c6a6100401
Dec 20 15:45:25.0675 ereport.fs.zfs.checksum 0x6dd4a8c6a6100401
Dec 20 15:45:25.0675 ereport.fs.zfs.checksum 0x6dd4a8c6a6100401
Dec 20 15:45:25.0675 ereport.fs.zfs.checksum 0x6dd4a8c6a6100401
Dec 20 15:45:25.0675 ereport.fs.zfs.checksum 0x6dd4a8c6a6100401
Dec 20 15:45:25.0675 ereport.fs.zfs.checksum 0x6dd4a8c6a6100401
Dec 20 15:45:25.0675 ereport.fs.zfs.checksum 0x6dd4a8c6a6100401
Dec 20 15:45:25.0675 ereport.fs.zfs.checksum 0x6dd4a8c6a6100401
Dec 20 15:45:25.0675 ereport.fs.zfs.checksum 0x6dd4a8c6a6100401
Dec 20 15:45:25.0675 ereport.fs.zfs.checksum 0x6dd4a8c6a6100401
Dec 20 15:45:25.0675 ereport.fs.zfs.checksum 0x6dd4a8c6a6100401
Dec 20 15:45:25.0675 ereport.fs.zfs.checksum 0x6dd4a8c6a6100401
Dec 20 15:45:25.0675 ereport.fs.zfs.checksum 0x6dd4a8c6a6100401
Dec 21 00:06:33.1621 ereport.fs.zfs.checksum 0x23668f4121100401
Dec 21 00:29:25.4337 ereport.fs.zfs.checksum 0x375ef60acb400001
Dec 21 11:30:27.9301 ereport.fs.zfs.data 0x7890c7825ea00001
Dec 21 11:30:27.9301 ereport.fs.zfs.checksum 0x7890c7825ea00001
Dec 21 11:30:27.9301 ereport.fs.zfs.checksum 0x7890c7825ea00001
Dec 21 11:30:27.9301 ereport.fs.zfs.checksum 0x7890c7825ea00001
Dec 21 11:30:27.9301 ereport.fs.zfs.checksum 0x7890c7825ea00001
Dec 21 11:30:27.9301 ereport.fs.zfs.checksum 0x7890c7825ea00001
Dec 21 11:30:27.9301 ereport.fs.zfs.checksum 0x7890c7825ea00001
Dec 21 11:30:27.9301 ereport.fs.zfs.checksum 0x7890c7825ea00001
Dec 21 11:30:27.9301 ereport.fs.zfs.checksum 0x7890c7825ea00001
Dec 21 11:30:27.9301 ereport.fs.zfs.checksum 0x7890c7825ea00001
Dec 21 11:30:27.9301 ereport.fs.zfs.checksum 0x7890c7825ea00001
Dec 21 11:30:27.9301 ereport.fs.zfs.checksum 0x7890c7825ea00001
Dec 21 11:30:27.9301 ereport.fs.zfs.checksum 0x7890c7825ea00001
Dec 21 12:19:54.9335 ereport.fs.zfs.checksum 0xa3be5b89cf600401
Dec 21 12:21:05.7922 ereport.fs.zfs.checksum 0xa4c6578815c00001
Dec 21 12:22:25.0750 ereport.fs.zfs.checksum 0xa5edb5ccbd400401
Dec 21 12:22:54.1633 ereport.fs.zfs.checksum 0xa65a142aaec00001
Dec 21 13:43:10.4746 ereport.io.scsi.cmd.disk.dev.rqs.derr 0x10438d031a600001
Dec 21 13:43:16.1539 ereport.io.scsi.cmd.disk.dev.rqs.derr 0x1058b539fe100001
Dec 21 13:44:22.0787 ereport.io.scsi.cmd.disk.dev.rqs.derr 0x114e4bf547400001
Dec 21 13:44:26.9645 ereport.io.scsi.cmd.disk.dev.rqs.derr 0x11607f7778300401
Dec 21 13:44:29.2503 ereport.io.scsi.cmd.disk.dev.rqs.derr 0x1169036a8de00401
Dec 21 13:44:31.0954 ereport.io.scsi.cmd.disk.dev.rqs.derr 0x116fe2f35d100401
Dec 21 13:44:33.1227 ereport.io.scsi.cmd.disk.dev.rqs.derr 0x11777061a2c00401
Dec 21 13:54:29.4339 ereport.io.scsi.cmd.disk.dev.rqs.derr 0x1a25947164300401
Dec 21 13:54:33.6320 ereport.io.scsi.cmd.disk.dev.rqs.derr 0x1a353810fbe00001
Dec 21 13:54:35.5247 ereport.io.scsi.cmd.disk.dev.rqs.derr 0x1a3c450c52400001
Dec 21 13:54:37.3388 ereport.io.scsi.cmd.disk.dev.rqs.derr 0x1a43071f9f900401
Dec 21 13:54:39.4122 ereport.io.scsi.cmd.disk.dev.rqs.derr 0x1a4ac073bbd00001
Dec 21 14:32:21.3726 ereport.io.scsi.cmd.disk.dev.rqs.derr 0x3b358e413d000001
Dec 21 14:32:26.9309 ereport.io.scsi.cmd.disk.dev.rqs.derr 0x3b4a4330d3300401
Dec 21 14:32:28.6441 ereport.io.scsi.cmd.disk.dev.rqs.derr 0x3b50a5187d200401
Dec 21 14:32:31.0320 ereport.io.scsi.cmd.disk.dev.rqs.derr 0x3b598a653d500401
Dec 21 14:32:32.8426 ereport.io.scsi.cmd.disk.dev.rqs.derr 0x3b60493cbcc00401
Dec 21 14:33:32.8827 ereport.fs.zfs.checksum 0x3c3ff5586b600001
Dec 21 14:35:46.3749 ereport.fs.zfs.checksum 0x3e31442abf300001
Dec 21 14:35:46.3749 ereport.fs.zfs.checksum 0x3e31442abf300001
Dec 21 14:35:46.4594 ereport.fs.zfs.checksum 0x3e3194c997a00401
Dec 21 14:35:46.4594 ereport.fs.zfs.checksum 0x3e3194c997a00401
Dec 21 14:35:50.6820 ereport.fs.zfs.checksum 0x3e414fe468000401
Dec 21 14:35:51.1294 ereport.fs.zfs.checksum 0x3e42fa84d6500001
Dec 21 14:35:51.1294 ereport.fs.zfs.checksum 0x3e42fa84d6500001
Dec 21 14:35:54.3318 ereport.fs.zfs.checksum 0x3e4ee8a786b00001
Dec 21 14:35:54.3318 ereport.fs.zfs.checksum 0x3e4ee8a786b00001
Dec 21 14:42:11.9818 ereport.fs.zfs.checksum 0x43cdd06b51e00401
Dec 21 14:42:11.9818 ereport.fs.zfs.checksum 0x43cdd06b51e00401
Dec 21 14:42:12.9112 ereport.fs.zfs.checksum 0x43d146c7b3800401
Dec 21 14:42:12.9112 ereport.fs.zfs.checksum 0x43d146c7b3800401
Dec 21 14:49:18.4714 ereport.io.scsi.cmd.disk.dev.rqs.derr 0x4a02ad6a0f500401
Dec 21 14:49:22.7455 ereport.io.scsi.cmd.disk.dev.rqs.derr 0x4a1299a8da200401
Dec 21 14:49:23.4798 ereport.io.scsi.cmd.disk.dev.rqs.derr 0x4a1555f048900401
Dec 21 14:49:25.2605 ereport.io.scsi.cmd.disk.dev.rqs.derr 0x4a1bf84174c00401
Dec 21 14:49:26.8081 ereport.io.scsi.cmd.disk.dev.rqs.derr 0x4a21bc41f8e00401
Dec 21 15:06:37.8081 ereport.fs.zfs.checksum 0x5922a89687100001
Dec 21 15:06:37.8081 ereport.fs.zfs.checksum 0x5922a89687100001
Dec 21 16:09:16.0315 ereport.io.scsi.cmd.disk.dev.rqs.derr 0x8fd3964b04c00001
Dec 21 16:09:19.6828 ereport.io.scsi.cmd.disk.dev.rqs.derr 0x8fe1309461b00001
Dec 21 16:09:20.8254 ereport.io.scsi.cmd.disk.dev.rqs.derr 0x8fe5724086c00401
Dec 21 16:09:21.9860 ereport.io.scsi.cmd.disk.dev.rqs.derr 0x8fe9c5325a200401
Dec 21 16:09:23.1430 ereport.io.scsi.cmd.disk.dev.rqs.derr 0x8fee14a0d3c00001
Dec 21 16:09:41.5650 ereport.fs.zfs.checksum 0x9032b5ead3100401
Dec 21 16:09:41.5650 ereport.fs.zfs.checksum 0x9032b5ead3100401
Dec 21 16:11:33.0525 ereport.fs.zfs.checksum 0x91d20c40d8100001
Dec 21 16:11:33.0525 ereport.fs.zfs.checksum 0x91d20c40d8100001
Dec 21 16:11:33.1082 ereport.fs.zfs.checksum 0x91d24153a8200001
Dec 21 16:11:33.1082 ereport.fs.zfs.checksum 0x91d24153a8200001
Dec 21 16:11:36.6849 ereport.fs.zfs.checksum 0x91df9469f2900001
Dec 21 16:11:36.6849 ereport.fs.zfs.checksum 0x91df9469f2900001
Dec 21 16:11:37.1319 ereport.fs.zfs.checksum 0x91e13ebe1cf00401
Dec 21 16:11:37.1319 ereport.fs.zfs.checksum 0x91e13ebe1cf00401
Dec 21 16:11:41.7997 ereport.fs.zfs.checksum 0x91f2a28088800001
Dec 21 16:11:41.7997 ereport.fs.zfs.checksum 0x91f2a28088800001
Dec 21 16:20:54.8897 ereport.fs.zfs.checksum 0x99ff209881d00401
Dec 21 16:20:54.8897 ereport.fs.zfs.checksum 0x99ff209881d00401
Dec 21 16:20:55.8360 ereport.fs.zfs.checksum 0x9a02a72227d00001
Dec 21 16:20:55.8360 ereport.fs.zfs.checksum 0x9a02a72227d00001
Dec 21 16:40:10.5784 ereport.fs.zfs.checksum 0xaad08db88ff00401
Dec 21 16:40:10.5784 ereport.fs.zfs.checksum 0xaad08db88ff00401
Dec 21 17:39:51.2304 ereport.fs.zfs.checksum 0xdeec03d766600001
Dec 21 17:39:51.2304 ereport.fs.zfs.checksum 0xdeec03d766600001
Dec 21 17:39:59.9474 ereport.io.scsi.cmd.disk.dev.rqs.derr 0xdf0c7d41fac00401
Dec 21 17:40:04.5210 ereport.io.scsi.cmd.disk.dev.rqs.derr 0xdf1d871db4600001
Dec 21 17:40:05.6893 ereport.io.scsi.cmd.disk.dev.rqs.derr 0xdf21e14ba3a00001
Dec 21 17:40:06.8237 ereport.io.scsi.cmd.disk.dev.rqs.derr 0xdf261b31d9100001
Dec 21 17:40:07.9910 ereport.io.scsi.cmd.disk.dev.rqs.derr 0xdf2a747e62f00001
Dec 21 17:43:37.1941 ereport.fs.zfs.checksum 0xe235d2cf5dc00001
Dec 21 17:43:37.1941 ereport.fs.zfs.checksum 0xe235d2cf5dc00001
Dec 21 17:43:37.2973 ereport.fs.zfs.checksum 0xe2363535aa500401
Dec 21 17:43:37.2973 ereport.fs.zfs.checksum 0xe2363535aa500401
Dec 21 17:43:40.8574 ereport.fs.zfs.checksum 0xe243788409000401
Dec 21 17:43:40.8574 ereport.fs.zfs.checksum 0xe243788409000401
Dec 21 17:43:41.3875 ereport.fs.zfs.checksum 0xe245721750500401
Dec 21 17:43:41.3875 ereport.fs.zfs.checksum 0xe245721750500401
Dec 21 17:43:45.3929 ereport.fs.zfs.checksum 0xe2545e0f14e00401
Dec 21 17:43:45.3929 ereport.fs.zfs.checksum 0xe2545e0f14e00401
Dec 21 17:53:32.0557 ereport.fs.zfs.checksum 0xeaddee9f10c00401
Dec 21 17:53:32.0557 ereport.fs.zfs.checksum 0xeaddee9f10c00401
Dec 21 17:53:32.8619 ereport.fs.zfs.checksum 0xeae0ef883f600001
Dec 21 17:53:32.8619 ereport.fs.zfs.checksum 0xeae0ef883f600001
Dec 21 18:13:13.7183 ereport.fs.zfs.checksum 0xfc10204d46400401
Dec 21 18:13:13.7183 ereport.fs.zfs.checksum 0xfc10204d46400401
Dec 21 19:01:42.8304 ereport.io.scsi.cmd.disk.dev.rqs.derr 0x2665cd0890800401
Dec 21 19:01:47.0346 ereport.io.scsi.cmd.disk.dev.rqs.derr 0x2675769b61100401
Dec 21 19:01:51.8947 ereport.io.scsi.cmd.disk.dev.rqs.derr 0x268791a84a900001
Dec 21 19:01:54.7808 ereport.io.scsi.cmd.disk.dev.rqs.derr 0x2692523029400401
Dec 21 19:01:56.7315 ereport.io.scsi.cmd.disk.dev.rqs.derr 0x2699968233200001
Dec 21 19:40:54.1171 ereport.fs.zfs.checksum 0x07edff41b8d00401
Dec 21 19:45:45.8638 ereport.fs.zfs.checksum 0x0c2cd69bd5300001
Dec 21 19:45:45.9143 ereport.fs.zfs.checksum 0x0c2d06c29e100401
Dec 21 19:45:50.2857 ereport.fs.zfs.checksum 0x0c3d4faaf2900001
Dec 21 19:45:50.6402 ereport.fs.zfs.checksum 0x0c3ea1c8bb400001
Dec 21 19:52:20.5475 ereport.fs.zfs.checksum 0x11eb264cbfa00401
Dec 21 19:52:21.4633 ereport.fs.zfs.checksum 0x11ee8fb5ce300401
Dec 21 20:16:23.0050 ereport.fs.zfs.checksum 0x26e90e5c32f00001
Dec 22 00:00:08.7555 ereport.fs.zfs.checksum 0xea49a29745000401
Dec 22 00:04:17.6368 ereport.fs.zfs.checksum 0xede8d33a61100001
Dec 22 00:34:30.6962 ereport.fs.zfs.checksum 0x084b2b20b9000001
Dec 22 11:15:50.1069 ereport.fs.zfs.checksum 0x38431f4a12900401

//Jim



-------------------------------------------
illumos-zfs
Archives: https://www.listbox.com/member/archive/182191/=now
RSS Feed: https://www.listbox.com/member/archive/rss/182191/21996034-de72a2b0
Modify Your Subscription: https://www.listbox.com/member/?&
Powered by Listbox: http://www.listbox.com

illumos-zfs | Archives | Modify Your Subscription


-------------------------------------------
illumos-zfs
Archives: https://www.listbox.com/member/archive/182191/=now
RSS Feed: https://www.listbox.com/member/archive/rss/182191/23047029-187a0c8d
Modify Your Subscription: https://www.listbox.com/member/?member_id=23047029&id_secret=23047029-2e85923f
Powered by Listbox: http://www.listbox.com

[Richard Elling]

Hi Jim,
Are you sure you mirrored the slice and not the disk?
— richard

Post by j***@cos.ru
I always have rpools at continue, no recent changes here
Typos courtesy of my Samsung Mobile
-------- Исходное сообщение --------
Дата: 2013.12.22 21:28 (GMT+04:00)
Тема: Re: [zfs] Boot from degraded rpool
Hello all, thanks for comments.
One thing I know from experience when mirroring hard drives is if

Post by Jim Bolmgren-jim99
the master has problems with broken files it will mirror the same
issues over to the slave. I experienced boot issues with an old

Windows server like that once.
* To Jim: ZFS mirroring is better than others (including Linux md
and motherboard-builtin software RAID) in exactly this regard: there
is no single trusted "master disk". There are checksums of every
block, so that errors are known and not replicated. And can be fixed
if another copy is available and thanks to checksums - known to be good.
ECC is not a "requirement" but an essential recommendation. For example
on this box without ECC support, I can not be certain if any problems
with RAM exist at all and/or are the reason for errors I see.
* To Saso: you outlined the same procedure I followed on that box.
I did not re-test this yet with a newer release, so am ready to write
this off as a glitch in older software which may have been fixed since.
At least it is nice to know that it probably does not need fixing
anymore :)
# zpool get failmode rpool
NAME PROPERTY VALUE SOURCE
rpool failmode continue local
You had it at continue previously or just changed it? I'd be curious to see if that fixes the issue on your current release if you re-test.
* fmdump took quite a while to get all the reports from back in
2011 onto my terminal, but most of that is checksum/data errors.
The few "scsi" errors listed below started after I issued smartctl
commands to check the disks and are not seen in the earlier days.
Dec 20 12:31:52.1691 ereport.fs.zfs.data 0xc4d4e4587c700001
Dec 20 12:31:52.1691 ereport.fs.zfs.checksum 0xc4d4e4587c700001
Dec 20 12:31:52.1691 ereport.fs.zfs.checksum 0xc4d4e4587c700001
Dec 20 12:31:52.1691 ereport.fs.zfs.checksum 0xc4d4e4587c700001
Dec 20 12:31:52.1691 ereport.fs.zfs.checksum 0xc4d4e4587c700001
Dec 20 12:31:52.1691 ereport.fs.zfs.checksum 0xc4d4e4587c700001
Dec 20 12:31:52.1691 ereport.fs.zfs.checksum 0xc4d4e4587c700001
Dec 20 12:31:52.1691 ereport.fs.zfs.checksum 0xc4d4e4587c700001
Dec 20 12:31:52.1691 ereport.fs.zfs.checksum 0xc4d4e4587c700001
Dec 20 12:31:52.1691 ereport.fs.zfs.checksum 0xc4d4e4587c700001
Dec 20 12:31:52.1691 ereport.fs.zfs.checksum 0xc4d4e4587c700001
Dec 20 12:31:52.1691 ereport.fs.zfs.checksum 0xc4d4e4587c700001
Dec 20 12:31:52.1691 ereport.fs.zfs.checksum 0xc4d4e4587c700001
Dec 20 12:32:00.0861 ereport.fs.zfs.data 0xc4f26304b1d00401
Dec 20 12:32:00.0861 ereport.fs.zfs.checksum 0xc4f26304b1d00401
Dec 20 12:32:00.0861 ereport.fs.zfs.checksum 0xc4f26304b1d00401
Dec 20 12:32:00.0861 ereport.fs.zfs.checksum 0xc4f26304b1d00401
Dec 20 12:32:00.0861 ereport.fs.zfs.checksum 0xc4f26304b1d00401
Dec 20 12:32:00.0861 ereport.fs.zfs.checksum 0xc4f26304b1d00401
Dec 20 12:32:00.0861 ereport.fs.zfs.checksum 0xc4f26304b1d00401
Dec 20 12:32:00.0861 ereport.fs.zfs.checksum 0xc4f26304b1d00401
Dec 20 12:32:00.0861 ereport.fs.zfs.checksum 0xc4f26304b1d00401
Dec 20 12:32:00.0861 ereport.fs.zfs.checksum 0xc4f26304b1d00401
Dec 20 12:32:00.0861 ereport.fs.zfs.checksum 0xc4f26304b1d00401
Dec 20 12:32:00.0861 ereport.fs.zfs.checksum 0xc4f26304b1d00401
Dec 20 12:32:00.0861 ereport.fs.zfs.checksum 0xc4f26304b1d00401
Dec 20 12:38:44.8779 ereport.fs.zfs.data 0xcad6708e44300001
Dec 20 12:38:44.8779 ereport.fs.zfs.checksum 0xcad6708e44300001
Dec 20 12:38:44.8779 ereport.fs.zfs.checksum 0xcad6708e44300001
Dec 20 12:38:44.8779 ereport.fs.zfs.checksum 0xcad6708e44300001
Dec 20 12:38:44.8779 ereport.fs.zfs.checksum 0xcad6708e44300001
Dec 20 12:38:44.8779 ereport.fs.zfs.checksum 0xcad6708e44300001
Dec 20 12:38:44.8779 ereport.fs.zfs.checksum 0xcad6708e44300001
Dec 20 12:38:44.8781 ereport.fs.zfs.checksum 0xcad6708e44300001
Dec 20 12:38:44.8779 ereport.fs.zfs.checksum 0xcad6708e44300001
Dec 20 12:38:44.8779 ereport.fs.zfs.checksum 0xcad6708e44300001
Dec 20 12:38:44.8779 ereport.fs.zfs.checksum 0xcad6708e44300001
Dec 20 12:38:44.8779 ereport.fs.zfs.checksum 0xcad6708e44300001
Dec 20 12:38:44.8779 ereport.fs.zfs.checksum 0xcad6708e44300001
Dec 20 15:02:07.5464 ereport.fs.zfs.data 0x4807967781b00401
Dec 20 15:02:07.5464 ereport.fs.zfs.checksum 0x4807967781b00401
Dec 20 15:02:07.5464 ereport.fs.zfs.checksum 0x4807967781b00401
Dec 20 15:02:07.5464 ereport.fs.zfs.checksum 0x4807967781b00401
Dec 20 15:02:07.5464 ereport.fs.zfs.checksum 0x4807967781b00401
Dec 20 15:02:07.5464 ereport.fs.zfs.checksum 0x4807967781b00401
Dec 20 15:02:07.5464 ereport.fs.zfs.checksum 0x4807967781b00401
Dec 20 15:02:07.5464 ereport.fs.zfs.checksum 0x4807967781b00401
Dec 20 15:02:07.5464 ereport.fs.zfs.checksum 0x4807967781b00401
Dec 20 15:02:07.5464 ereport.fs.zfs.checksum 0x4807967781b00401
Dec 20 15:02:07.5464 ereport.fs.zfs.checksum 0x4807967781b00401
Dec 20 15:02:07.5464 ereport.fs.zfs.checksum 0x4807967781b00401
Dec 20 15:02:07.5464 ereport.fs.zfs.checksum 0x4807967781b00401
Dec 20 15:25:26.6338 ereport.fs.zfs.data 0x5c63e44016700401
Dec 20 15:25:26.6338 ereport.fs.zfs.checksum 0x5c63e44016700401
Dec 20 15:25:26.6338 ereport.fs.zfs.checksum 0x5c63e44016700401
Dec 20 15:25:26.6338 ereport.fs.zfs.checksum 0x5c63e44016700401
Dec 20 15:25:26.6338 ereport.fs.zfs.checksum 0x5c63e44016700401
Dec 20 15:25:26.6338 ereport.fs.zfs.checksum 0x5c63e44016700401
Dec 20 15:25:26.6338 ereport.fs.zfs.checksum 0x5c63e44016700401
Dec 20 15:25:26.6338 ereport.fs.zfs.checksum 0x5c63e44016700401
Dec 20 15:25:26.6338 ereport.fs.zfs.checksum 0x5c63e44016700401
Dec 20 15:25:26.6338 ereport.fs.zfs.checksum 0x5c63e44016700401
Dec 20 15:25:26.6338 ereport.fs.zfs.checksum 0x5c63e44016700401
Dec 20 15:25:26.6338 ereport.fs.zfs.checksum 0x5c63e44016700401
Dec 20 15:25:26.6338 ereport.fs.zfs.checksum 0x5c63e44016700401
Dec 20 15:45:25.0675 ereport.fs.zfs.data 0x6dd4a8c6a6100401
Dec 20 15:45:25.0675 ereport.fs.zfs.checksum 0x6dd4a8c6a6100401
Dec 20 15:45:25.0675 ereport.fs.zfs.checksum 0x6dd4a8c6a6100401
Dec 20 15:45:25.0675 ereport.fs.zfs.checksum 0x6dd4a8c6a6100401
Dec 20 15:45:25.0675 ereport.fs.zfs.checksum 0x6dd4a8c6a6100401
Dec 20 15:45:25.0675 ereport.fs.zfs.checksum 0x6dd4a8c6a6100401
Dec 20 15:45:25.0675 ereport.fs.zfs.checksum 0x6dd4a8c6a6100401
Dec 20 15:45:25.0675 ereport.fs.zfs.checksum 0x6dd4a8c6a6100401
Dec 20 15:45:25.0675 ereport.fs.zfs.checksum 0x6dd4a8c6a6100401
Dec 20 15:45:25.0675 ereport.fs.zfs.checksum 0x6dd4a8c6a6100401
Dec 20 15:45:25.0675 ereport.fs.zfs.checksum 0x6dd4a8c6a6100401
Dec 20 15:45:25.0675 ereport.fs.zfs.checksum 0x6dd4a8c6a6100401
Dec 20 15:45:25.0675 ereport.fs.zfs.checksum 0x6dd4a8c6a6100401
Dec 21 00:06:33.1621 ereport.fs.zfs.checksum 0x23668f4121100401
Dec 21 00:29:25.4337 ereport.fs.zfs.checksum 0x375ef60acb400001
Dec 21 11:30:27.9301 ereport.fs.zfs.data 0x7890c7825ea00001
Dec 21 11:30:27.9301 ereport.fs.zfs.checksum 0x7890c7825ea00001
Dec 21 11:30:27.9301 ereport.fs.zfs.checksum 0x7890c7825ea00001
Dec 21 11:30:27.9301 ereport.fs.zfs.checksum 0x7890c7825ea00001
Dec 21 11:30:27.9301 ereport.fs.zfs.checksum 0x7890c7825ea00001
Dec 21 11:30:27.9301 ereport.fs.zfs.checksum 0x7890c7825ea00001
Dec 21 11:30:27.9301 ereport.fs.zfs.checksum 0x7890c7825ea00001
Dec 21 11:30:27.9301 ereport.fs.zfs.checksum 0x7890c7825ea00001
Dec 21 11:30:27.9301 ereport.fs.zfs.checksum 0x7890c7825ea00001
Dec 21 11:30:27.9301 ereport.fs.zfs.checksum 0x7890c7825ea00001
Dec 21 11:30:27.9301 ereport.fs.zfs.checksum 0x7890c7825ea00001
Dec 21 11:30:27.9301 ereport.fs.zfs.checksum 0x7890c7825ea00001
Dec 21 11:30:27.9301 ereport.fs.zfs.checksum 0x7890c7825ea00001
Dec 21 12:19:54.9335 ereport.fs.zfs.checksum 0xa3be5b89cf600401
Dec 21 12:21:05.7922 ereport.fs.zfs.checksum 0xa4c6578815c00001
Dec 21 12:22:25.0750 ereport.fs.zfs.checksum 0xa5edb5ccbd400401
Dec 21 12:22:54.1633 ereport.fs.zfs.checksum 0xa65a142aaec00001
Dec 21 13:43:10.4746 ereport.io.scsi.cmd.disk.dev.rqs.derr 0x10438d031a600001
Dec 21 13:43:16.1539 ereport.io.scsi.cmd.disk.dev.rqs.derr 0x1058b539fe100001
Dec 21 13:44:22.0787 ereport.io.scsi.cmd.disk.dev.rqs.derr 0x114e4bf547400001
Dec 21 13:44:26.9645 ereport.io.scsi.cmd.disk.dev.rqs.derr 0x11607f7778300401
Dec 21 13:44:29.2503 ereport.io.scsi.cmd.disk.dev.rqs.derr 0x1169036a8de00401
Dec 21 13:44:31.0954 ereport.io.scsi.cmd.disk.dev.rqs.derr 0x116fe2f35d100401
Dec 21 13:44:33.1227 ereport.io.scsi.cmd.disk.dev.rqs.derr 0x11777061a2c00401
Dec 21 13:54:29.4339 ereport.io.scsi.cmd.disk.dev.rqs.derr 0x1a25947164300401
Dec 21 13:54:33.6320 ereport.io.scsi.cmd.disk.dev.rqs.derr 0x1a353810fbe00001
Dec 21 13:54:35.5247 ereport.io.scsi.cmd.disk.dev.rqs.derr 0x1a3c450c52400001
Dec 21 13:54:37.3388 ereport.io.scsi.cmd.disk.dev.rqs.derr 0x1a43071f9f900401
Dec 21 13:54:39.4122 ereport.io.scsi.cmd.disk.dev.rqs.derr 0x1a4ac073bbd00001
Dec 21 14:32:21.3726 ereport.io.scsi.cmd.disk.dev.rqs.derr 0x3b358e413d000001
Dec 21 14:32:26.9309 ereport.io.scsi.cmd.disk.dev.rqs.derr 0x3b4a4330d3300401
Dec 21 14:32:28.6441 ereport.io.scsi.cmd.disk.dev.rqs.derr 0x3b50a5187d200401
Dec 21 14:32:31.0320 ereport.io.scsi.cmd.disk.dev.rqs.derr 0x3b598a653d500401
Dec 21 14:32:32.8426 ereport.io.scsi.cmd.disk.dev.rqs.derr 0x3b60493cbcc00401
Dec 21 14:33:32.8827 ereport.fs.zfs.checksum 0x3c3ff5586b600001
Dec 21 14:35:46.3749 ereport.fs.zfs.checksum 0x3e31442abf300001
Dec 21 14:35:46.3749 ereport.fs.zfs.checksum 0x3e31442abf300001
Dec 21 14:35:46.4594 ereport.fs.zfs.checksum 0x3e3194c997a00401
Dec 21 14:35:46.4594 ereport.fs.zfs.checksum 0x3e3194c997a00401
Dec 21 14:35:50.6820 ereport.fs.zfs.checksum 0x3e414fe468000401
Dec 21 14:35:51.1294 ereport.fs.zfs.checksum 0x3e42fa84d6500001
Dec 21 14:35:51.1294 ereport.fs.zfs.checksum 0x3e42fa84d6500001
Dec 21 14:35:54.3318 ereport.fs.zfs.checksum 0x3e4ee8a786b00001
Dec 21 14:35:54.3318 ereport.fs.zfs.checksum 0x3e4ee8a786b00001
Dec 21 14:42:11.9818 ereport.fs.zfs.checksum 0x43cdd06b51e00401
Dec 21 14:42:11.9818 ereport.fs.zfs.checksum 0x43cdd06b51e00401
Dec 21 14:42:12.9112 ereport.fs.zfs.checksum 0x43d146c7b3800401
Dec 21 14:42:12.9112 ereport.fs.zfs.checksum 0x43d146c7b3800401
Dec 21 14:49:18.4714 ereport.io.scsi.cmd.disk.dev.rqs.derr 0x4a02ad6a0f500401
Dec 21 14:49:22.7455 ereport.io.scsi.cmd.disk.dev.rqs.derr 0x4a1299a8da200401
Dec 21 14:49:23.4798 ereport.io.scsi.cmd.disk.dev.rqs.derr 0x4a1555f048900401
Dec 21 14:49:25.2605 ereport.io.scsi.cmd.disk.dev.rqs.derr 0x4a1bf84174c00401
Dec 21 14:49:26.8081 ereport.io.scsi.cmd.disk.dev.rqs.derr 0x4a21bc41f8e00401
Dec 21 15:06:37.8081 ereport.fs.zfs.checksum 0x5922a89687100001
Dec 21 15:06:37.8081 ereport.fs.zfs.checksum 0x5922a89687100001
Dec 21 16:09:16.0315 ereport.io.scsi.cmd.disk.dev.rqs.derr 0x8fd3964b04c00001
Dec 21 16:09:19.6828 ereport.io.scsi.cmd.disk.dev.rqs.derr 0x8fe1309461b00001
Dec 21 16:09:20.8254 ereport.io.scsi.cmd.disk.dev.rqs.derr 0x8fe5724086c00401
Dec 21 16:09:21.9860 ereport.io.scsi.cmd.disk.dev.rqs.derr 0x8fe9c5325a200401
Dec 21 16:09:23.1430 ereport.io.scsi.cmd.disk.dev.rqs.derr 0x8fee14a0d3c00001
Dec 21 16:09:41.5650 ereport.fs.zfs.checksum 0x9032b5ead3100401
Dec 21 16:09:41.5650 ereport.fs.zfs.checksum 0x9032b5ead3100401
Dec 21 16:11:33.0525 ereport.fs.zfs.checksum 0x91d20c40d8100001
Dec 21 16:11:33.0525 ereport.fs.zfs.checksum 0x91d20c40d8100001
Dec 21 16:11:33.1082 ereport.fs.zfs.checksum 0x91d24153a8200001
Dec 21 16:11:33.1082 ereport.fs.zfs.checksum 0x91d24153a8200001
Dec 21 16:11:36.6849 ereport.fs.zfs.checksum 0x91df9469f2900001
Dec 21 16:11:36.6849 ereport.fs.zfs.checksum 0x91df9469f2900001
Dec 21 16:11:37.1319 ereport.fs.zfs.checksum 0x91e13ebe1cf00401
Dec 21 16:11:37.1319 ereport.fs.zfs.checksum 0x91e13ebe1cf00401
Dec 21 16:11:41.7997 ereport.fs.zfs.checksum 0x91f2a28088800001
Dec 21 16:11:41.7997 ereport.fs.zfs.checksum 0x91f2a28088800001
Dec 21 16:20:54.8897 ereport.fs.zfs.checksum 0x99ff209881d00401
Dec 21 16:20:54.8897 ereport.fs.zfs.checksum 0x99ff209881d00401
Dec 21 16:20:55.8360 ereport.fs.zfs.checksum 0x9a02a72227d00001
Dec 21 16:20:55.8360 ereport.fs.zfs.checksum 0x9a02a72227d00001
Dec 21 16:40:10.5784 ereport.fs.zfs.checksum 0xaad08db88ff00401
Dec 21 16:40:10.5784 ereport.fs.zfs.checksum 0xaad08db88ff00401
Dec 21 17:39:51.2304 ereport.fs.zfs.checksum 0xdeec03d766600001
Dec 21 17:39:51.2304 ereport.fs.zfs.checksum 0xdeec03d766600001
Dec 21 17:39:59.9474 ereport.io.scsi.cmd.disk.dev.rqs.derr 0xdf0c7d41fac00401
Dec 21 17:40:04.5210 ereport.io.scsi.cmd.disk.dev.rqs.derr 0xdf1d871db4600001
Dec 21 17:40:05.6893 ereport.io.scsi.cmd.disk.dev.rqs.derr 0xdf21e14ba3a00001
Dec 21 17:40:06.8237 ereport.io.scsi.cmd.disk.dev.rqs.derr 0xdf261b31d9100001
Dec 21 17:40:07.9910 ereport.io.scsi.cmd.disk.dev.rqs.derr 0xdf2a747e62f00001
Dec 21 17:43:37.1941 ereport.fs.zfs.checksum 0xe235d2cf5dc00001
Dec 21 17:43:37.1941 ereport.fs.zfs.checksum 0xe235d2cf5dc00001
Dec 21 17:43:37.2973 ereport.fs.zfs.checksum 0xe2363535aa500401
Dec 21 17:43:37.2973 ereport.fs.zfs.checksum 0xe2363535aa500401
Dec 21 17:43:40.8574 ereport.fs.zfs.checksum 0xe243788409000401
Dec 21 17:43:40.8574 ereport.fs.zfs.checksum 0xe243788409000401
Dec 21 17:43:41.3875 ereport.fs.zfs.checksum 0xe245721750500401
Dec 21 17:43:41.3875 ereport.fs.zfs.checksum 0xe245721750500401
Dec 21 17:43:45.3929 ereport.fs.zfs.checksum 0xe2545e0f14e00401
Dec 21 17:43:45.3929 ereport.fs.zfs.checksum 0xe2545e0f14e00401
Dec 21 17:53:32.0557 ereport.fs.zfs.checksum 0xeaddee9f10c00401
Dec 21 17:53:32.0557 ereport.fs.zfs.checksum 0xeaddee9f10c00401
Dec 21 17:53:32.8619 ereport.fs.zfs.checksum 0xeae0ef883f600001
Dec 21 17:53:32.8619 ereport.fs.zfs.checksum 0xeae0ef883f600001
Dec 21 18:13:13.7183 ereport.fs.zfs.checksum 0xfc10204d46400401
Dec 21 18:13:13.7183 ereport.fs.zfs.checksum 0xfc10204d46400401
Dec 21 19:01:42.8304 ereport.io.scsi.cmd.disk.dev.rqs.derr 0x2665cd0890800401
Dec 21 19:01:47.0346 ereport.io.scsi.cmd.disk.dev.rqs.derr 0x2675769b61100401
Dec 21 19:01:51.8947 ereport.io.scsi.cmd.disk.dev.rqs.derr 0x268791a84a900001
Dec 21 19:01:54.7808 ereport.io.scsi.cmd.disk.dev.rqs.derr 0x2692523029400401
Dec 21 19:01:56.7315 ereport.io.scsi.cmd.disk.dev.rqs.derr 0x2699968233200001
Dec 21 19:40:54.1171 ereport.fs.zfs.checksum 0x07edff41b8d00401
Dec 21 19:45:45.8638 ereport.fs.zfs.checksum 0x0c2cd69bd5300001
Dec 21 19:45:45.9143 ereport.fs.zfs.checksum 0x0c2d06c29e100401
Dec 21 19:45:50.2857 ereport.fs.zfs.checksum 0x0c3d4faaf2900001
Dec 21 19:45:50.6402 ereport.fs.zfs.checksum 0x0c3ea1c8bb400001
Dec 21 19:52:20.5475 ereport.fs.zfs.checksum 0x11eb264cbfa00401
Dec 21 19:52:21.4633 ereport.fs.zfs.checksum 0x11ee8fb5ce300401
Dec 21 20:16:23.0050 ereport.fs.zfs.checksum 0x26e90e5c32f00001
Dec 22 00:00:08.7555 ereport.fs.zfs.checksum 0xea49a29745000401
Dec 22 00:04:17.6368 ereport.fs.zfs.checksum 0xede8d33a61100001
Dec 22 00:34:30.6962 ereport.fs.zfs.checksum 0x084b2b20b9000001
Dec 22 11:15:50.1069 ereport.fs.zfs.checksum 0x38431f4a12900401
//Jim
-------------------------------------------
illumos-zfs
Archives: https://www.listbox.com/member/archive/182191/=now
RSS Feed: https://www.listbox.com/member/archive/rss/182191/21996034-de72a2b0
Modify Your Subscription: https://www.listbox.com/member/?&
Powered by Listbox: http://www.listbox.com
illumos-zfs | Archives | Modify Your Subscription
illumos-zfs | Archives | Modify Your Subscription

[Jim Klimov]

Post by Jim Bolmgren-jim99
Hi Jim,
Are you sure you mirrored the slice and not the disk?
— richard

Yes, of course. Given that rpools can only live on slices...

//Jim