Discussion:
Data loss in ZFS filesystem: directory disappears after extattr issues
javocado
2014-05-02 22:28:04 UTC
Permalink
OS: freebsd 8.3-RELEASE

I am seeing an issue where a directory in a ZFS filesystem has just
disappeared.

When I do an 'ls' in that filesystem, I get:

# ls
ls: backup: No such file or directory
regular_file1
regular_file2
regular_file3
#


... so the OS knows it is there, but then it can't show it. Note that I was
running just a plain old 'ls' command, so if it really wasn't there, it
would just show nothing.

Also, I can autocomplete the "backup" directory from the shell ... so
somehow it sees it there, but won't do anything with it.

This behavior also appears in the snapshots I am making. Unfortunately I
did not notice this before several older snapshots had been destroyed. So
I'll never know whether the snapshots (rotated daily) taken before this
directory became inaccessible were also affected.

I have tried the following, but nothing has fixed the issue:

- rebooting
- scrub'ing the pool
- zfs send | receive to a new filesystem

The only interesting thing about this filesystem, and this directory, is
that it
contains a lot of files with extended attributes.

Further, these extattrs were crashing the system regularly until we applied
a patch from Pawel:

http://pastebin.com/0mu50B7T

... now the crashing has stopped, but that directory has ... disappeared.
This directory that is now gone is absolutely the directory whose contents
were causing the crashes.

We have a lot of questions about this, but maybe we can start with: What
in the world is going on here ?

Thanks.



-------------------------------------------
illumos-zfs
Archives: https://www.listbox.com/member/archive/182191/=now
RSS Feed: https://www.listbox.com/member/archive/rss/182191/23047029-187a0c8d
Modify Your Subscription: https://www.listbox.com/member/?member_id=23047029&id_secret=23047029-2e85923f
Powered by Listbox: http://www.listbox.com
Chris Ridd via illumos-zfs
2014-05-03 09:07:49 UTC
Permalink
Post by javocado
OS: freebsd 8.3-RELEASE
[...]
Post by javocado
The only interesting thing about this filesystem, and this directory, is that it
contains a lot of files with extended attributes.
http://pastebin.com/0mu50B7T
... now the crashing has stopped, but that directory has ... disappeared. This directory that is now gone is absolutely the directory whose contents were causing the crashes.
We have a lot of questions about this, but maybe we can start with: What in the world is going on here ?
The code looks like it is calling zfs_zget to get the xattrs on some object and failing, and the change now prints/logs the failure instead of asserting (crashing). What's it printing/logging?

Can you 'cd' into the disappeared directory?

Chris
javocado
2014-05-05 04:29:01 UTC
Permalink
The log output looks like:

Apr 19 12:28:16 xxxx kernel: zfs_remove: zfs_zget(pool/zfs, 59497)
returned 2

We don't think it is the patch that caused the problem, it is just
background details for our problem.

We are not opposed to the idea that the patch caused the problem, but our
current assumption is that crashing a bunch of times with the extattr
problem is what caused this directory to get messed up ...



On Sat, May 3, 2014 at 2:07 AM, Chris Ridd via illumos-zfs <
Post by javocado
Post by javocado
OS: freebsd 8.3-RELEASE
[...]
Post by javocado
The only interesting thing about this filesystem, and this directory, is
that it
Post by javocado
contains a lot of files with extended attributes.
Further, these extattrs were crashing the system regularly until we
http://pastebin.com/0mu50B7T
... now the crashing has stopped, but that directory has ...
disappeared. This directory that is now gone is absolutely the directory
whose contents were causing the crashes.
What in the world is going on here ?
The code looks like it is calling zfs_zget to get the xattrs on some
object and failing, and the change now prints/logs the failure instead of
asserting (crashing). What's it printing/logging?
Can you 'cd' into the disappeared directory?
Chris
-------------------------------------------
illumos-zfs
Archives: https://www.listbox.com/member/archive/182191/=now
https://www.listbox.com/member/archive/rss/182191/26026854-f5dfad01
https://www.listbox.com/member/?&
Powered by Listbox: http://www.listbox.com
-------------------------------------------
illumos-zfs
Archives: https://www.listbox.com/member/archive/182191/=now
RSS Feed: https://www.listbox.com/member/archive/rss/182191/23047029-187a0c8d
Modify Your Subscription: https://www.listbox.com/member/?member_id=23047029&id_secret=23047029-2e85923f
Powered by Listbox: http://www.listbox.com

Loading...