determining if a file system has changed since a given snapshot

Discussion:

Paul B. Henson via illumos-zfs

2014-07-15 21:21:49 UTC

I'm trying to figure out what the best way is to determine if a given file
system has changed at all since a given snapshot of it. At this point, the
only reliable method we have found is to run a zfs diff between the snapshot
and the file system and see if it generates any output. While that certainly
works, it seems a bit inefficient, as zfs diff will enumerate all of the
actual changes, whereas all I actually care about is whether or not any
change occurred. Is there any better way to do this? If not, what about an
option to zfs diff that causes it to only return whether or not there are
changes, but not the actual changes? Whether that was via some type of
output or a status return code, it could then stop processing as soon as it
sees the first change, rather than continuing on to enumerate all of them.

Thanks.

Matthew Ahrens via illumos-zfs

2014-07-15 21:37:40 UTC

Permalink

I don't think there's any foolproof way to determine this using the
existing administrative interfaces.

If the "referenced" property of the snapshot and the filesystem is
different, then there have definitely been changes. But the inverse is not
necessarily true -- if the referenced is the same, there might or might not
have been changes.

Similarly, if the "used" property of the snapshot is nonzero, then there
have been changes. But used==0 doesn't necessarily mean that there have
not been changes.

--matt

On Tue, Jul 15, 2014 at 2:21 PM, Paul B. Henson via illumos-zfs <

Post by Paul B. Henson via illumos-zfs
I'm trying to figure out what the best way is to determine if a given file
system has changed at all since a given snapshot of it. At this point, the
only reliable method we have found is to run a zfs diff between the snapshot
and the file system and see if it generates any output. While that certainly
works, it seems a bit inefficient, as zfs diff will enumerate all of the
actual changes, whereas all I actually care about is whether or not any
change occurred. Is there any better way to do this? If not, what about an
option to zfs diff that causes it to only return whether or not there are
changes, but not the actual changes? Whether that was via some type of
output or a status return code, it could then stop processing as soon as it
sees the first change, rather than continuing on to enumerate all of them.
Thanks.
-------------------------------------------
illumos-zfs
Archives: https://www.listbox.com/member/archive/182191/=now
https://www.listbox.com/member/archive/rss/182191/21635000-ebd1d460
https://www.listbox.com/member/?&
Powered by Listbox: http://www.listbox.com

-------------------------------------------
illumos-zfs
Archives: https://www.listbox.com/member/archive/182191/=now
RSS Feed: https://www.listbox.com/member/archive/rss/182191/23047029-187a0c8d
Modify Your Subscription: https://www.listbox.com/member/?member_id=23047029&id_secret=23047029-2e85923f
Powered by Listbox: http://www.listbox.com

Timothy Coalson via illumos-zfs

2014-07-15 22:14:10 UTC

Permalink

To throw another property in there, the written@<snapshot> property of the
filesystem gives you the total size of any changed/new data on the
filesystem since the snapshot. This should leave only needing to see if
there has only been data removed, which could be done by comparing the
referenced properties (use -p to prevent rounding to human-friendly units).

Of course, since metadata is copy on write like everything else, this seems
to imply that changing anything will cause a new block of some kind to be
written, which would suggest that written@<snapshot> will never be zero if
any filesystem changes exist, is this guaranteed? Some quick testing says
chmod and rm both result in written@<snapshot> (among other properties)
becoming nonzero. It seems kind of dirty to rely on this, but...

Tim

On Tue, Jul 15, 2014 at 4:37 PM, Matthew Ahrens via illumos-zfs <

Post by Matthew Ahrens via illumos-zfs
I don't think there's any foolproof way to determine this using the
existing administrative interfaces.
If the "referenced" property of the snapshot and the filesystem is
different, then there have definitely been changes. But the inverse is not
necessarily true -- if the referenced is the same, there might or might not
have been changes.
Similarly, if the "used" property of the snapshot is nonzero, then there
have been changes. But used==0 doesn't necessarily mean that there have
not been changes.
--matt
On Tue, Jul 15, 2014 at 2:21 PM, Paul B. Henson via illumos-zfs <

Post by Paul B. Henson via illumos-zfs
I'm trying to figure out what the best way is to determine if a given file
system has changed at all since a given snapshot of it. At this point, the
only reliable method we have found is to run a zfs diff between the snapshot
and the file system and see if it generates any output. While that certainly
works, it seems a bit inefficient, as zfs diff will enumerate all of the
actual changes, whereas all I actually care about is whether or not any
change occurred. Is there any better way to do this? If not, what about an
option to zfs diff that causes it to only return whether or not there are
changes, but not the actual changes? Whether that was via some type of
output or a status return code, it could then stop processing as soon as it
sees the first change, rather than continuing on to enumerate all of them.
Thanks.
-------------------------------------------
illumos-zfs
Archives: https://www.listbox.com/member/archive/182191/=now
https://www.listbox.com/member/archive/rss/182191/21635000-ebd1d460
Modify Your Subscription: https://www.listbox.com/member/?&
Powered by Listbox: http://www.listbox.com

*illumos-zfs* | Archives
<https://www.listbox.com/member/archive/182191/=now>
<https://www.listbox.com/member/archive/rss/182191/24077749-f7a4e87a> |
Modify
<https://www.listbox.com/member/?&>
Your Subscription <http://www.listbox.com>

Paul B. Henson via illumos-zfs

2014-07-15 22:37:34 UTC

Permalink

Post by Timothy Coalson via illumos-zfs
filesystem gives you the total size of any changed/new data on the
filesystem since the snapshot. This should leave only needing to see if
there has only been data removed, which could be done by comparing the
referenced properties (use -p to prevent rounding to human-friendly units).

Fails the touch test:

# zfs get -p written,referenced export/user/***@zfsrepl-2014.07.15-15.34.00Z
NAME PROPERTY VALUE
SOURCE
export/user/***@zfsrepl-2014.07.15-15.34.00Z written 0 -
export/user/***@zfsrepl-2014.07.15-15.34.00Z referenced 31744 -

# touch /export/user/test997/testfile

# zfs get -p written,referenced export/user/***@zfsrepl-2014.07.15-15.34.00Z
NAME PROPERTY VALUE
SOURCE
export/user/***@zfsrepl-2014.07.15-15.34.00Z written 0 -
export/user/***@zfsrepl-2014.07.15-15.34.00Z referenced 31744 -

As well as actually adding data:

# echo foo > /export/user/test997/testfile
# zfs get -p written,referenced export/user/***@zfsrepl-2014.07.15-15.34.00Z
NAME PROPERTY VALUE
SOURCE
export/user/***@zfsrepl-2014.07.15-15.34.00Z written 0 -
export/user/***@zfsrepl-2014.07.15-15.34.00Z referenced 31744 -

Unless I'm misunderstanding your suggestion?

Thanks...

Matthew Ahrens via illumos-zfs

2014-07-15 22:39:25 UTC

Permalink

Post by Timothy Coalson via illumos-zfs
the

units).
# zfs get -p written,referenced
NAME PROPERTY VALUE
SOURCE
# touch /export/user/test997/testfile

You need to run "sync" here.

$ man zfs
...
The amount of space used, available, or referenced [or
"written", which is defined in terms of "referenced"] does
not take into account pending changes. Pending changes
are generally accounted for within a few seconds. Com-
mitting a change to a disk using fsync(3c) or O_SYNC
does not necessarily guarantee that the space usage
information is updated immediately.

Post by Timothy Coalson via illumos-zfs
# zfs get -p written,referenced
NAME PROPERTY VALUE
SOURCE
# echo foo > /export/user/test997/testfile

And here.

Post by Timothy Coalson via illumos-zfs
# zfs get -p written,referenced
NAME PROPERTY VALUE
SOURCE
Unless I'm misunderstanding your suggestion?
Thanks...

Paul B. Henson via illumos-zfs

2014-07-15 23:47:04 UTC

Permalink

From: Matthew Ahrens
Sent: Tuesday, July 15, 2014 3:39 PM
You need to run "sync" here.

D'oh, thanks for the clarification :). This looks like it will do the trick, thanks much everybody…

Matthew Ahrens via illumos-zfs

2014-07-15 22:37:41 UTC

Permalink

the filesystem gives you the total size of any changed/new data on the
filesystem since the snapshot. This should leave only needing to see if
there has only been data removed, which could be done by comparing the
referenced properties (use -p to prevent rounding to human-friendly units).
Of course, since metadata is copy on write like everything else, this
seems to imply that changing anything will cause a new block of some kind
zero if any filesystem changes exist, is this guaranteed? Some quick
properties) becoming nonzero. It seems kind of dirty to rely on this,
but...

Ah, of course, how could I forget about "written"! That will work
perfectly -- it is guaranteed that if anything is changed (even a
deletion), some data will be written. And note that the simple "written"
property (without @<latest snapshot>) will do what you want.

$ man zfs
...
written
The amount of referenced space written to this dataset
since the previous snapshot.
...
This is the space that is
referenced by this dataset but was not referenced by the
[previous] snapshot.

--matt

Tim
On Tue, Jul 15, 2014 at 4:37 PM, Matthew Ahrens via illumos-zfs <

Post by Paul B. Henson via illumos-zfs
I'm trying to figure out what the best way is to determine if a given file
system has changed at all since a given snapshot of it. At this point, the
only reliable method we have found is to run a zfs diff between the snapshot
and the file system and see if it generates any output. While that certainly
works, it seems a bit inefficient, as zfs diff will enumerate all of the
actual changes, whereas all I actually care about is whether or not any
change occurred. Is there any better way to do this? If not, what about an
option to zfs diff that causes it to only return whether or not there are
changes, but not the actual changes? Whether that was via some type of
output or a status return code, it could then stop processing as soon as it
sees the first change, rather than continuing on to enumerate all of them.
Thanks.
-------------------------------------------
illumos-zfs
Archives: https://www.listbox.com/member/archive/182191/=now
https://www.listbox.com/member/archive/rss/182191/21635000-ebd1d460
Modify Your Subscription: https://www.listbox.com/member/?&
Powered by Listbox: http://www.listbox.com