Migrating from Solaris to Illumos ZFS

I have not tried this, but AFAIK, you'd only be in trouble if you were
running ZFS v6 from Solaris 11.1 - the zpool versions have changed,
but the ZFS version (and thus, I'd expect, the zfs send version)
didn't change until then?

I'd be curious to know either way. Importing the pool is just not
going to happen, unless someone spends a lot of time writing it, but
zfs send |recv might work.

- Rich

Post by Ian Collins
Hello,
I've seen this topic pop up once in a while and now I have a requirement to
move a lot of data from Solaris 11 to Illumos. I don't really want to have
to resort to rsync (I would like to retain snapshots) so what I'd like to
know is has anyone been looking at adapting zfs receive to downgrade
versions? In my case, the pools are at version 31, but don't use any
Solaris specific features such as encryption.
Is this even feasible? If it is, I probably have the time to contribute to
making changes if someone would point me at the appropriate bits of the
code.
Thanks,
--
Ian.
-------------------------------------------
illumos-zfs
Archives: https://www.listbox.com/member/archive/182191/=now
https://www.listbox.com/member/archive/rss/182191/24536931-5d25148d
https://www.listbox.com/member/?&
Powered by Listbox: http://www.listbox.com

Richard Elling

2014-03-21 23:25:56 UTC

If the pool version > 28, you won't be able to import the pool.

If the zfs version is 5 and of a sufficiently old vintage that a direct send/receive
can work, then you can send/receive.

If the zfs version is 5 and patched, then there is a potential workaround to attempt
send/receive to get past some error checking and is not guaranteed to work. Contact me
offlist for info.

If the zfs version is > 5, then rsync.

It is possible to sync from the oldest snapshot first, thus keeping the basic structure of
your snapshots intact.
rsync snapshot -> new fs
snapshot new fs
rsync next snapshot -> new fs
snapshot new fs
...

-- richard

Ian Collins

2014-03-21 23:34:48 UTC

Yes please, some of the data is on a Sun (that old!) storage appliance.

Post by Richard Elling
If the zfs version is > 5, then rsync.

What happened between 5 and 6?

Post by Richard Elling
It is possible to sync from the oldest snapshot first, thus keeping the basic structure of
your snapshots intact.
rsync snapshot -> new fs
snapshot new fs
rsync next snapshot -> new fs
snapshot new fs
...

I know, that's what I've been trying to avoid with 30TB of data and
several thousand snapshots! I think the Sun would go cold before I
finished...

--
Ian.

Richard Elling

2014-03-21 23:49:32 UTC

Yes please, some of the data is on a Sun (that old!) storage appliance.

Post by Richard Elling
If the zfs version is > 5, then rsync.

What happened between 5 and 6?

VER DESCRIPTION
--- --------------------------------------------------------
1 Initial ZFS filesystem version
2 Enhanced directory entries
3 Case insensitive and SMB credentials support
4 userquota, groupquota properties
5 System attributes
6 Multilevel file system support

Multilevel file system support is a Trusted Solaris feature.

I know, that's what I've been trying to avoid with 30TB of data and several thousand snapshots! I think the Sun would go cold before I finished...

A long time^TM ;-)
Fortunately you have snapshots. Think how painful this is without snapshots...
In the past, I've seen this sort of thing go at 50-100MB/sec per disk, depending
on how big the files are and what the churn was.
-- richard

Richard Laager

2014-03-21 23:49:59 UTC

I know, that's what I've been trying to avoid with 30TB of data and
several thousand snapshots! I think the Sun would go cold before I
finished...

Try combining zfs diff + rsync --files-from. I've never done it, but it
looks like it shouldn't be too hard. This would save massive amounts of
rsync time.

Remember to use --inplace when rsync'ing to ZFS (assuming you don't need
the atomic file replace, which you don't here).

--

Ian Collins

2014-03-22 00:28:34 UTC

I know, that's what I've been trying to avoid with 30TB of data and
several thousand snapshots! I think the Sun would go cold before I
finished...

Try combining zfs diff + rsync --files-from. I've never done it, but it
looks like it shouldn't be too hard. This would save massive amounts of
rsync time.

Yes, I would look into that (I also use zfs diff to feed files to
clamscan), although zfs diff on one 6TB file system takes about 100
minutes to run.

Post by Richard Laager
Remember to use --inplace when rsync'ing to ZFS (assuming you don't need
the atomic file replace, which you don't here).

I will, thanks for reminding me.

--
Ian.

Jim Klimov

2014-03-22 07:40:51 UTC

Post by Richard Elling
It is possible to sync from the oldest snapshot first, thus keeping

the basic structure of

Post by Richard Elling
your snapshots intact.
rsync snapshot -> new fs
snapshot new fs
rsync next snapshot -> new fs
snapshot new fs
...

I know, that's what I've been trying to avoid with 30TB of data and
several thousand snapshots! I think the Sun would go cold before I
finished...

Try combining zfs diff + rsync --files-from. I've never done it, but

Post by Richard Laager
looks like it shouldn't be too hard. This would save massive amounts

Post by Richard Laager
rsync time.

Yes, I would look into that (I also use zfs diff to feed files to
clamscan), although zfs diff on one 6TB file system takes about 100
minutes to run.

Post by Richard Laager
Remember to use --inplace when rsync'ing to ZFS (assuming you don't

need

Post by Richard Laager
the atomic file replace, which you don't here).

I will, thanks for reminding me.

Also don't forget that if you use clones, then a check on "origin" would help you maintain the clone hierarchy on target.

Also you might want to skip snapshots that 'used' little or esp. zero data, so that you don't spend ages rsync-comparing the same data with few/no changes.
--
Typos courtesy of K-9 Mail on my Samsung Android

Aneurin Price

2014-03-24 11:21:34 UTC

I know, that's what I've been trying to avoid with 30TB of data and
several thousand snapshots! I think the Sun would go cold before I
finished...

Try combining zfs diff + rsync --files-from. I've never done it, but it
looks like it shouldn't be too hard. This would save massive amounts of
rsync time.

Test this assumption first. In my experience rsync is at least an
order of magnitude faster at generating the change list, at least with
files in the tens of thousands. If you have millions or more, it's
possible that zfs diff would scale better, but I'd be surprised.

Post by Richard Laager
Remember to use --inplace when rsync'ing to ZFS (assuming you don't need
the atomic file replace, which you don't here).

Also, if you're doing this on one system (I guess not, but just in
case), you'll want to use --no-whole-file, otherwise your --inplace
won't help you (--whole-file is the default for local transfers).

Richard Elling

2014-03-25 04:34:11 UTC

Post by Aneurin Price

I know, that's what I've been trying to avoid with 30TB of data and
several thousand snapshots! I think the Sun would go cold before I
finished...

Try combining zfs diff + rsync --files-from. I've never done it, but it
looks like it shouldn't be too hard. This would save massive amounts of
rsync time.

ZFS kicks butt over rsync for millions of files. At the billions range, it is
not even a fair fight :-) For this case, however, the zfs stream format might
not be compatible from Oracle’s closed-source ZFS to the wonderful OpenZFS :-(

Post by Aneurin Price

Post by Richard Laager
Remember to use --inplace when rsync'ing to ZFS (assuming you don't need
the atomic file replace, which you don't here).

good idea
— richard

--
ZFS and performance consulting
http://www.RichardElling.com

Ian Collins via illumos-zfs

2014-06-27 09:00:55 UTC

Post by Aneurin Price

I know, that's what I've been trying to avoid with 30TB of data and
several thousand snapshots! I think the Sun would go cold before I
finished...

Try combining zfs diff + rsync --files-from. I've never done it, but it
looks like it shouldn't be too hard. This would save massive amounts of
rsync time.

Well after no end of delays, this migration is underway. For the
biggest filesystem (>20million files) rsync was taking up to 7 hours a
pass...

Running a zfs diff is significantly faster (60-80 minutes). I've ended
up writing an application to perform the updates using the zfs diff
output which sends the updates just about as fast as the hardware will
allow.

--
Ian.

Schweiss, Chip

2014-03-22 16:12:41 UTC

Post by Richard Elling
It is possible to sync from the oldest snapshot first, thus keeping the

Post by Richard Elling
basic structure of
your snapshots intact.
rsync snapshot -> new fs
snapshot new fs
rsync next snapshot -> new fs
snapshot new fs
...

I know, that's what I've been trying to avoid with 30TB of data and
several thousand snapshots! I think the Sun would go cold before I
finished...

I've done this in the past for an even larger data set (~50TB) and finished
in a less than 1 week. You will be surprised how fast rsync passes will
go as long as you have enough ARC and L2ARC to hold all of the metadata.
With 90+ million files I have seen rsync complete in mere seconds when all
the metadata is cached.

-Chip

-------------------------------------------
illumos-zfs
Archives: https://www.listbox.com/member/archive/182191/=now
RSS Feed: https://www.listbox.com/member/archive/rss/182191/23047029-187a0c8d
Modify Your Subscription: https://www.listbox.com/member/?member_id=23047029&id_secret=23047029-2e85923f
Powered by Listbox: http://www.listbox.com

Darren Reed

2014-03-22 21:22:44 UTC

Presumably this is all just to do with metadata being changed or
restructured on the disk?

My recollection is that a "zfs send" is an ordered "dump" (for lack of a
better word) of the filesystem contents, as if you were dumping raw
inodes and all from ufs.

So if send/receive can work then shouldn't it be technically feasible to
rewrite the metadata on disk to do the equivalent of send/recv but inplace?

Darren

Toomas Soome

2014-03-22 11:05:32 UTC

Presumably this is all just to do with metadata being changed or restructured on the disk?
My recollection is that a "zfs send" is an ordered "dump" (for lack of a better word) of the filesystem contents, as if you were dumping raw inodes and all from ufs.
So if send/receive can work then shouldn't it be technically feasible to rewrite the metadata on disk to do the equivalent of send/recv but inplace?
Darren

in case of incompatible zfs where you cant use zfs send, you still can use rsync or other FS specific tools to get your data over, even the snapshots - as you can iterate over snapshots on source from oldest to more recent; make copy of data in snapshot, use zfs snapshot at target, then copy next one, create snap and so on, and as last task, you copy the current dataset and you are done. easier than hacking with internals and way less error prone. it wont be identical copy by block allocation, but the data is.

toomas

Richard Elling

2014-03-22 15:51:29 UTC