OT VirtualBox ZFS Question

[Paul Kraus]

Apologies for the slightly off topic question here.

Does anyone know if VirtualBox disk write commits are sync or async or does the driver just pass through what the app asks for ? I have found documentation that implies that if an app running in a VB VM makes a sync call it will be honored, but if the app makes an async write call will it come out the other end async or sync ?

The reason I am asking is because I have a number of VB systems with local ZFS storage and debating whether adding SSD log devices (mirrored, of course) would help handle write load. We are upgrading drives and I have an opportunity to add SSDs.

--
Paul Kraus
Deputy Technical Director, LoneStarCon 3
Sound Coordinator, Schenectady Light Opera Company

[Joe Nyilas - RAS SSE]

Post by Paul Kraus
Apologies for the slightly off topic question here.
Does anyone know if VirtualBox disk write commits are sync or async or does the driver just pass through what the app asks for ? I have found documentation that implies that if an app running in a VB VM makes a sync call it will be honored, but if the app makes an async write call will it come out the other end async or sync ?
The reason I am asking is because I have a number of VB systems with local ZFS storage and debating whether adding SSD log devices (mirrored, of course) would help handle write load. We are upgrading drives and I have an opportunity to add SSDs.
--
Paul Kraus
Deputy Technical Director, LoneStarCon 3
Sound Coordinator, Schenectady Light Opera Company
-------------------------------------------
illumos-zfs
Archives: https://www.listbox.com/member/archive/182191/=now
RSS Feed: https://www.listbox.com/member/archive/rss/182191/25087534-8d3acc3a
Modify Your Subscription: https://www.listbox.com/member/?&
Powered by Listbox: http://www.listbox.com

Paul,
Off the top of my head -- no idea. One thing you could do is truss
(solaris) or strace (linux) the virtualbox process during VM activity
and see how open() syscall is handled, like this:
# truss -ftopen -p <VB_PID>
1234: open64("DAFILE", O_WRONLY|O_DSYNC|O_CREAT, 0644) = 3

Files open()d with the O_SYNC, or O_DSYNC flags means subsequent writes
will be synchronous and the log devices may help.

Regards,

[Jim Klimov]

Post by Joe Nyilas - RAS SSE
# truss -ftopen -p <VB_PID>
1234: open64("DAFILE", O_WRONLY|O_DSYNC|O_CREAT, 0644) = 3
Files open()d with the O_SYNC, or O_DSYNC flags means subsequent writes
will be synchronous and the log devices may help.

Isn't it possible for the app (VB) to open the file async, but call
flush to sync it every now and then?

I think it might rather help to put some write load in the VM and
watch the storage with zilstat, or just set zil=disabled or enabled
and see if performance jumps considerably.

//Jim

[Paul Kraus]

Post by Joe Nyilas - RAS SSE
Paul,
# truss -ftopen -p <VB_PID>
1234: open64("DAFILE", O_WRONLY|O_DSYNC|O_CREAT, 0644) = 3
Files open()d with the O_SYNC, or O_DSYNC flags means subsequent writes will be synchronous and the log devices may help.

Hurm… I’m on FreeBSD, so I have DTrace. I should be able to find a DTrace script that can tell me what I need to know.

--
Paul Kraus
Deputy Technical Director, LoneStarCon 3
Sound Coordinator, Schenectady Light Opera Company

[Joe Nyilas - RAS SSE]

Post by Paul Kraus
Hurm… I’m on FreeBSD, so I have DTrace. I should be able to find a DTrace script that can tell me what I need to know.

I suppose so. dtruss? ;-)
http://www.freebsd.org/cgi/man.cgi?query=dtruss&sektion=1
I think the DTrace book has writetype.d, which can easily be modified to
look at O_SYNCs.

If you're on a Solaris variant, you could have a look at fsstat(1M) with
-f switch, it will report kernel file operations, fsync() being a metric
which is tracked.

Also, turning things around. If you're ZFS is running inside a VB VM,
make sure you enable cache flushes to ensure data integrity after an
unexpected event.
http://www.solarisinternals.com/wiki/index.php/ZFS_Best_Practices_Guide#OpenSolaris.2FZFS.2FVirtual_Box_Recommendations

/Joe

[Pawel Jakub Dawidek]

Post by Joe Nyilas - RAS SSE
Paul,
# truss -ftopen -p <VB_PID>
1234: open64("DAFILE", O_WRONLY|O_DSYNC|O_CREAT, 0644) = 3
Files open()d with the O_SYNC, or O_DSYNC flags means subsequent writes will be synchronous and the log devices may help.

Hurm
 I’m on FreeBSD, so I have DTrace. I should be able to find a DTrace script that can tell me what I need to know.

On FreeBSD you can use ktrace(1).

My expectation is that all writes are either sync or async. When I/O
request goes through FreeBSD's GEOM layer and hits (virtual) disk there
is no info if it is sync or async.

[Geoff Nordli]

Post by Paul Kraus
Apologies for the slightly off topic question here.
Does anyone know if VirtualBox disk write commits are sync or async or does the driver just pass through what the app asks for ? I have found documentation that implies that if an app running in a VB VM makes a sync call it will be honored, but if the app makes an async write call will it come out the other end async or sync ?
The reason I am asking is because I have a number of VB systems with local ZFS storage and debating whether adding SSD log devices (mirrored, of course) would help handle write load. We are upgrading drives and I have an opportunity to add SSDs.

Hi Paul.

Make sure you set the IgnoreFlush parameter or vbox itself will cache
requests.

http://www.virtualbox.org/manual/ch12.html#idp57452320

VBoxManage setextradata "VM name"
"VBoxInternal/Devices/piix3ide/0/LUN#[x]/Config/IgnoreFlush" 0

I am not sure how it works when using a file backed disk image. I use
the built-in iscsi initiator and comstar backend. I disable the cache
in comstar, just to make sure writes are not cached anywhere in the stack.

Another note, I don't put a lot of write load on the system, so I don't
need an external slog. I generally have lots of memory so the ARC feeds
the reads and the disks can easily handle write traffic.

How much load are you putting on these?

This is interesting though, I wonder how it works using the file based
backend. To reduce the complication I would rather use zvols + vmdk
wrapper. I just need to figure out how to deal with the user
permissions since vbox isn't running as root and has no access to the
zvol unless we change permissions.

Geoff

[Jim Klimov]

Post by Geoff Nordli
backend. To reduce the complication I would rather use zvols + vmdk
wrapper. I just need to figure out how to deal with the user
permissions since vbox isn't running as root and has no access to the
zvol unless we change permissions.

Shameless plug: If you are on solaris or descendant, take a looj at vboxsvc.sf.net - it includes a service to maintain zvol user permissions across reboots for the vboxes. And vboxsvc itself can run non-root boxes as smf services too ;)

//jim

--
Typos courtesy of K-9 Mail on my Samsung Android

[Geoff Nordli]

Post by Jim Klimov

Post by Geoff Nordli
backend. To reduce the complication I would rather use zvols + vmdk
wrapper. I just need to figure out how to deal with the user
permissions since vbox isn't running as root and has no access to the
zvol unless we change permissions.

Shameless plug: If you are on solaris or descendant, take a looj at vboxsvc.sf.net - it includes a service to maintain zvol user permissions across reboots for the vboxes. And vboxsvc itself can run non-root boxes as smf services too ;)

Hi Jim.

I use the vboxsvc to manage the instances now? When did that feature
get added?

That is fantastic!!

thanks,

Geoff

[Eric Sproul]

There is also the `zilstat` script from
https://github.com/richardelling/tools that will show you ZIL
activity.

[Jim Klimov]

Post by Geoff Nordli

Post by Jim Klimov

Post by Geoff Nordli
backend. To reduce the complication I would rather use zvols + vmdk
wrapper. I just need to figure out how to deal with the user
permissions since vbox isn't running as root and has no access to the
zvol unless we change permissions.

Shameless plug: If you are on solaris or descendant, take a looj at
vboxsvc.sf.net - it includes a service to maintain zvol user
permissions across reboots for the vboxes. And vboxsvc itself can run
non-root boxes as smf services too ;)

Hi Jim.
I use the vboxsvc to manage the instances now? When did that feature
get added?

Quite a bit ago, at least a year or so. Last published release was 0.17,
and there were not many urgent requests for any newer features, so this
version stuck around for a while, so at least since then ;)

According to my logs, the zfs-zvolrights script was added in the 0.17
version, and since 0.15 there were (and evolved) capabilities for ZFS
snapshots of (local or remote) POSIX and (local) ZVOL datasets used to
back the VM's components before start/after stop/on request.

This particular script can save the access rights (POSIX and ACL) and
ownership information that you've set for a particular zvol, and reapply
these after reboot (illumos itself does a poor job if any at this, I am
not sure if this was addressed in the past year). The information is
saved as ZFS user attributes, so this should be migratable to other
boxes if the storage fails over.

HTH,
//Jim

[Geoff Nordli]

Post by Jim Klimov

Post by Geoff Nordli

Post by Jim Klimov

Post by Geoff Nordli
backend. To reduce the complication I would rather use zvols + vmdk
wrapper. I just need to figure out how to deal with the user
permissions since vbox isn't running as root and has no access to the
zvol unless we change permissions.

Shameless plug: If you are on solaris or descendant, take a looj at
vboxsvc.sf.net - it includes a service to maintain zvol user
permissions across reboots for the vboxes. And vboxsvc itself can run
non-root boxes as smf services too ;)

Hi Jim.
I use the vboxsvc to manage the instances now? When did that feature
get added?

Quite a bit ago, at least a year or so. Last published release was 0.17,
and there were not many urgent requests for any newer features, so this
version stuck around for a while, so at least since then ;)
According to my logs, the zfs-zvolrights script was added in the 0.17
version, and since 0.15 there were (and evolved) capabilities for ZFS
snapshots of (local or remote) POSIX and (local) ZVOL datasets used to
back the VM's components before start/after stop/on request.
This particular script can save the access rights (POSIX and ACL) and
ownership information that you've set for a particular zvol, and reapply
these after reboot (illumos itself does a poor job if any at this, I am
not sure if this was addressed in the past year). The information is
saved as ZFS user attributes, so this should be migratable to other
boxes if the storage fails over.

Thanks for the update Jim.

I guess I need to pay more attention to the release notes :)

Geoff

[Paul Kraus]

Make sure you set the IgnoreFlush parameter or vbox itself will cache requests.
http://www.virtualbox.org/manual/ch12.html#idp57452320
VBoxManage setextradata "VM name" "VBoxInternal/Devices/piix3ide/0/LUN#[x]/Config/IgnoreFlush” 0

Thanks for the pointer.

I have also been pouring over 5.7 Host I/O caching ( https://www.virtualbox.org/manual/ch05.html ), and from my read of that section it sounds like if you DISABLE the Host I/O cache then you are instructing VB to use sync writes. When I created these VMs I took the defaults, which for IDE controllers is to Use the Host I/O Cache (async writes) and for SATA controllers Disable Host I/O Cache (which would be sync writes).

I’ll have to test (with a non-production VM) and see what the difference is.

How much load are you putting on these?

Not that much, but I had a zpool get to under 10% avail and I was clearly thrashing the I/O subsystem (disk busy of over 50% constantly and service times between 20 and 100 ms), once I got the avail back up over 20% the problem went away (destroyed lots of snapshots). Disk busy usually runs 1 to 2% with occasional peaks of 10% (10 second averages). I was NOT CPU bound, but the zfs code was asking for i/o much faster than the disks could respond. The system has 24GB of RAM and the ARC has between 6 and 8 GB to play with when all the VMs are up and running.

The critical VM is a Windows 2008R2 RDP server. There is also a Mail Archive VM (Mdir format).

--
Paul Kraus
Deputy Technical Director, LoneStarCon 3
Sound Coordinator, Schenectady Light Opera Company

[Matthew Ahrens]

Post by Paul Kraus
Apologies for the slightly off topic question here.
Does anyone know if VirtualBox disk write commits are sync or async or
does the driver just pass through what the app asks for ? I have found
documentation that implies that if an app running in a VB VM makes a sync
call it will be honored, but if the app makes an async write call will it
come out the other end async or sync ?
The reason I am asking is because I have a number of VB systems with local
ZFS storage and debating whether adding SSD log devices (mirrored, of
course) would help handle write load. We are upgrading drives and I have an
opportunity to add SSDs.

You could always try setting sync=disabled and see if that improves
performance. (Read the zfs manpage for caveats on this.) If so, adding a
SSD log device should also improve performance (though to a lesser degree).

--matt

Post by Paul Kraus
--
Paul Kraus
Deputy Technical Director, LoneStarCon 3
Sound Coordinator, Schenectady Light Opera Company
-------------------------------------------
illumos-zfs
Archives: https://www.listbox.com/member/archive/182191/=now
https://www.listbox.com/member/archive/rss/182191/21635000-ebd1d460
https://www.listbox.com/member/?&
Powered by Listbox: http://www.listbox.com

-------------------------------------------
illumos-zfs
Archives: https://www.listbox.com/member/archive/182191/=now
RSS Feed: https://www.listbox.com/member/archive/rss/182191/23047029-187a0c8d
Modify Your Subscription: https://www.listbox.com/member/?member_id=23047029&id_secret=23047029-2e85923f
Powered by Listbox: http://www.listbox.com