RFC: Exposing a storage API

[Karl Wagner]

Hi All

I have a suggestion for a future development of ZFS: as the subject,
exposing a "Storage API" for ZFS.

The idea is fairly straight forward. We expose the functionality to store
objects in a ZPool. Currently (AFAIK) the only ways to do this are via a
ZFS filesystem tree or a ZVol. Neither are optimal in all situations, so
offering access to a layer below this could encourage developers to use ZFS
in a targetted way to enhance their projects.

The main things I am thinking of are, of course, databases. With a lower
level of access than is currently accessibly, a database could allocate the
correct sized "blocks", and generally optimised it's use of the pool.

Another area which could be of use if filesystem development. A different
filesystem could be hooked in optimally. For example, a modified Ext4 with
access to the underlying pool could, in some circumstances, work better
than ZFS itself. I'm not saying it would, but in some circumstances it may
be a preferable option.

This could also be a good opportunity to "separate" the projects, so that
the ZPool, ZFS and ZVol can be worked on independently, communicating
through this API. Going a stage further, the VDevs could be separated out,
too. It would be good to be able to develop new VDev drivers, knowing there
is a stable API to work within, and develop, build and distribute them
independently of ZFS.

Let me know your thoughts. This has been at the back of my mind for a while
now, and I just thought I'd throw it out there for feedback.

Regards
Karl



-------------------------------------------
illumos-zfs
Archives: https://www.listbox.com/member/archive/182191/=now
RSS Feed: https://www.listbox.com/member/archive/rss/182191/23047029-187a0c8d
Modify Your Subscription: https://www.listbox.com/member/?member_id=23047029&id_secret=23047029-2e85923f
Powered by Listbox: http://www.listbox.com

[Schlacta, Christ]

I'd like an extension to export raw Zpool over the network in a way that a
remote zfs is aware of the target zpool, ids redundancy, topology, etc. for
*large* single image zfs deployments

The idea being that a single host be dedicated to cache, hash, log,
compress, (encrypt), and dedup for a few jbod, but passes the top level
management up to higher level zfs front ends.

Your api is a perfect demarcation point for this

Post by Karl Wagner
Hi All
I have a suggestion for a future development of ZFS: as the subject,
exposing a "Storage API" for ZFS.
The idea is fairly straight forward. We expose the functionality to store
objects in a ZPool. Currently (AFAIK) the only ways to do this are via a
ZFS filesystem tree or a ZVol. Neither are optimal in all situations, so
offering access to a layer below this could encourage developers to use ZFS
in a targetted way to enhance their projects.
The main things I am thinking of are, of course, databases. With a lower
level of access than is currently accessibly, a database could allocate the
correct sized "blocks", and generally optimised it's use of the pool.
Another area which could be of use if filesystem development. A different
filesystem could be hooked in optimally. For example, a modified Ext4 with
access to the underlying pool could, in some circumstances, work better
than ZFS itself. I'm not saying it would, but in some circumstances it may
be a preferable option.
This could also be a good opportunity to "separate" the projects, so that
the ZPool, ZFS and ZVol can be worked on independently, communicating
through this API. Going a stage further, the VDevs could be separated out,
too. It would be good to be able to develop new VDev drivers, knowing there
is a stable API to work within, and develop, build and distribute them
independently of ZFS.
Let me know your thoughts. This has been at the back of my mind for a
while now, and I just thought I'd throw it out there for feedback.
Regards
Karl
*illumos-zfs* | Archives<https://www.listbox.com/member/archive/182191/=now>
<https://www.listbox.com/member/archive/rss/182191/23054485-60ad043a> |
Modify<https://www.listbox.com/member/?&>Your Subscription
<http://www.listbox.com>

-------------------------------------------
illumos-zfs
Archives: https://www.listbox.com/member/archive/182191/=now
RSS Feed: https://www.listbox.com/member/archive/rss/182191/23047029-187a0c8d
Modify Your Subscription: https://www.listbox.com/member/?member_id=23047029&id_secret=23047029-2e85923f
Powered by Listbox: http://www.listbox.com

[Gary Mills]

Post by Schlacta, Christ
I'd like an extension to export raw Zpool over the network in a way
that a remote zfs is aware of the target zpool, ids redundancy,
topology, etc. for *large* single image zfs deployments
The idea being that a single host be dedicated to cache, hash, log,
compress, (encrypt), and dedup for a few jbod, but passes the top level
management up to higher level zfs front ends.
Your api is a perfect demarcation point for this

Ah, I like that one. I could have used it a few years back. We had a
single large LUN imported from a SAN device. This device did all of
the disk management including automatic replacement of failed disks.
Wanting to use ZFS, I built a zpool on that LUN. That added another
layer of disk management, except that this one was of no use.

To make matters worse, the SAN device was a COW syste, so we had two
levels of COW running too.

The challange is the end-to-end integrity checking that's part of ZFS.
How do you make that work properly when the filesystem is split into
upper and lower parts?

[Karl Wagner]

This isn't exactly the use case I had in mind, and goes beyond what Christ
said too (if I read correctly).

The only way you could maintain end to end integrity checking is if the San
exposed that information. Christ's idea seemed to be making a zpool on a
remote machine available over the network, which could then be a vdev for
another pool. By doing so, and exposing more of the pool, a lot could be
handled by the remote pool, but seen as local to the zfs (if that makes
sense).

Post by Gary Mills

Post by Schlacta, Christ
I'd like an extension to export raw Zpool over the network in a way
that a remote zfs is aware of the target zpool, ids redundancy,
topology, etc. for *large* single image zfs deployments
The idea being that a single host be dedicated to cache, hash, log,
compress, (encrypt), and dedup for a few jbod, but passes the top

level

Post by Schlacta, Christ
management up to higher level zfs front ends.
Your api is a perfect demarcation point for this

Ah, I like that one. I could have used it a few years back. We had a
single large LUN imported from a SAN device. This device did all of
the disk management including automatic replacement of failed disks.
Wanting to use ZFS, I built a zpool on that LUN. That added another
layer of disk management, except that this one was of no use.
To make matters worse, the SAN device was a COW syste, so we had two
levels of COW running too.
The challange is the end-to-end integrity checking that's part of ZFS.
How do you make that work properly when the filesystem is split into
upper and lower parts?
--
-Gary Mills- -refurb- -Winnipeg, Manitoba,
Canada-
-------------------------------------------
illumos-zfs
Archives: https://www.listbox.com/member/archive/182191/=now
https://www.listbox.com/member/archive/rss/182191/24409195-16edb367
https://www.listbox.com/member/?&
Powered by Listbox: http://www.listbox.com

-------------------------------------------
illumos-zfs
Archives: https://www.listbox.com/member/archive/182191/=now
RSS Feed: https://www.listbox.com/member/archive/rss/182191/23047029-187a0c8d
Modify Your Subscription: https://www.listbox.com/member/?member_id=23047029&id_secret=23047029-2e85923f
Powered by Listbox: http://www.listbox.com

[Will Andrews]

I have thought for some time that it would be interesting & potentially
useful to expose the DMU and ZAP interfaces directly to userland. There
are some applications that are primarily interested in managing object sets
on their storage, and don't need any file or filesystem access interfaces.

It could also be an useful jumping off point for improving the existing
dataset-objset abstraction & relationship.

--Will.

Post by Karl Wagner
Hi All
I have a suggestion for a future development of ZFS: as the subject,
exposing a "Storage API" for ZFS.
The idea is fairly straight forward. We expose the functionality to store
objects in a ZPool. Currently (AFAIK) the only ways to do this are via a
ZFS filesystem tree or a ZVol. Neither are optimal in all situations, so
offering access to a layer below this could encourage developers to use ZFS
in a targetted way to enhance their projects.
The main things I am thinking of are, of course, databases. With a lower
level of access than is currently accessibly, a database could allocate the
correct sized "blocks", and generally optimised it's use of the pool.
Another area which could be of use if filesystem development. A different
filesystem could be hooked in optimally. For example, a modified Ext4 with
access to the underlying pool could, in some circumstances, work better
than ZFS itself. I'm not saying it would, but in some circumstances it may
be a preferable option.
This could also be a good opportunity to "separate" the projects, so that
the ZPool, ZFS and ZVol can be worked on independently, communicating
through this API. Going a stage further, the VDevs could be separated out,
too. It would be good to be able to develop new VDev drivers, knowing there
is a stable API to work within, and develop, build and distribute them
independently of ZFS.
Let me know your thoughts. This has been at the back of my mind for a
while now, and I just thought I'd throw it out there for feedback.
Regards
Karl
*illumos-zfs* | Archives<https://www.listbox.com/member/archive/182191/=now>
<https://www.listbox.com/member/archive/rss/182191/22812380-5ee856a2> |
Modify<https://www.listbox.com/member/?member_id=22812380&id_secret=22812380-ce106d1b>Your Subscription
<http://www.listbox.com>

[Darren Reed]

Post by Karl Wagner
Hi All
I have a suggestion for a future development of ZFS: as the subject,
exposing a "Storage API" for ZFS.

Before going there, is there a REST API that is available as a package
for IllumOS?

Darren

[Keith Wesolowski]

Post by Darren Reed

Post by Karl Wagner
I have a suggestion for a future development of ZFS: as the subject,
exposing a "Storage API" for ZFS.

Before going there, is there a REST API that is available as a package
for IllumOS?

What does that even mean? There are undoubtedly a great many people who
have created REST APIs using various language platforms and packages to
interface with the system. There's no "REST API for illumos", though,
because that would be a monumentally huge undertaking. By my estimate
it would probably take a team of 10 or so people about 3 years just to
get something basically functional. And by "estimate", I mean
direct experience.

If you're asking whether there's software in widespread use that could
be used as the foundation for such an undertaking, the answer is yes.
In fact, there are dozens. Our preference is for node.js and Restify
(https://github.com/mcavage/node-restify) and we do use this to expose a
number of higher-level systems behaviours to our operators and
customers. It is available as an npm (http://npmjs.org) package. That
said, I'm sure you could construct a table of dozens more that includes
all your favourite languages. What's the point?

[Darren Reed]

Post by Keith Wesolowski
...
What does that even mean? There are undoubtedly a great many people who
have created REST APIs using various language platforms and packages to
interface with the system. There's no "REST API for illumos", though,
because that would be a monumentally huge undertaking. By my estimate
it would probably take a team of 10 or so people about 3 years just to
get something basically functional. And by "estimate", I mean
direct experience.
If you're asking whether there's software in widespread use that could
be used as the foundation for such an undertaking, the answer is yes.
In fact, there are dozens. Our preference is for node.js and Restify
(https://github.com/mcavage/node-restify) and we do use this to expose a
number of higher-level systems behaviours to our operators and
customers. It is available as an npm (http://npmjs.org) package. That
said, I'm sure you could construct a table of dozens more that includes
all your favourite languages. What's the point?

When I look at where interfaces (both programming and administrative)
are going, they are increasingly becoming web based and REST is the
keyword that storage vendors like to mention when it comes to providing
an external interface.

Darren

[Radio młodych bandytów]

Sounds like Ceph's RADOS. In general, a good thing, but in case of ZFS I
don't view it as very useful. I see 3 areas of use for it:
* block storage for virtualisation
* file storage for most of other uses
* object storage for cases where blocks are too rough, yet POSIX
semantics are a factor limiting performance or scalability.
ZFS covers the first and the second case already. Since it's not
distributed, it won't cover the scalability-limited uses well.

A FS implementation optimised for particular uses might work well, but
the *need* is a niche one. Getting applications to work a step closer to
hardware might make them faster, but the cost / benefit of optimising
for this single platform will be worthwhile for few.
--
Twoje radio

Post by Karl Wagner
Hi All
I have a suggestion for a future development of ZFS: as the subject,
exposing a "Storage API" for ZFS.
The idea is fairly straight forward. We expose the functionality to
store objects in a ZPool. Currently (AFAIK) the only ways to do this are
via a ZFS filesystem tree or a ZVol. Neither are optimal in all
situations, so offering access to a layer below this could encourage
developers to use ZFS in a targetted way to enhance their projects.
The main things I am thinking of are, of course, databases. With a lower
level of access than is currently accessibly, a database could allocate
the correct sized "blocks", and generally optimised it's use of the pool.
Another area which could be of use if filesystem development. A
different filesystem could be hooked in optimally. For example, a
modified Ext4 with access to the underlying pool could, in some
circumstances, work better than ZFS itself. I'm not saying it would, but
in some circumstances it may be a preferable option.
This could also be a good opportunity to "separate" the projects, so
that the ZPool, ZFS and ZVol can be worked on independently,
communicating through this API. Going a stage further, the VDevs could
be separated out, too. It would be good to be able to develop new VDev
drivers, knowing there is a stable API to work within, and develop,
build and distribute them independently of ZFS.
Let me know your thoughts. This has been at the back of my mind for a
while now, and I just thought I'd throw it out there for feedback.
Regards
Karl
*illumos-zfs* | Archives
<https://www.listbox.com/member/archive/182191/=now>
<https://www.listbox.com/member/archive/rss/182191/22867980-58f9c4c4> |
Modify
<https://www.listbox.com/member/?&>
Your Subscription [Powered by Listbox] <http://www.listbox.com>