ZFS

Differences between revisions 47 and 48
⇤ ← Revision 47 as of 2010-08-03 20:01:52 →
Size: 12551
Editor: 186
Comment:
← Revision 48 as of 2010-08-03 20:03:00 → ⇥
Size: 12571
Editor: 186
Comment:
Deletions are marked like this. Additions are marked like this.
Line 10: Line 10:
ZFS is an open source file system licensed under the WikiPedia:CDDL. ZFS is used in Solaris, OpenSolaris, and FreeBSD. It has powerful applications for server and workstation use cases, and would be a radical and attractive feature or basis for Ubuntu. Due to CDDL incompatibility with GPL, and the fact that all Linux kernel drivers must be GPL compatible, ZFS remains outside the Linux kernel. There is talk about ZFS having dual licensing CDDL and GPL, but, quoting Linus Torvalds, "I can almost guarantee that Sun won't release ZFS under the GPLv3 even if they release other parts. Because if they did, they'd lose the patent protection." Linus's email about ZFS can be found [[http://lkml.org/lkml/2007/6/12/232 | here]]. ZFS is an open source file system licensed under the WikiPedia:CDDL. ZFS is used in Solaris, OpenSolaris, FreeBSD and Debian GNU/kFreeBSD. It has powerful applications for server and workstation use cases, and would be a radical and attractive feature or basis for Ubuntu. Due to CDDL incompatibility with GPL, and the fact that all Linux kernel drivers must be GPL compatible, ZFS remains outside the Linux kernel. There is talk about ZFS having dual licensing CDDL and GPL, but, quoting Linus Torvalds, "I can almost guarantee that Sun won't release ZFS under the GPLv3 even if they release other parts. Because if they did, they'd lose the patent protection." Linus's email about ZFS can be found [[http://lkml.org/lkml/2007/6/12/232 | here]].

Summary

ZFS is an open source file system licensed under the CDDL. ZFS is used in Solaris, OpenSolaris, FreeBSD and Debian GNU/kFreeBSD. It has powerful applications for server and workstation use cases, and would be a radical and attractive feature or basis for Ubuntu. Due to CDDL incompatibility with GPL, and the fact that all Linux kernel drivers must be GPL compatible, ZFS remains outside the Linux kernel. There is talk about ZFS having dual licensing CDDL and GPL, but, quoting Linus Torvalds, "I can almost guarantee that Sun won't release ZFS under the GPLv3 even if they release other parts. Because if they did, they'd lose the patent protection." Linus's email about ZFS can be found here.

Rationale

Ubuntu server, and Linux servers in general compete with other Unixes and Microsoft Windows. ZFS is a killer-app for Solaris, as it allows straightforward administration of a pool of disks, while giving intelligent performance and data integrity.

ZFS does away with partitioning, EVMS, LVM, MD, etc. The available disks (of any size) are used to the best of their ability. Compression can be used to increase bandwidth. (q.v. Reiser 4, and cloop?)

ZFS is 128-bit, meaning it is very scalable. (e.g. 16 exabyte limit)

ZFS achieves its impressive performance through a number of techniques:

  • Dynamic striping across all devices to maximize throughput
  • Copy-on-write design makes most disk writes sequential
  • Multiple block sizes, automatically chosen to match workload
  • Explicit I/O priority with deadline scheduling
  • Globally optimal I/O sorting and aggregation
  • Multiple independent prefetch streams with automatic length and stride detection
  • Unlimited, instantaneous read/write snapshots
  • Parallel, constant-time directory operations

Sun ZFS article

Use cases

  • Dani is a sys admin for a medium-size business. She spends 20% of her time planning/changing/administering/repairing the disk arrays of three servers. With ZFS, this time is cut in half.
  • Ari has a single disk workstation. She buys a new disk and plugs it in. ZFS automatically adds the new disk space into the pool. Her home directory is mirrored, while her OS and temp space is striped automatically in the background.
  • Power user Petra has a four-disk RAID5 workstation. Parity calculations make this a fairly slow set-up because of the number of writes of small files. She upgrades to ZFS and sees performance benefits, because small files are mirrored instead of included in parity calculations.
  • Jack has three disks of different sizes. Configuring a sensible partition and RAID set-up is completely trial and error. ZFS abstracts the three disks into one pool of space, and gives the best balance of performance and security. Jack declares that some media directories do not need to be fault-tolerant, and ZFS transparently stripes them across all the disks.

Scope

Server use case is the most obvious.

Design

ZFS can be used as a drop-in POSIX replacement for existing file systems. However, it provides powerful capabilities which can permeate Ubuntu. E.g. each user is allocate a file-system, cloned from an original: unchanged files are only stored once. This paradigm would suit LTSP, where each user would have filesystem. This sounds more a bit more radical than it is.

Implementation

  • Port ZFS to Linux. N.b. people are working on a BSD port already.

  • A ZFS will be readable but not writable in OS X 10.5 ("Leopard"), Mac OS forge now has a write support for ZFS on Leopard.

  • Integrate already ported http://zfs-fuse.net/

  • Investigate use of ZFS features to replace clunky old linux tools. e.g. Version Control could be done at least partly by the FS ZFS can look back in time to see previous state, so could this be used for reconciling changes, e.g. for Unison or iFolder style file syncing for people who work in several locations.

Installation on Ubuntu 9.04

Filip Brcic is kindly providing Ubuntu packages for zfs-fuse.

To install zfs-fuse add the Filip Brcic's launchpad repo to a source list. 

sudo nano /etc/apt/sources.list.d/zfs-fuse.list

Add... 

deb http://ppa.launchpad.net/brcha/ubuntu jaunty main
deb-src http://ppa.launchpad.net/brcha/ubuntu jaunty main

Then update apt. 

sudo aptitude update

Now install zfs-fuse. 

sudo aptitude install zfs-fuse

Installation on Ubuntu 8.10

Filip Brcic is kindly providing Ubuntu packages for zfs-fuse.

To install zfs-fuse add the Filip Brcic's launchpad repo to a source list. 

sudo nano /etc/apt/sources.list.d/zfs-fuse.list

Add... 

deb http://ppa.launchpad.net/brcha/ubuntu intrepid main
deb-src http://ppa.launchpad.net/brcha/ubuntu intrepid main

Then update apt. 

sudo aptitude update

Now install zfs-fuse. 

sudo aptitude install zfs-fuse

Installation on Ubuntu 8.04

Same as 8.10, just replace intrepid with hardy in the deb lines.

Installation on Ubuntu 7.10

Install dependencies: 

aptitude install devscripts build-essential zlib1g-dev libfuse-dev scons debhelper fakeroot dpatch docbook-xsl

Grab the debian source package: 

wget http://fushizen.net//zfs-fuse/zfs-fuse_0.4.0~beta1.hg20070418.227.22a65c23850b-0~pre0.dsc''

Pack the debian source archive: 

''dpkg-source -x zfs-fuse_0.4.0~beta1.hg20070418.227.22a65c23850b-0~pre0.dsc''

Change into the zfs-fuse directory: 

cd zfs-fuse[tab]

Build the debian package: 

dpkg-buildpackage -rfakeroot -us -uc -b

Install the zfs debian package: 

sudo dpkg -i ../zfs-fuse*.deb

(instructions adapted from Debian installation instructions in References section below)

Installation on Ubuntu 7.10 Alternative from Source

Install dependencies: 

sudo aptitude install make build-essential scons fuse-utils libfuse-dev mercurial zlib1g-dev libaio-dev

You should get the most recent version of zfs-fuse from the trunk (as advised in this post). 

hg clone http://www.wizy.org/mercurial/zfs-fuse/trunk

Then go to the source directory 

cd trunk/src

Now build 

scons

And install 

sudo scons install

Finished!

Now, we can run zfs-fuse, by using this command 

/usr/local/sbin/zfs-fuse

or just 

zfs-fuse

For help, run 

zfs-fuse --help

ZPools

ZFS/ZPool Creating and managing ZFS ZPools.

Code

Data preservation and migration

Outstanding issues

BoF agenda and discussion

References

Community adopted a new development project head for zfs-fuse (GIT repo, thread)

Unstable branch is now at pool version 22 (data dedup). Proposed project home is at http://zfs-fuse.net/

News and blogs articles

Comments

Joey Stanford: +1. It should also be noted that ext2 and ext3 will no longer under active development as new development is scheduled for ext4.

Brit Butler: +1. It's pretty clear that Sun has got a terrific filesystem on their hands and that many people are interested in moving in this direction. I realize there are still large issues with getting ZFS on Linux but there's also nothing better than our community for making it happen.

Philbull: I agree with adding ZFS support as it seems to have many benefits and features. However, is ZFS able to work with dual-boot setups (e.g. Windows or other Linux distros)? Is there any way to read a ZFS disk from Windows, as there is with ext3?

Jack L.: It's been announced as 'the filesystem for OS X 10.5.' I would see this as critical to dualbooting Macintosh users.

Paul K.: Jack L.- Sun announced it would be in MacOS, but Apple has since stated they will be able to read from those volumes should they exist and it will not be the default FS. Philbul - Yes, you can define how much and which disks are available in a "Pool." If you want to dual-boot Windows or another OS, leave some "unpooled freespace" for your other OS to use.

Jack W.: So how is it allowed that ati and nvidia can link binary modules with no source whatsoever to the kernel? Is the problem a that ZFS wouldn't be able to go in main, or that compiling it with or into the kernel is a licence violation, or both? Suppose someone were to make a ZFS module that could be distributed independently; would this be a workaround?

Graeme H.: From what I understand, ATI and NVidia have open source modules that then load binary compiled code, which is their closed source driver. Presumably, this could be done for ZFS as well. However, this is difficult to write and debug, especially when ZFS needs to interact at the block, lvm, and raid layers.

Seth H.: @Graeme: I don't see why ZFS would need to interact at the lvm/raid layers. ZFS has it's own code for striping, mirroring, parity and redundancy (oh and compression and end-to-end checksumming... the list goes on). The current zfs-fuse port duplicate most all these features from Solaris ZFS without needing any other access than at the block device. In other words, there is no need to deal with lvm/dmraid or whatnot since zfs implements these features independently. This may also greatly reduce the licensing implications.

Ivan K.: Guys, how about installation security? Why there is nothing about importing proper GPG signature? Are you waiting for a debfile proxy sploit like this to become popular? I've imported all of known Filip Brcic's GPG signatures but still get the warning that zfs-fuse package cannot be authenticated.

Grizzly: I basically enjoy zfs on linux - still, 0.5.0 is very outdated as the zfs-fuse.net webpage has 0.6.0 as src and rpm available. What could be done to get zfs-fuse into let's say restricted repository and provide 0.6.0 or 0.6.1 as fully-featured .deb the same way, filip has accomplished? I simply do not (yet) know enough about packaging...

Tuco: ZFS is now available on Debian thanks to GNU/kFreeBSD project, see my blog entry about the subject. Maybe Ubuntu could support kFreeBSD and then ZFS?

CategorySpec

ZFS (last edited 2019-01-22 22:16:08 by seth-arnold)