ZFS
Differences between revisions 13 and 79 (spanning 66 versions)
|
Size: 5673
Comment:
|
← Revision 79 as of 2019-01-22 22:16:08 ⇥
Size: 5646
Comment: remove 'stories' that aren't actually ZFS features; fix one that was close; fix typo in another
|
| Deletions are marked like this. | Additions are marked like this. |
| Line 1: | Line 1: |
| ##(see the SpecSpec for an explanation) | == Summary == ZFS is a combined [[https://en.wikipedia.org/wiki/File_system|file system]] and [[https://en.wikipedia.org/wiki/Logical_volume_management|logical volume manager]] designed and implemented by a team at Sun Microsystems led by Jeff Bonwick and Matthew Ahrens. Its development started in 2001 and it was officially announced in 2004. In 2005 it was integrated into the main trunk of Solaris and released as part of OpenSolaris. Currently, as of January 2015, it is native to [[http://www.oracle.com/solaris/|Solaris]], [[http://www.opensolaris.org/|OpenSolaris]], [[http://openindiana.org/|OpenIndiana]], [[https://www.illumos.org/projects|illumos]], [[https://smartos.org/|Joyent SmartOS]], [[http://omnios.omniti.com/|OmniOS]], [[http://www.freebsd.org/|FreeBSD]], [[http://www.debian.org/ports/kfreebsd-gnu/|Debian GNU/kFreeBSD systems]], [[http://www.netbsd.org/|NetBSD]], [[http://osv.io/|OSv]] and supported on [[https://en.wikipedia.org/wiki/Mac_OS|Mac OS]] with [[http://code.google.com/p/maczfs/|MacZFS]]. |
| Line 3: | Line 5: |
| * '''Launchpad Entry''': https://launchpad.net/distros/ubuntu/+spec/zfs-filesystem * '''Created''': [[Date(2006-05-02T09:59:41Z)]] by JackWasey * '''Contributors''': JackWasey * '''Packages affected''': |
The name "ZFS" originally stood for "Zettabyte File System". Currently it can store up to 256 ZiB (zebibytes). |
| Line 8: | Line 7: |
| == Summary == ZFS is an open source filesystem used in Solaris 10. It has powerful applications for server and workstation use cases, and would be a radical and attractive feature or basis for Ubuntu. It is released under the [http://en.wikipedia.org/wiki/CDDL CDDL] licence, which looks like it will be in universe, not main for now. |
== Installing ZFS on Ubuntu == The ZFS filesystem is available for Ubuntu as either a FUSE module or a native kernel module. The kernel module is provided by default. To install the user-level tools, simply install: {{{ sudo apt install zfsutils-linux }}} For all current versions from 16.04 onward. In addition to be able to have ZFS on root, install: {{{ sudo apt install zfs-initramfs }}} See also: * [[https://wiki.ubuntu.com/Kernel/Reference/ZFS|Ubuntu Kernel Team ZFS Reference Guide]] Otherwise: * [[https://launchpad.net/~zfs-native/+archive/stable/| Native ZFS for Ubuntu PPA]] (for [[http://zfsonlinux.org/faq.html#WhyShouldIUseA64BitSystem| 64b systems only]]) * [[http://www.zfsonlinux.org/| Native ZFS for Linux]] project home. * ZFS-FUSE project (deprecated). |
| Line 12: | Line 31: |
| Ubuntu server, and linux servers in general compete with other unices and windows. ZFS is a killer-app for Solaris, as it allows straightforward administration of a pool of disks, while giving intelligent performance and data integrity. | Ubuntu server, and Linux servers in general compete with other Unixes and Microsoft Windows. ZFS is a killer-app for Solaris, as it allows straightforward administration of a pool of disks, while giving intelligent performance and data integrity. |
| Line 14: | Line 33: |
| ZFS does away with partitioning, EVMS, LVM, MD, etc., etc.. The available disks (of any size) are used to the best of their ability. Compression can be used to increase bandwidth. (q.v. Reiser 4, and cloop?) | ZFS does away with partitioning, EVMS, LVM, MD, etc. The available disks (of any size) are used to the best of their ability. Compression can be used to increase bandwidth. (q.v. Reiser 4, and cloop?) |
| Line 16: | Line 35: |
| ZFS is 128 bit, meaning it is excessively scalable. (e.g. 16 exabyte limit) | ZFS is 128-bit, meaning it is very scalable. (e.g. 16 exabyte limit). Along with its record-setting scalability, it has a number of features that no other file system in production has: * State-of-the-art ''unsurpassed'' data reliability (including silent corruption detection and automatic data repair, if available) from end to end, thanks to transparent checksums in parent block pointers * Elimination of the RAID5 write hole that usually requires battery-powered disk technology or UPS technology to keep RAID arrays consistent in case of power failures * Online array reconstruction / reassemble, in a fraction of the time usually required to reconstruct an array, thanks to reconstruction dependent on contents rather than on disk topology * Online check (scrub) and online resilver of faulty, to verify and preserve the validity of on-disk data in the face of silent disk data corruption * Automatic deduplication and compression of data, selectable per volume or filesystem according to administrator policy |
| Line 25: | Line 50: |
| * Multiple independent prefetch streams with automatic length and stride detection | * Multiple independent prefetch streams with automatic length and stride detection for maximum streaming performance |
| Line 29: | Line 54: |
| [http://www.sun.com/2004-0914/feature/ Sun ZFS article] | [[http://web.archive.org/web/20071015014209/http://www.sun.com/2004-0914/feature/|Sun ZFS article (archived on archive.org)]] |
| Line 33: | Line 58: |
| * Ari has a single disk workstation. She buys a new disk and plugs it in. ZFS automatically adds the new disk space into the pool. Her home directory is mirrored, while her OS and temp space is striped automatically in the background. | |
| Line 35: | Line 59: |
| * Jack has three disks of different sizes. Configuring a sensible partition and RAID set-up is completely trial and error. ZFS abstracts the three disks into one pool of space, and gives the best balance of performance and security. Jack declares that some media directories do not need to be fault-tolerant, and ZFS transparently stripes them across all the disks. == Scope == Server use case is the most obvious. == Design == ZFS can be used as a drop-in POSIX replacement for existing file systems. However, it provides powerful capabilities which can permeate Ubuntu. E.g. each user is allocate a file-system, cloned from an original: unchanged files are only stored once. This paradigm would suit LTSP, where each user would have filesystem. This sounds more a bit more radical than it is. == Implementation == * Port ZFS to Linux. N.b. people are working on a BSD port already. * ZFS will be available in OS X 10.5 ("Leopard") and will be the default filesystem for new systems. * Integrate already ported http://www.wizy.org/wiki/ZFS_on_FUSE * Investigate use of ZFS features to replace clunky old linux tools. e.g. Version Control could be done at least partly by the FS ZFS can look back in time to see previous state, so could this be used for reconciling changes, e.g. for Unison or iFolder style file syncing for people who work in several locations. === Code === === Data preservation and migration === == Outstanding issues == == BoF agenda and discussion == == References == [http://www.opensolaris.org/os/community/zfs/ opensolaris ZFS community] [http://en.wikipedia.org/wiki/ZFS wikipedia entry on ZFS] === News and blogs articles === [http://www.internetnews.com/dev-news/article.php/3565221 internetnews] "We would not at all be surprised to see ZFS start to show up in other operating systems at some point in the not-too-distant future," Weinberg said. [http://blogs.sun.com/roller/page/jwadams?entry=an_initial_encounter_with_zfs a sun blog] [http://node-0.mneisen.org/2006/12/31/zfs-unter-ubuntu-kubuntu-610-edgy-eft/ German blogpost on how to compile and install ZFS on (K)Ubuntu.] Just follow the supplied command line instructions if you do not speak German :-D ] == Comments == [wiki:JoeyStanford Joey Stanford]: +1. It should also be noted that ext2 and ext3 will no longer under active development as new development is scheduled for [http://lkml.org/lkml/2006/6/28/454 ext4]. Brit Butler: +1. It's pretty clear that Sun has got a terrific filesystem on their hands and that many people are interested in moving in this direction. I realize there are still large issues with getting ZFS on Linux but there's also nothing better than our community for making it happen. ["Philbull"]: I agree with adding ZFS support as it seems to have many benefits and features. However, is ZFS able to work with dual-boot setups (e.g. Windows or other Linux distros)? Is there any way to read a ZFS disk from Windows, as there is with ext3? Jack L.: It's been announced as 'the filesystem for OS X 10.5.' I would see this as critical to dualbooting Macintosh users. ---- CategorySpec |
* Manuel has a mirrored disk array. Fluctuations in his computer's PSU are causing silent data corruption errors. ZFS detects the errors and reports back those errors to Manuel, protecting the data already on disk from further corruption. * Andreas has a disk that has suffered a partial failure. Since he chose a policy of two copies for his personal file system volume, he may be able to recover some of his latest personal data without having to resort to tape backups. * John has an 18-disk SAS array configured to run mirrored, each nine-disk group containing one RAIDZ2 leg. One of the disks experiences a catastrophic disk motor failure. John deactivates the disk online, replaces it with a fresh one, activates that disk, then uses the ZFS `zpool replace` command to replace the faulty disk. ZFS automatically reconstructs the data on that disk with zero downtime and minimal data transfer or performance impact to the array. |
Summary
ZFS is a combined file system and logical volume manager designed and implemented by a team at Sun Microsystems led by Jeff Bonwick and Matthew Ahrens. Its development started in 2001 and it was officially announced in 2004. In 2005 it was integrated into the main trunk of Solaris and released as part of OpenSolaris. Currently, as of January 2015, it is native to Solaris, OpenSolaris, OpenIndiana, illumos, Joyent SmartOS, OmniOS, FreeBSD, Debian GNU/kFreeBSD systems, NetBSD, OSv and supported on Mac OS with MacZFS.
The name "ZFS" originally stood for "Zettabyte File System". Currently it can store up to 256 ZiB (zebibytes).
Installing ZFS on Ubuntu
The ZFS filesystem is available for Ubuntu as either a FUSE module or a native kernel module. The kernel module is provided by default. To install the user-level tools, simply install:
sudo apt install zfsutils-linux
For all current versions from 16.04 onward.
In addition to be able to have ZFS on root, install:
sudo apt install zfs-initramfs
See also:
Otherwise:
Native ZFS for Linux project home.
- ZFS-FUSE project (deprecated).
Rationale
Ubuntu server, and Linux servers in general compete with other Unixes and Microsoft Windows. ZFS is a killer-app for Solaris, as it allows straightforward administration of a pool of disks, while giving intelligent performance and data integrity.
ZFS does away with partitioning, EVMS, LVM, MD, etc. The available disks (of any size) are used to the best of their ability. Compression can be used to increase bandwidth. (q.v. Reiser 4, and cloop?)
ZFS is 128-bit, meaning it is very scalable. (e.g. 16 exabyte limit). Along with its record-setting scalability, it has a number of features that no other file system in production has:
State-of-the-art unsurpassed data reliability (including silent corruption detection and automatic data repair, if available) from end to end, thanks to transparent checksums in parent block pointers
- Elimination of the RAID5 write hole that usually requires battery-powered disk technology or UPS technology to keep RAID arrays consistent in case of power failures
- Online array reconstruction / reassemble, in a fraction of the time usually required to reconstruct an array, thanks to reconstruction dependent on contents rather than on disk topology
- Online check (scrub) and online resilver of faulty, to verify and preserve the validity of on-disk data in the face of silent disk data corruption
- Automatic deduplication and compression of data, selectable per volume or filesystem according to administrator policy
ZFS achieves its impressive performance through a number of techniques:
- Dynamic striping across all devices to maximize throughput
- Copy-on-write design makes most disk writes sequential
- Multiple block sizes, automatically chosen to match workload
- Explicit I/O priority with deadline scheduling
- Globally optimal I/O sorting and aggregation
- Multiple independent prefetch streams with automatic length and stride detection for maximum streaming performance
- Unlimited, instantaneous read/write snapshots
- Parallel, constant-time directory operations
Sun ZFS article (archived on archive.org)
Use cases
- Dani is a sys admin for a medium-size business. She spends 20% of her time planning/changing/administering/repairing the disk arrays of three servers. With ZFS, this time is cut in half.
- Power user Petra has a four-disk RAID5 workstation. Parity calculations make this a fairly slow set-up because of the number of writes of small files. She upgrades to ZFS and sees performance benefits, because small files are mirrored instead of included in parity calculations.
- Manuel has a mirrored disk array. Fluctuations in his computer's PSU are causing silent data corruption errors. ZFS detects the errors and reports back those errors to Manuel, protecting the data already on disk from further corruption.
- Andreas has a disk that has suffered a partial failure. Since he chose a policy of two copies for his personal file system volume, he may be able to recover some of his latest personal data without having to resort to tape backups.
John has an 18-disk SAS array configured to run mirrored, each nine-disk group containing one RAIDZ2 leg. One of the disks experiences a catastrophic disk motor failure. John deactivates the disk online, replaces it with a fresh one, activates that disk, then uses the ZFS zpool replace command to replace the faulty disk. ZFS automatically reconstructs the data on that disk with zero downtime and minimal data transfer or performance impact to the array.
ZFS (last edited 2019-01-22 22:16:08 by seth-arnold)