Sun's new file system, Zettabyte File System (ZFS) is changing the world. Well, the computing world, at least. ZFS brings with it a host of advantages, but it also has limitations, and performance considerations and manageability should be factored into the migration decision and deployment process.
For starters, sites with large amounts of storage, say .5 PB or more, use Veritas almost without exception. Their volume manager, VxVM, combined with the file system, VxFS, makes storage management very robust and easy to handle, compared with native OS file systems and volume managers. That is, until ZFS. The main remaining advantage for VxVM/FS is that you can use the same volumes on multiple platforms. AIX, Linux, Solaris and other operating systems can all use the same file system, and frequently the Veritas file system outperforms native alternatives. Being able to use the same file system across multiple platforms is important, especially when a SAN is involved. Migrating services to new servers on different platforms is far simpler when there is no need to worry about file systems.
ZFS outperforms Veritas VxVM/FS for most uses, and is far more flexible and easy to mange. However, ZFS is available only on Solaris, right now.
Performance is an important consideration for file system selection. Running a general file server (perhaps for home directories), a mail server, and an Oracle database are all very different problems that demand certain optimizations at the file system level. Advanced file systems layered on top of a volume manager enable block size, or Record Size as ZFS calls it, to be set on the fly. Being able to tune the file system without re-creating it saves tremendous amounts of downtime and headache. A great FileBench comparison between Veritas and ZFS, done by a Sun engineer, shows the areas where ZFS excels most.
ZFS also supports compression, which dramatically increases performance. Even on a slower machine by today's standards, ZFS with compression on still provides much faster data access for most uses. The slow part in reading or writing to disks is in waiting for the slow heads and spindles, so reading and writing less (because it's compressed) is very helpful. Don't forget that compression can sometimes double your amount of space, too. Benchmarks agree. James Dickens blogged about his findings while installing Solaris Zones on UFS, ZFS, and ZFS with compression.
Dominic Kay, an engineer from Sun, wrote about the differences in managing ZFS vs. VxVM/FS. The second table shows the error-prone list of commands required to set up Vertias storage compared to the two ZFS commands. Of course, storage administrators must still understand every detail of what they're doing, but the key difference here is that the ZFS file systems are created instantly, whereas the Veritas ones can take hours to initialize.
Unfortunately, the concept of "managing storage" must also take into account the previously mentioned interoperability aspects. Veritas is stable and available on all platforms; ZFS is not. ZFS is open source. People are working on porting it to other operating systems, but the going is slow. If you can get away with all-Solaris, and know that you'll never want to move these file systems to another platform, ZFS is the clear winner.
One major issue with ZFS is that it's available only for Solaris right now. That's not a limitation for Solaris shops.
Another frequent complaint is that ZFS cannot live on the root file system. People would like to use ZFS instead of Solaris SVM to mirror operating system disks. This is being taken care of, but its usefulness is debatable. Certainly, when ZFS is ported to other operating systems, they won't want or need to support having a ZFS root file system. Many believe the more time-tested file systems and volume managers should handle the OSes file system.
A big point of confusion for new ZFS adopters is backups. There's no "dump" command for ZFS, they expect you to use snapshots and "zfs send." The ZFS "send" command enables administrators to send a file system wherever they wish: tape, a file, wherever. Traditionally, backups are done incrementally; backup everything that changes over a period of time, and every once in a while backup the entire file system as well. The problem is that you cannot perform incremental backups with traditional tools because there's no zfsdump equivalent.
A ZFS RAIDz pool will survive if a disk dies, so Sun is trying to change how people backup data. The idea is to use snapshots daily, hourly, or however often you need. A snapshot stores only things that change since the previous snapshot, using very little space. The incremental backup is online and ready for access. Daily or weekly full dumps of a file system can still be performed for disaster recovery purposes.
This is all well and good most people can handle the fact that incremental backups are stored in snapshots, since the ZFS pool can be made impervious to failures but there are drawbacks.
ZFS does not support quotas. It does, but the notion of "quota" in ZFS means that you're limiting the size of the file system. The intended design is that each user or project will get its own file system, and the appropriate quotas get applied. ZFS handles thousands of file systems very well, so this isn't a problem, until we start thinking about backups again. ZFS snapshots can only be stored on the file system in question, in a .zfs directory. If there are active snapshots, all changes get copied into them. So when users come close to their quota, and want to delete files to free up space, they cannot. The data just gets moved into the snapshot, so an administrator must free his space by removing snapshots. The ability to store snapshots at a higher level, or on another file system would solve this problem. Sun hasn't announced any plans to fix this though.
The final annoyance is that you cannot remove devices from pools. You can replace them if one happens to fail, but the pool itself will not allow you to decrease its size. Sun is working on this problem.
Minor annoyances aside, ZFS is great. In Solaris environments there is no reason not to use ZFS file systems. In mixed environments, where you'd like the portability Veritas products provides, ZFS will likely be an option in a few years.