At one of the large Internet companies that I worked at, NFS was heavily used for sharing files for internal processing.
Here are some of my opinions based on that experience.
The first thing to do for greater performance and reliability is to plug server nodes and the fileserver into the same backplane in the same switch using Cat6 factory-made cables.
Next, you have to pick NFS client and server software.
Some implementations:
- linux NFS4 – really scary because linux NFS3 is historically flaky, and this is newer and more complicated
- linux NFS3 – scary because linux NFS is historically flaky
- FreeBSD NFS3 – prolly works
- Solaris NFS3 – prolly works
- NetApp filer – works pretty well
To emphasize the different experiences that linux and Sun folks have had with NFS, a roomful of linux people gasped when a Sun storage engineer asked why you wouldn’t run MySQL on NFS.
Some filesystem stress test tools:
- fsx.c (ported from Next to FreeBSD and linux)
- bonnie
- make -j 10x in loop
- unpack and build emacs on a network/distributed filesystem (traditional acceptance test by scientific cluster sponsors – rarely passed)
Check the nfs configuration file on each node, also ifconfig output for errors.
Obviously, write temp files to local disk or a a RAM disk like linux tmpfs.
If you can design your application to retry disk IO a few times on read or write errors, that also helps, especially under heavy fileserver load.
Regarding distributed filesystems or cluster filesystems, the few success stories that I’ve heard are implementations on homogeneous hardware, where the code was tweaked for their specific servers, NICs, switches and apps by an on-site team of kernel and networking programmers.
