In a previous post, I described some of my early approaches to building a NAS for my home. In summary, I decided to use Linux software raid formatted with Linux’s built-in ext3 format. This setup worked, but left me with three main problems:

  1. Volume size limitations: ext3 is capable of handling a maximum volume size of 16TB, when formatted with a 4kb block size. It is theoretically possible to use a larger block size and thus a larger maximum volume size. However, changing the block size requires reformatting the volume, which I could not do. Additionally, most hard drives natively use a 4kb block size. Matching filesystem block size to hard drive block size generally achieves best performance, for reasons such as avoiding a single write operation becoming multiple operations - read-modify-write.

  2. High Availability: I enjoyed having a NAS so much, that I quickly began running additional servers that would mount the NAS’s storage volume and use it for whatever large storage was needed. I can’t recall which network filesystem I was using at the time, but it was one that required a re-mount after the server rebooted. NFS would make this less of a problem. Anyway - reboots aside - I wanted a NAS that would remain available even after one of the computers backing it failed, or needed to be rebooted for updates, for example.

  3. Space limitations: physical space becomes a limiting factor for NAS side as well. Not many server chassis had physical space for many hard drives, outside of rackable enterprise-grade systems that I couldn’t afford. Networks allow multiple computers to communicate, so why would my filesystem be limited to one machine? I wanted a NAS that could be “scaled up” by adding more computer servers and hard drives to it.

GlusterFS⌗

Enter GlusterFS - GlusterFS is a distributed network filesystem. GlusterFS provides data safety features similar to RAID, but offered in a manner that spans multiple computers instead of multiple hard drives.

The general way in which GlusterFS functions is simple: you format your hard drives with whatever filesystem you prefer - I used ZFS - and GlusterFS creates a virtual filesystem on top of them. GlusterFS calls each sub-volume a “brick”, and while each brick only spans one server, the virtual GlusterFS filesystem can incorporate multiple bricks from multiple servers.

What’s great about GlusterFS’s approach to storage is that it single-handedly solves each of the 3 issues above:

  1. Volume size limitations: GlusterFS uses its own type of filesystem with astronomically high volume size limitations.

  2. High Availability: with a “replicated” GlusterFS volume, multiple copies of each file are stored on separate bricks. Since separate bricks are on separate servers, the file remains accessible even if one server or hard drive fails.

  3. Space limitations: since GlusterFS volumes can span across multiple servers, space becomes nearly limitless. You can always plug more servers into the cluster.

Design⌗

I started with two servers, but later moved to four. Each could hold up to 8 hard drives. A common build at the time was a mini-NAS, with a case by U-NAS, the NSC-800. For a motherboard/cpu combo, I chose AsRock Rack. I started with their Mini-ITX form factor server motherboard, the C2750D4I, and later upgraded to the C3758D4I. Both feature 8 cores, low power consumption, and small physical size.

As mentioned above, I used ZFS to manage my server’s hard drives. Initially, I had 16 drives spread across two servers. I divided these into 4 ZFS mirrors - like RAID 1 - of 2 drives each per server. Then, each ZFS mirror was replicated with GlusterFS to a corresponding ZFS mirror on the other server. This worked well, but I later moved to two ZFS RAIDZ pools per server, made from 4 drives each. This reduced the number of bricks which made GlusterFS simpler to manage.

Results⌗

Overall, this was a fairly reliable and reasonably performant NAS. Generally speaking, I was bottlenecked by drive speed as I used “green” hard drives that put out less noise and heat and consumed less power at the expense of some performance. Sequential access was great but one area the NAS really suffered in was random access. Listing large directories was noticeably slow as well. Despite these issues, this NAS went on to serve me for many years!

If you run a similar setup, let me know!