Linux on davepedu.com

highly available redis with redis-sentinel-proxy

dave@davepedu.com (dave) — Thu, 05 Mar 2020 00:00:00 +0000

Redis is a widely used and extremely fast cache server. Redis supports high availability out-of-the-box, but requires a cluster-aware client library to accomplish it. In this project, I explore making cluster-unaware clients compatible with highly available Redis.

High availability

What is high availability? First, let’s cover a few basic concepts.

Computer hardware and software are not perfect. Software can crash due to bugs or other problems. Hardware can fail due to physical problems that prevent it from operating, like a short circuit.

But in today’s world, software services that break sometimes just aren’t good enough. We could talk about the flight computers on an aircraft - these absolutely cannot fail, or the consequences for those on board could be deadly. Your car has a computer-controlled airbag that will save your life in the event of a crash.

Less serious scenarios exist too. Retail giants like Amazon.com are almost always online.

First, let’s define availability. Availability describes how reachable and usable a piece of software or service is, in that all incoming requests receive a response. If you’re able to open Amazon.com in your web browser and buy something, then Amazon.com is said to be “available”: it is online and able to fulfill the purpose for which it was created.

High availability, as the name suggests, describes a higher level of availability. That is, a system that is designed to continue operating despite certain supporting component failures. Generally speaking, this means running the software - and supporting components - across many separate computer servers and often in different geographical locations.

High availability represents a challenge in the software industry. Creating a program that runs across many computers has challenges and caveats that are completely different from those of programs that run in one place.

The CAP theorem is a brief description of the type of challenge faced. It was written to describe distributed data stores, but the idea applies to highly available systems in general:

The CAP theorem … states that any distributed data store can provide at most two of the following three guarantees:

Consistency - every read receives the most recent write or an error. Consistency means that all clients see the same data at the same time, no matter which node they connect to.

Availability - every request received by a non-failing node in the system must result in a response, without the guarantee that it contains the most recent version of the data.

Partition tolerance - the system continues to operate despite an arbitrary number of messages being dropped (or delayed) by the network between nodes.

- Wikipedia

In practice, network partitions are unavoidable, and systems must choose between consistency and availability.

But why does this matter? I’ll illustrate using an example:

Credit cards are a staple of modern society. In America, almost a third of all transactions are made using a credit card, according to the Federal Reserve. Thus, any disruption to the ability of consumers to use a credit card creates massive disruptions in everyday life, perhaps even to the point of being life-threatening. When issues like this happen, it makes headline news.

That makes it clear why availability is important: people have things to do, and they need your systems to do it.

But what about consistency?

Credit cards have a limit. When you make a transaction, the amount transacted counts towards your limit and the credit card provider may refuse the transaction if it would put you over the limit. This exists to protect you from accidentally spending too much money, and protects the credit card companies from lending you more money than they want to.

But what if you make two credit card transactions at the exact same time? Your credit card provider needs to be able to handle this situation. One transaction needs to be processed before the next, and the subsequent transaction needs to be processed using up-to-date data. For example, if your credit limit is $100 and you make two $75 transactions at once, one of them must not go through as it would leave you over your limit.

That all sounds rather obvious. But what about partition tolerance?

As mentioned above, highly available software is often run in different geographic locations. This is to protect against environmental threats to availability, such as natural disasters knocking down power lines or communications equipment. This stuff happens, and even if some servers are knocked offline, others exist elsewhere to handle the workload.

But, machines being located in separate locations create a risk for partitions: that is, when the two sets of machines cannot communicate with each other, but can still reach the outside world. To use the credit card transaction example again - for this example to work correctly, the entire fleet of transaction-handling servers needs a consistent view on how much money is in your account. This is impossible if they cannot communicate.

Redis

While high availability can be used to describe entire services, it can also describe subcomponents of a system.

Redis was described as a cache server above, but technically, it is a database server. And, as software systems are often backed by a database, it is often very important within a system. A system that needs Redis to function can only be as reliable as the Redis instances it communicates with.

As such, Redis has built-in support for high availability. The exact details can be read in Redis’s documentation, but the gist is as follows: a program called Sentinel monitors multiple distinct Redis instances, and orchestrates the instances so that if one fails, another instantly takes over.

This works great in practice, but using such a cluster requires some additional work. Specifically, Redis provides mechanisms and client libraries that are cluster-aware that integrates with Sentinel, so that from your application’s perspective, any turmoil in the Redis cluster is hidden away.

But not all software requiring Redis is written using this special library.

And that’s where redis-sentinel-proxy comes in.

Redis-sentinel-proxy

Redis-sentinel-proxy is a reverse proxy that exposes a standard - that is, non-cluster/Sentinel-aware - Redis connection, that standard Redis clients can use. My project is a fork of @enriclluelles’s original, and makes some reliability and usability improvements.

It works as follows:

Continuously query Sentinel for the Redis cluster’s status.
When a Redis client connects, connect the client to the active Redis instance.
When the Redis instance fails, close connections to clients and allow them to connect anew.

That’s it! Now, cluster-unaware Redis clients can take advantage of highly available Redis.

This was a fun project. If you find some use in it, let me know!

Source code: https://git.davepedu.com/dave/redisproxy

ubuntu kickstart / preseed builder

dave@davepedu.com (dave) — Sun, 07 Apr 2019 00:00:00 +0000

Kickstart and Preseed are tools for automatically installing Linux distributions such as Ubuntu. Installing new copies of Linux can be time-consuming, especially if you must do so from a ISO image. These tools automate the install process.

From the Ubuntu documentation:

Many system administrators would prefer to use an automated installation method to install Ubuntu on their machines. The preseeding feature of debian-installer answers this need.

Using Kickstart, a system administrator can create a single file containing the answers to all the questions that would normally be asked during a typical Ubuntu installation.

After creating a few ISO images with embedded Kickstart/Preseed configurations, I decided I needed faster iteration, and so I created a tool for generating Ubuntu Linux .iso files with embedded automated installers: Ubuntu Kickstart Builder.

The tool works in a fairly straightforward way:

Unpack Ubuntu ISO images in the tool’s data directory.
Open the web UI and edit the Kickstart and Preseed configuration files.
Optionally, upload additional files to be included in the ISO image.
Click Build, and your customized CD image will be downloaded.

The ISO image can then be written to physical media or booted in a VM.

While Kickstart and Preseed configurations can also be served over the network, embedding them directly in an installation image has some advantages. A self-contained ISO is easy to distribute, works in environments without network infrastructure, and guarantees that the installer always uses the intended configuration. This can be especially convenient for testing in virtual machines, provisioning isolated lab machines, or sharing reproducible install media.

single system image linux clustering

dave@davepedu.com (dave) — Thu, 10 Aug 2017 00:00:00 +0000

Recently, I came across the concept of SSI - “Single System Image” - Linux clustering. In short, Single System Image is an approach to clustering where many generic Linux computers, running a modified kernel, present themselves as one larger computer with access to the combined sum of all the underlying systems’ RAM, CPU cores, and other resources.

It was a useful technique because it decoupled your workloads from your hardware, which is the same goal as today’s cloud computing platforms. Your hardware can be generic, you just need enough total silicon for whatever your workload is, and this technology makes it all work together.

This approach, along with some other nifty Linux boot tricks, allows for some interesting ways to scale. These ideas are mostly superseded today by modern cloud computing or solutions like Kubernetes, but I think they are interesting enough to be worth discussing.

What I will attempt to describe in this article is a relatively unknown but easy to scale Linux clustering solution, some interesting ways to run it, and use cases for the technology. Understanding past job scheduling technologies like these make it easier to understand why modern cloud computing is designed how it is.

Kerrighed

Kerrighed is one notable open-source example of an SSI implementation. In order to discuss Kerrighed, we need to discuss SMP - Symmetric Multiprocessing - first.

In the context of Linux, SMP - Symmetric Multiprocessing - is the technology through which Linux gains its ability to run on systems with multiple CPU cores or discrete processors, treating them as equals to share workload, memory, and I/O devices. In other words: when writing a program to run on Linux, you generally need not worry about how many or what kind of CPU cores the system running your program will have and how you will distribute your program’s workload across them. This is because Linux is in charge of scheduling processes or threads - that is, assigning them - to CPUs, and moving them to another CPU later if need be. Additionally, the word “symmetric” in SMP comes from the fact that no one CPU is the “leader”; rather they all work together in a cooperative way. This is SMP.

While Linux’s included SMP implementation runs on just a single system, Kerrighed ships as modifications to the Linux kernel that, when applied, implements SMP across many systems instead of one. Kerrighed is integrated in such a way that Linux tools previously meant to inspect just your local system - such as ps to list processes - now operate across the entire cluster. That is, ps would show you all processes running across all nodes in the Kerrighed cluster.

It goes deeper than that, too. A feature of SMP is that it will move processes to different cores depending on where resources are available. Kerrighed does this too, but the moves can be to a completely different physical system.

Because Kerrighed is implemented at the kernel level, this is completely transparent to the process being scheduled or moved. This is how Kerrighed achieves binary compatibility with existing Linux programs. This is Kerrighed’s killer feature - existing programs need no modification to take advantage of a Kerrighed cluster.

Diskless boot

Having a cluster of computers is great, but the challenge becomes managing the cluster. This section will discuss a different kind of “single image” - booting a fleet of diskless systems from a single operating system image, with the disk image being accessed as-needed over the network.

While diskless booting is not inherently related to SSI clustering, it complements the model nicely by making it easy to provision large numbers of identical nodes.

I first heard about this technique in the following tutorial articles:

A quick explanation of what these articles describe: all of the operating system’s root filesystem is shared via an NFS server. The network that these machines run on is configured with TFTP, a technology for booting over the network. When a host boots, it contacts the TFTP server and downloads initial configuration and a Linux kernel. These configs instruct the host to mount the NFS share, and to boot the operating system from the NFS share.

What is notable about the above approaches is how they handle writes to the filesystem. We can’t write to the NFS share under this setup because it is shared across however many servers are in the cluster. Instead, a special filesystem called AUFS comes into play. AUFS is a “layered” filesystem, in that it allows you to combine multiple “layers” into one coherent filesystem.

It is worth noting that AUFS inspired the more modern OverlayFS, which is probably a better choice today.

In this case, the bottom-most, immutable layer is our NFS share. On top of that, is an in-memory tmpfs filesystem. AUFS is instructed that writes should go to the tmpfs, and reads come from the NFS share. Of course, we want to be able to read back things that we wrote, and that is exactly how AUFS works: since the tmpfs is the “top” layer, a file present on it will be read instead of a file with an identical path on the “bottom” layer. This allows our network-booted operating system to write whatever it needs without modifying the NFS share and affecting other hosts.

Of course, memory is a sparse and valuable system resource, and consuming it for filesystem storage is a bit of a waste. AUFS does not need to use tmpfs; formatting an actual disk to a typical format like ext4 or xfs and using it as the “top” layer is an option too.

Combining it all

Now that we understand our components, it should be clear how we will create our SSI cluster:

Servers can be added to the cluster simply by plugging them into the network, enabling TFTP netboot, and turning them on. Each system will boot into our pre-configured system image and join the Kerrighed cluster. Kerrighed, by doing its task of emulating SMP, will balance processes onto newly joined servers and make their resources available to the workload running on the cluster.

The cluster is simple to manage: add machines simply by plugging them in and turning them on. Remove machines by shutting them down and doing the opposite. Update machines by configuring TFTP to point to a different NFS share with an updated system image and reboot each host.

Additional components may be needed for the cluster to be useful - such as mounting a read/write NFS share for workload data - but this covers the basics.

Use cases

Single System Image clustering is most useful for workloads that benefit from large amounts of parallel compute or memory but are not designed to run across multiple machines. Because SSI systems aggregate memory across nodes, the cluster can appear to applications as a system with a much larger pool of RAM. This can be useful for simulations, data processing tasks, or other applications that need large datasets in memory. This overlaps with High Performance Computing (HPC), which are generally scientific and engineering workloads. These are a natural fit for SSI systems as many of these programs are already written to take advantage of multiple CPU cores, but not necessarily multiple machines. An SSI cluster can allow these programs to scale beyond the limits of a single system by providing access to additional CPU cores across the cluster without requiring the software to be rewritten.

SSI clusters can also provide a degree of resilience. If a node becomes overloaded or unavailable, processes may be migrated to other machines in the cluster, allowing workloads to continue running. While this does not replace dedicated high-availability systems, it can help improve utilization and reduce the impact of individual node failures.

Modern comparable software

It was mentioned earlier on that Single System Image clustering is comparable to modern cloud computing. However, I refer only to Kerrighed in this comparison; the diskless boot design described above is orthogonal to SSI clustering.

Popcorn Linux seems to be the only open source, single-image clustering Linux distribution that is reasonably up to date. However, Popcorn is not binary-compatible with existing Linux programs like Kerrighed is. Instead, Popcorn uses a special LLVM-based compiler to compile programs in a specific way so that they may run on a Popcorn cluster. However, Popcorn also offers support for mixing different Instruction Set Architectures within a cluster and even supports migrating running processes to hosts with different instruction sets! Popcorn’s special compiler is what makes this possible.

Kubernetes is another modern comparable. Unlike Single System Image clustering, Kubernetes requires software to be packaged in a specific way - Open Container Initiative Images aka Docker Images - and “migrates” workloads by terminating them, moving required resources such as storage disks, and starting them anew on a new host. While Kubernetes has nothing to do with Single System Image clustering, it is similar in that its primary role is to schedule work on clustered hosts.

And that is what all of these tools do - Kerrighed, Popcorn Linux, Kubernetes - you give it your hardware and your desired workloads, and it figures out where to run it, and how to keep it online to the best of its ability. Single system image computing has its roots all the way back in the 1980s, whereas Kubernetes is relatively new, but both serve the same purpose, then and now.

If this interest you, other systems that explored similar ideas from over the years, include MOSIX and Sprite, both of which experimented with process migration and distributed operating system design.

pfSense and ESXi setup on OVH's network

dave@davepedu.com (dave) — Mon, 12 Jun 2017 00:00:00 +0000

I host my projects on a dedicated server rented from OVH, I’m a huge VMware fan, so I’ve used their free product, the ESXi Hypervisor, to cram lots of virtual machines inside my one server. Since not all my virtual machines need internet-facing IPs and ESXi has only basic NAT capability, an easy solution here is running a pfSense install as a NAT router, as a virtual machine. OVH’s network has some unique setup required for this specific situation. Unfortunately, this is best documented in various blog posts across the internet.

Installation

You can grab the latest pfSense from here: https://www.pfsense.org/download/index.html. At the time of writing, the latest and pfSense version I’m using is 2.3.4, though any version later than 2.1 should work nicely. Thorough IPv6 support was introduced in 2.1, but if you don’t need that I suppose an earlier version would be fine as well. So grab an ISO and upload it to your ESXi machine.

Virtual machine creation

I used the 64-bit version of pfSense, so the “type” of virtual machine I used was Linux - Other Linux (64-bit). pfSense is based on BSD, so any similiar configuration should be fine. 512mb of ram allocated will be more than enough, but 256mb is doable. 4GB disk space allocated is fine as well. Next, other than adding a second network adapter, the default options that ESXi presents will work fine. One specialty for OVH - in your control panel, you must generate a MAC address for the address this host will be using - assign it to the public-facing network adapter. Also, create a virtual switch and attach it to the “private” network adapter. This switch is where your private network and virtual machines will live.

Installing pfSense to the virtual machine is pretty straightforward, but [check out this guide] (https://doc.pfsense.org/index.php/Installing_pfSense) if you need help.

OVH customization

After you finish pfSense’s command-line setup wizard, you won’t have internet connectivity. You need to add some routes specific to your server’s in the OVH network. If your server’s primary IP is A.B.C.D and the IP 1.2.3.4 will be assigned to this new host, run these commands in the shell:

route add -net A.B.C.254 1.2.3.4

route add default A.B.C.254

These commands will add routes needed to access your host’s gateway. You’ll have connectivity now, but need to make this fix permanent. Log into the UI, and in pfSense’s package repo the package Shellcmd is a convenient way to do this. Install it and add the two commands above as startup commands.

Wrap up

If you’ve enabled the DHCP server on your pfSense install you can now create privately-networked virtual machines with internet access via NAT, but with no exposure to the internet. Ports can be forwarded to public services and everything else is hidden behind a layer of NAT.

Next time: setting up IPv6 with TunnelBroker.net!

Editor’s Note: originally composed in 2014, but rotted in the drafts bin…

1Password on Linux

dave@davepedu.com (dave) — Tue, 28 Feb 2017 00:00:00 +0000

I’ve been a long time user of 1Password, a password vault by AgileBits. However, it only supports OS X and Windows; which is a problem for Linux desktop converts like myself. Their desktop application - a very nice app and also required for the browser extension - is only for Windows and OS X. AgileBits also offers 1Password Families which does support Linux, but you lose control over where your password data is stored.

In the past, 1Password had a browser-based user interface that could be served out of Dropbox or by opening the local file from your browser (called 1PasswordAnywhere, a single HTML file). However, browser security has “improved” and now block XHR requests to local filesystems and Dropbox discontinued the feature that allowed you to serve up this HTML file privately. So AgileBits was forced to sunset this feature.

Despite this announcement, 1PasswordAnywhere still seems to work! And as long as you can browse the file tree of 1Password’s data, this method should work for you.

I use Dropbox to sync my 1Password data, so I’ve opened a terminal and browsed to where the 1Password.agilekeychain directory is kept. For me, that’s ~/Dropbox/Apps/1Password/1Password.agilekeychain/. Within this directory is where 1PasswordAnywhere.html should be placed. I renamed mine to index.html.

Next, set up an web server to serve up this directory. If you’re running Mac or Linux you probably already have Python, and you can use its built-in minimal server:

$ pwd
/home/dave/Dropbox/Apps/1Password/1Password.agilekeychain
$ python -m SimpleHTTPServer
Serving HTTP on 0.0.0.0 port 8000 ...

Next, browse to http://127.0.0.1:8000/ and you’ll be presented with a working - but visually outdated - 1PasswordAnywhere instance! Of course, this isn’t the most secure - anyone on your network can access the web interface too! To be fair - your password is needed to decrypt the data - though it would be better to serve only on localhost. But I’ll leave this as homework for the reader.

Also, first post of 2017!

Update: I also stumbled across gonepass, a native Linux desktop app intended to replace the 1Password app. I’ve not tried it yet, so expect a follow-up article!

Update 2: I downloaded and compiled gonepass on my system. It was pretty easy and works great! Definitely a safer idea than the web-based version above.

upgrading my NAS, part 2: high availability

dave@davepedu.com (dave) — Thu, 21 Jul 2016 00:00:00 +0000

In a previous post, I described some of my early approaches to building a NAS for my home. In summary, I decided to use Linux software raid formatted with Linux’s built-in ext3 format. This setup worked, but left me with three main problems:

Volume size limitations: ext3 is capable of handling a maximum volume size of 16TB, when formatted with a 4kb block size. It is theoretically possible to use a larger block size and thus a larger maximum volume size. However, changing the block size requires reformatting the volume, which I could not do. Additionally, most hard drives natively use a 4kb block size. Matching filesystem block size to hard drive block size generally achieves best performance, for reasons such as avoiding a single write operation becoming multiple operations - read-modify-write.
High Availability: I enjoyed having a NAS so much, that I quickly began running additional servers that would mount the NAS’s storage volume and use it for whatever large storage was needed. I can’t recall which network filesystem I was using at the time, but it was one that required a re-mount after the server rebooted. NFS would make this less of a problem. Anyway - reboots aside - I wanted a NAS that would remain available even after one of the computers backing it failed, or needed to be rebooted for updates, for example.
Space limitations: physical space becomes a limiting factor for NAS side as well. Not many server chassis had physical space for many hard drives, outside of rackable enterprise-grade systems that I couldn’t afford. Networks allow multiple computers to communicate, so why would my filesystem be limited to one machine? I wanted a NAS that could be “scaled up” by adding more computer servers and hard drives to it.

GlusterFS

Enter GlusterFS - GlusterFS is a distributed network filesystem. GlusterFS provides data safety features similar to RAID, but offered in a manner that spans multiple computers instead of multiple hard drives.

The general way in which GlusterFS functions is simple: you format your hard drives with whatever filesystem you prefer - I used ZFS - and GlusterFS creates a virtual filesystem on top of them. GlusterFS calls each sub-volume a “brick”, and while each brick only spans one server, the virtual GlusterFS filesystem can incorporate multiple bricks from multiple servers.

What’s great about GlusterFS’s approach to storage is that it single-handedly solves each of the 3 issues above:

Volume size limitations: GlusterFS uses its own type of filesystem with astronomically high volume size limitations.
High Availability: with a “replicated” GlusterFS volume, multiple copies of each file are stored on separate bricks. Since separate bricks are on separate servers, the file remains accessible even if one server or hard drive fails.
Space limitations: since GlusterFS volumes can span across multiple servers, space becomes nearly limitless. You can always plug more servers into the cluster.

Design

I started with two servers, but later moved to four. Each could hold up to 8 hard drives. A common build at the time was a mini-NAS, with a case by U-NAS, the NSC-800. For a motherboard/cpu combo, I chose AsRock Rack. I started with their Mini-ITX form factor server motherboard, the C2750D4I, and later upgraded to the C3758D4I. Both feature 8 cores, low power consumption, and small physical size.

As mentioned above, I used ZFS to manage my server’s hard drives. Initially, I had 16 drives spread across two servers. I divided these into 4 ZFS mirrors - like RAID 1 - of 2 drives each per server. Then, each ZFS mirror was replicated with GlusterFS to a corresponding ZFS mirror on the other server. This worked well, but I later moved to two ZFS RAIDZ pools per server, made from 4 drives each. This reduced the number of bricks which made GlusterFS simpler to manage.

Results

Overall, this was a fairly reliable and reasonably performant NAS. Generally speaking, I was bottlenecked by drive speed as I used “green” hard drives that put out less noise and heat and consumed less power at the expense of some performance. Sequential access was great but one area the NAS really suffered in was random access. Listing large directories was noticeably slow as well. Despite these issues, this NAS went on to serve me for many years!

If you run a similar setup, let me know!

building a distributed / redundant data storage backbone - part 1 of x

dave@davepedu.com (dave) — Mon, 23 Nov 2015 00:00:00 +0000

I’m not a fan of losing data, so I always have some sort of redundant storage at my disposal. At present I possess over 10TB of irreplaceable data, code, photographs, etc, (and much more replaceable data - though at an inconvenience) and housing that data is no easy task. Be it for backups, media storage, or even temp space, having a huge pool of “safe” storage is always a useful tool.

A bit of history first - we’ll get into the cutting edge stuff, with more technical detail, in part 2.

My first adventure into this involved a number of SATA hard drives attached to a Mac Mini via USB 3.0. A Linux VM assembled these drives into a RAID 5 array and served their contents over SMB (‘server message block’ - windows sharing). Unfortunately this was in the early days of USB 3.0 so the pass-through adapter to the Linux virtual machine only supported USB 2.0 speeds. So this array maxed out around 15MB/s constant write. Not great, but at least it’s safe.

RAID

Anyway, why RAID? RAID is a immensely valuable idea that allows the use of many independent hard or solid state drives for a single logical task. RAID can be used for speed gains beyond what an individual drive can handle or failure proof storage where even the death of one or more drives causes no data loss - and in the best case, no interruption to services using the data. There are many different kinds of traditional RAID, with the most commonly used being:

RAID 0 - performance gains by spreading load across multiple drives
RAID 1 - stability gains by mirroring data on two or more drives
RAID 5 - stability gains by smart use of striping & parity

These (and other) levels of RAID can also be combined for a mix of the effects a single RAID level brings. For example, RAID 10 (1+0, striped set of mirrors) or is fairly common.

Traditional RAID - or “hardware” RAID - typically involves a hardware expansion card for the dedicated purpose of handling RAID computations and drives. The RAID card would have a builtin processor, memory, and SATA ports to which the involved disks would be attached to. Up until recently, this sort of configuration was necessary for top performance - software RAID has been around for awhile, but historically there was a performance gap. A risk of hardware RAID has always been the RAID card itself. RAID cards typically have an onboard memory module a battery to keep the memory “alive”. If any of these parts cease to function - or any other part of the card - it’s possible that all data on the RAID array can be lost! In my opinion, this isn’t a viable option. If you have your data on a single drive or a RAID array controlled by one RAID card, either there is a single piece of hardware that can die and cause data loss. Not good.

Software RAID addresses this issue. It may have originally been created as a cheaper (and slower) alternative to hardware RAID, but I believe nowadays software options are just as valid and performant has hardware RAID is. Software RAID, in a nutshell, is, technically speaking, the same as hardware RAID, but with the computation and memory required for operation handled by a system’s main CPU and memory as opposed to a dedicated piece of hardware. Done right, these components can die or be swapped out with no fear of data loss.

In my opinion the most notable advances in software RAID tech are as follows:

Mdadm - often referred to as ‘Linux software RAID’ - was introduced in 2001 and quickly became the de facto software RAID tool.
ZFS - introduced in 2005 (and from my perspective, catching on in the early 2010s) was an entirely new beast.

While ZFS is impressively (and I’m using it today!), let’s examine mdadm first so we can see how ZFS differs. Mdadm offers block-level RAID. In other words, it doesn’t care which (and you have to use) filesystem is “on top” of it. On Linux, mdadm consumes block devices and provides a virtual block device. So if the physical drives on your system are /dev/sda and /dev/sdb, mdadm can consume these and provide /dev/md0. From there, md0 can be treated just like a plain block device - create partition(s) and format them with a filesystem of choice.

My first and second storage arrays used Mdadm. It was an easy choice as tools that work with native block devices work with mdadm virtual block devices. I could partition the virtual device with common disk management tools like fdisk and use common filesystems like ext.

And I did exactly that - my first array comprised of ext4 on top of mdadm. And it worked! …until I needed an array bigger than 16TB. Because of how ext4 was programmed and integrated into Linux operating systems it is incapable of handling filesystems greater in size than 16TB when using 4k blocks. Considering it originated in the early 90s and focused on backwards compatibility, there wasn’t a workaround. Back to the drawing board.

What’s nice about Linux is that filesystems are ubiquitous - if the system can mount a filesystem, all programs on the system that depend on file access and use it and all features it expects a filesystem to support. A beauty of modern system design. An up-and-coming filesystem around the time I needed to expand was btrfs. With a maximum supported filesystem size of 16EB (exabytes!), or a million times larger than ext4’s limits, I was sure I’d never outgrow it. So I live-converted my array to btrfs and it was full steam ahead from there!

Great! So in early 2011 I had a storage backbone capable of supporting an astronomical amount of data, the flexibility to expand over time, and safety of software RAID. What more could be desired? Well, the world of computing constantly improves itself so I wasn’t happy sticking with a single methodology. But more on that, and ZFS especially, in part 2!