Recently, I came across the concept of SSI - “Single System Image” - Linux clustering. In short, Single System Image is an approach to clustering where many generic Linux computers, running a modified kernel, present themselves as one larger computer with access to the combined sum of all the underlying systems’ RAM, CPU cores, and other resources.

It was a useful technique because it decoupled your workloads from your hardware, which is the same goal as today’s cloud computing platforms. Your hardware can be generic, you just need enough total silicon for whatever your workload is, and this technology makes it all work together.

This approach, along with some other nifty Linux boot tricks, allows for some interesting ways to scale. These ideas are mostly superseded today by modern cloud computing or solutions like Kubernetes, but I think they are interesting enough to be worth discussing.

What I will attempt to describe in this article is a relatively unknown but easy to scale Linux clustering solution, some interesting ways to run it, and use cases for the technology. Understanding past job scheduling technologies like these make it easier to understand why modern cloud computing is designed how it is.

Kerrighed⌗

Kerrighed is one notable open-source example of an SSI implementation. In order to discuss Kerrighed, we need to discuss SMP - Symmetric Multiprocessing - first.

In the context of Linux, SMP - Symmetric Multiprocessing - is the technology through which Linux gains its ability to run on systems with multiple CPU cores or discrete processors, treating them as equals to share workload, memory, and I/O devices. In other words: when writing a program to run on Linux, you generally need not worry about how many or what kind of CPU cores the system running your program will have and how you will distribute your program’s workload across them. This is because Linux is in charge of scheduling processes or threads - that is, assigning them - to CPUs, and moving them to another CPU later if need be. Additionally, the word “symmetric” in SMP comes from the fact that no one CPU is the “leader”; rather they all work together in a cooperative way. This is SMP.

While Linux’s included SMP implementation runs on just a single system, Kerrighed ships as modifications to the Linux kernel that, when applied, implements SMP across many systems instead of one. Kerrighed is integrated in such a way that Linux tools previously meant to inspect just your local system - such as ps to list processes - now operate across the entire cluster. That is, ps would show you all processes running across all nodes in the Kerrighed cluster.

It goes deeper than that, too. A feature of SMP is that it will move processes to different cores depending on where resources are available. Kerrighed does this too, but the moves can be to a completely different physical system.

Because Kerrighed is implemented at the kernel level, this is completely transparent to the process being scheduled or moved. This is how Kerrighed achieves binary compatibility with existing Linux programs. This is Kerrighed’s killer feature - existing programs need no modification to take advantage of a Kerrighed cluster.

Diskless boot⌗

Having a cluster of computers is great, but the challenge becomes managing the cluster. This section will discuss a different kind of “single image” - booting a fleet of diskless systems from a single operating system image, with the disk image being accessed as-needed over the network.

While diskless booting is not inherently related to SSI clustering, it complements the model nicely by making it easy to provision large numbers of identical nodes.

I first heard about this technique in the following tutorial articles:

A quick explanation of what these articles describe: all of the operating system’s root filesystem is shared via an NFS server. The network that these machines run on is configured with TFTP, a technology for booting over the network. When a host boots, it contacts the TFTP server and downloads initial configuration and a Linux kernel. These configs instruct the host to mount the NFS share, and to boot the operating system from the NFS share.

What is notable about the above approaches is how they handle writes to the filesystem. We can’t write to the NFS share under this setup because it is shared across however many servers are in the cluster. Instead, a special filesystem called AUFS comes into play. AUFS is a “layered” filesystem, in that it allows you to combine multiple “layers” into one coherent filesystem.

It is worth noting that AUFS inspired the more modern OverlayFS, which is probably a better choice today.

In this case, the bottom-most, immutable layer is our NFS share. On top of that, is an in-memory tmpfs filesystem. AUFS is instructed that writes should go to the tmpfs, and reads come from the NFS share. Of course, we want to be able to read back things that we wrote, and that is exactly how AUFS works: since the tmpfs is the “top” layer, a file present on it will be read instead of a file with an identical path on the “bottom” layer. This allows our network-booted operating system to write whatever it needs without modifying the NFS share and affecting other hosts.

Of course, memory is a sparse and valuable system resource, and consuming it for filesystem storage is a bit of a waste. AUFS does not need to use tmpfs; formatting an actual disk to a typical format like ext4 or xfs and using it as the “top” layer is an option too.

Combining it all⌗

Now that we understand our components, it should be clear how we will create our SSI cluster:

Servers can be added to the cluster simply by plugging them into the network, enabling TFTP netboot, and turning them on. Each system will boot into our pre-configured system image and join the Kerrighed cluster. Kerrighed, by doing its task of emulating SMP, will balance processes onto newly joined servers and make their resources available to the workload running on the cluster.

The cluster is simple to manage: add machines simply by plugging them in and turning them on. Remove machines by shutting them down and doing the opposite. Update machines by configuring TFTP to point to a different NFS share with an updated system image and reboot each host.

Additional components may be needed for the cluster to be useful - such as mounting a read/write NFS share for workload data - but this covers the basics.

Use cases⌗

Single System Image clustering is most useful for workloads that benefit from large amounts of parallel compute or memory but are not designed to run across multiple machines. Because SSI systems aggregate memory across nodes, the cluster can appear to applications as a system with a much larger pool of RAM. This can be useful for simulations, data processing tasks, or other applications that need large datasets in memory. This overlaps with High Performance Computing (HPC), which are generally scientific and engineering workloads. These are a natural fit for SSI systems as many of these programs are already written to take advantage of multiple CPU cores, but not necessarily multiple machines. An SSI cluster can allow these programs to scale beyond the limits of a single system by providing access to additional CPU cores across the cluster without requiring the software to be rewritten.

SSI clusters can also provide a degree of resilience. If a node becomes overloaded or unavailable, processes may be migrated to other machines in the cluster, allowing workloads to continue running. While this does not replace dedicated high-availability systems, it can help improve utilization and reduce the impact of individual node failures.

Modern comparable software⌗

It was mentioned earlier on that Single System Image clustering is comparable to modern cloud computing. However, I refer only to Kerrighed in this comparison; the diskless boot design described above is orthogonal to SSI clustering.

Popcorn Linux seems to be the only open source, single-image clustering Linux distribution that is reasonably up to date. However, Popcorn is not binary-compatible with existing Linux programs like Kerrighed is. Instead, Popcorn uses a special LLVM-based compiler to compile programs in a specific way so that they may run on a Popcorn cluster. However, Popcorn also offers support for mixing different Instruction Set Architectures within a cluster and even supports migrating running processes to hosts with different instruction sets! Popcorn’s special compiler is what makes this possible.

Kubernetes is another modern comparable. Unlike Single System Image clustering, Kubernetes requires software to be packaged in a specific way - Open Container Initiative Images aka Docker Images - and “migrates” workloads by terminating them, moving required resources such as storage disks, and starting them anew on a new host. While Kubernetes has nothing to do with Single System Image clustering, it is similar in that its primary role is to schedule work on clustered hosts.

And that is what all of these tools do - Kerrighed, Popcorn Linux, Kubernetes - you give it your hardware and your desired workloads, and it figures out where to run it, and how to keep it online to the best of its ability. Single system image computing has its roots all the way back in the 1980s, whereas Kubernetes is relatively new, but both serve the same purpose, then and now.

If this interest you, other systems that explored similar ideas from over the years, include MOSIX and Sprite, both of which experimented with process migration and distributed operating system design.