highly available redis with redis-sentinel-proxy
Redis is a widely used and extremely fast cache server. Redis supports high availability out-of-the-box, but requires a cluster-aware client library to accomplish it. In this project, I explore making cluster-unaware clients compatible with highly available Redis.
High availability⌗
What is high availability? First, let’s cover a few basic concepts.
Computer hardware and software are not perfect. Software can crash due to bugs or other problems. Hardware can fail due to physical problems that prevent it from operating, like a short circuit.
But in today’s world, software services that break sometimes just aren’t good enough. We could talk about the flight computers on an aircraft - these absolutely cannot fail, or the consequences for those on board could be deadly. Your car has a computer-controlled airbag that will save your life in the event of a crash.
Less serious scenarios exist too. Retail giants like Amazon.com are almost always online.
First, let’s define availability. Availability describes how reachable and usable a piece of software or service is, in that all incoming requests receive a response. If you’re able to open Amazon.com in your web browser and buy something, then Amazon.com is said to be “available”: it is online and able to fulfill the purpose for which it was created.
High availability, as the name suggests, describes a higher level of availability. That is, a system that is designed to continue operating despite certain supporting component failures. Generally speaking, this means running the software - and supporting components - across many separate computer servers and often in different geographical locations.
High availability represents a challenge in the software industry. Creating a program that runs across many computers has challenges and caveats that are completely different from those of programs that run in one place.
The CAP theorem is a brief description of the type of challenge faced. It was written to describe distributed data stores, but the idea applies to highly available systems in general:
The CAP theorem … states that any distributed data store can provide at most two of the following three guarantees:
Consistency - every read receives the most recent write or an error. Consistency means that all clients see the same data at the same time, no matter which node they connect to.
Availability - every request received by a non-failing node in the system must result in a response, without the guarantee that it contains the most recent version of the data.
Partition tolerance - the system continues to operate despite an arbitrary number of messages being dropped (or delayed) by the network between nodes.
In practice, network partitions are unavoidable, and systems must choose between consistency and availability.
But why does this matter? I’ll illustrate using an example:
Credit cards are a staple of modern society. In America, almost a third of all transactions are made using a credit card, according to the Federal Reserve. Thus, any disruption to the ability of consumers to use a credit card creates massive disruptions in everyday life, perhaps even to the point of being life-threatening. When issues like this happen, it makes headline news.
That makes it clear why availability is important: people have things to do, and they need your systems to do it.
But what about consistency?
Credit cards have a limit. When you make a transaction, the amount transacted counts towards your limit and the credit card provider may refuse the transaction if it would put you over the limit. This exists to protect you from accidentally spending too much money, and protects the credit card companies from lending you more money than they want to.
But what if you make two credit card transactions at the exact same time? Your credit card provider needs to be able to handle this situation. One transaction needs to be processed before the next, and the subsequent transaction needs to be processed using up-to-date data. For example, if your credit limit is $100 and you make two $75 transactions at once, one of them must not go through as it would leave you over your limit.
That all sounds rather obvious. But what about partition tolerance?
As mentioned above, highly available software is often run in different geographic locations. This is to protect against environmental threats to availability, such as natural disasters knocking down power lines or communications equipment. This stuff happens, and even if some servers are knocked offline, others exist elsewhere to handle the workload.
But, machines being located in separate locations create a risk for partitions: that is, when the two sets of machines cannot communicate with each other, but can still reach the outside world. To use the credit card transaction example again - for this example to work correctly, the entire fleet of transaction-handling servers needs a consistent view on how much money is in your account. This is impossible if they cannot communicate.
Redis⌗
While high availability can be used to describe entire services, it can also describe subcomponents of a system.
Redis was described as a cache server above, but technically, it is a database server. And, as software systems are often backed by a database, it is often very important within a system. A system that needs Redis to function can only be as reliable as the Redis instances it communicates with.
As such, Redis has built-in support for high availability. The exact details can be read in Redis’s documentation, but the gist is as follows: a program called Sentinel monitors multiple distinct Redis instances, and orchestrates the instances so that if one fails, another instantly takes over.
This works great in practice, but using such a cluster requires some additional work. Specifically, Redis provides mechanisms and client libraries that are cluster-aware that integrates with Sentinel, so that from your application’s perspective, any turmoil in the Redis cluster is hidden away.
But not all software requiring Redis is written using this special library.
And that’s where redis-sentinel-proxy comes in.
Redis-sentinel-proxy⌗
Redis-sentinel-proxy is a reverse proxy that exposes a standard - that is, non-cluster/Sentinel-aware - Redis connection, that standard Redis clients can use. My project is a fork of @enriclluelles’s original, and makes some reliability and usability improvements.
It works as follows:
- Continuously query Sentinel for the Redis cluster’s status.
- When a Redis client connects, connect the client to the active Redis instance.
- When the Redis instance fails, close connections to clients and allow them to connect anew.
That’s it! Now, cluster-unaware Redis clients can take advantage of highly available Redis.
This was a fun project. If you find some use in it, let me know!
Source code: https://git.davepedu.com/dave/redisproxy