Replication Lag: Read-Your-Writes Consistency
Introduction
For ready heavy workloads, one of the most common database patterns to increase your application’s ability to handle many concurrent reads is to move to a Single-Leader replication pattern. But with every distributed system, you leave yourself open to numerous consistency issues (you can read about some of them here). This post will cover how to guarantee read-your-writes consistency in a single-leader database architecture.
What is Single-Leader Replication?
This architecture is relatively straight forward. There is one database instance designated as the Leader, and there are other instances that are designated as followers (or replicas). All writes made by clients are sent to the leader and the leader propagates these changes to the replicas. Reads made by the client can be directed to either the leader or any of the replicas.
Sidenote
Single-Leader replication is also known as master-slave replication and this is the terminology used by the PostgreSQL community. For that reason, you might see us use the term master interchangeably with leader. But in general, we will refer to the followers as replicas since in the context of databases, I think replica is a more descriptive term than follower.
Read your writes consistency
Imagine you are using an application like google slides and you would like to make some last minute changes to some slides right before the presentation. You make your changes right in time to start your presentation and receive a success response and are redirected to the view page of the presentation. When you begin presenting, you realize that the changes you made are not reflected. What happened here?
What happened was that the writes that the user made to the database were in fact successful, but the read query that happened immediately after was routed to a read replica that did not have that version of the data yet. So from the client's perspective, it seems like the write they made didn't apply. You expected to “read your own writes”, but in a distributed system this type of consistency is not necessarily guaranteed.
Since these two requests occur immediately after one another, all it takes is a few hundred milliseconds of replica lag to face this issue.
Solution
Force your application to read from master for a period of time, after committing a write. We call this sticking to master, and ideally you want the time period to be some period greater than replica lag at any point in time.
The most common solution is to make this time period something like 10 seconds, and just hope that replication lag never gets that bad (for many use cases this is probably fine). But for more specific use cases you may want to dynamically adjust the time period based on what the current replica lag is, and adding some appropriate buffer.
To make this a little easier, you can use this handy gem Makara for ruby applications. It provides some examples of basic usage, and also gives us some specific methods stick_to_master! and without_sticking to give us more fine-grained control over whether we want to read from a master or a replica.
Drawbacks
This simple solution works quite well for many use cases, but it has its drawbacks. Let’s look at the following example.
A user might add a product to their shopping cart on an e-commerce website, and then be redirected to the previous category page which shows a list of products in that category. Adding the item to the shopping_cart will add a record to the user_shopping_cart table but the reads immediately after will be reading from the products table. Since we automatically stick to master after writing to the database, the subsequent read to the products table will also be routed to master. In this case, you are unnecessarily reading from the master database when it is sufficient to read from a read replica. During high traffic periods like Black Friday, you don’t want to be routing all traffic to your master database instance if you can help it.
In this case the user is writing to a specific table, but we are forcing the user to stick to master for subsequent reads to unrelated tables. In this case it might make sense to determine whether you should read from the leader or replica based on what table the client is attempting to read.
A simple way of implementing this is to have a whitelist of tables where a read will be always routed to a read replica. In the example above, we could add the products table to the whitelist so all read requests to that table are routed to the replica. For more complex use cases we may even consider keeping track of which tables the user has written too, and only route to master if the user is trying to read a table the user has recently changed.
For brevity, I will omit showing an example of implementing either of these solutions in an application. But you should have a good idea of how to implement this as a middleware of sorts using a library like Makara.
TLDR
Problem: Users may experience inconsistent UX if there's replica lag and subsequent reads are routed to a read replica.
Solution: Force reads from the master for a period after a write. Use static or dynamic buffers based on replica lag.
Tool: Use Makara’s
stick_to_master!orwithout_stickingfor fine-grained control.
