When working on backend systems, we often follow a common pattern for handling incoming data. A server receives data from a message broker or an API, processes it according to the business rules, and inserts or updates it in the database. It then sends the processed data to the next server through a message broker or an API. This pattern is called dual write, sometimes also called multi-write or sync-write.

Problem⌗

This pattern seems pretty intuitive, so what is the problem? Its operations can fail, often leaving data inconsistent across systems. Tracking down the cause can require hair-pulling debug sessions, and even then we might not find it.

The dual write failing: the row is saved to the database but the receiver is never notified

The problem arises when we successfully save the processed data to the database but fail to send it to the message broker or the next server. Our system has been updated, but the downstream systems are unaware of the change. The data is now inconsistent across systems.

There are several possible causes. A network partition might make the receiver unreachable. The receiver might be down because of an error or maintenance. It could even be a human error, such as sending data with the wrong schema to the wrong API. Whatever the cause, the data no longer matches, and we have to fix it.

Examples of dual write in e-commerce⌗

E-commerce architectures are often split into services with defined boundaries to make it easier to scale both the system and the engineering teams. Key services include a payment service or gateway for customer payments and checkout, an order management system for managing and tracking orders, an inventory management system for sellers to manage products, stock, and storage, and a search service that lets buyers browse, search, and filter products based on their queries, interests, and history.

Checkout as a dual write: the payment is saved but the notification to the OMS fails

During checkout, once the customer has paid successfully, the payment service must tell the order management system that the order is paid so it can move to the next step. If that notification fails, the order gets stuck. The customer waits, eventually realizes something is wrong, and has to contact support. That is a poor user experience.

Listing creation as a dual write: the listing is saved but the search service is never notified, so the buyer cannot find it

After a seller creates a product listing, the inventory management system notifies the search service. The search service creates an entry in its database and indexes it to optimize search performance and make the listing searchable. If the notification fails, buyers cannot find the listing, and the seller loses potential customers and income. This can also damage the platform’s reputation among sellers.

Some Solutions That Do Not Work⌗

Now that we know what dual write is and what problems it can cause, let’s look at some possible solutions and why they do not solve the problem.

Revert the data⌗

Can we revert the data? Yes, but the revert operation can also fail. What do we do then? We also need to store the previous state before reverting.

What if we only commit to the database after successfully notifying the downstream services? We would not need to store the previous state, but the commit itself can still fail.

In my humble opinion, this approach gets messy really quickly.

Save after sending⌗

What if we only write to the database after successfully sending the data to downstream services? This is similar to committing after sending, as discussed above. The database write is not guaranteed to succeed, so simply changing the order of operations solves nothing.

Retry⌗

If an operation fails, retrying seems like a good enough solution. But how many times should we retry, and how long should we wait between attempts?

If the receiver is down for a long time, say more than five minutes, most retry configurations will have given up by then. The data will still be inconsistent after they give up. We need a better solution.

What if we extend the retry period using a technique such as exponential backoff? That could work, but what if our service crashes while retrying? When it comes back up, it will have lost the context of what it was trying to do. We need to store some state for this approach to work. We are getting closer.

Some Solutions That Do Work⌗

Here are a few patterns for handling dual write properly. This post is already quite long, so I will only introduce their names and basic ideas. I’ve started exploring them in more detail in Dual write solutions:

  • Transactional Outbox Pattern: Ensure atomicity by updating the business data and storing the message in an outbox table within a single database transaction. A separate process sends the messages later.
  • Listen to Yourself Pattern: Publish an event representing the intended data change, then consume it to update the database.
  • Change Data Capture Pattern: Use the database’s durability guarantee by reading its transaction log to capture changes and send them as messages.

Read More⌗