Revisiting the Outbox Pattern - Gunnar Morling
Visit Website
Gunnar Morling
Oct 31, 2024
Summary
This blog post revisits the Outbox Pattern, a common solution for data exchange between microservices. It discusses the pattern's implementation, considerations, criticisms, and alternatives, evaluating its relevance in 2024.
Content Sections
Recap: What’s The Outbox Pattern?
The outbox pattern provides a solution which doesn’t require distributed transactions. As part of its local database transaction, the appointment service inserts the notification message which it wants to send into a table in that database. A separate process, called the outbox relay, picks up that message from this outbox table and sends it to Kafka. This happens asynchronously and can be retried if needed, without impacting the clients of the appointment service in any way.
Implementation Considerations
Key implementation details of the Outbox Pattern include choosing between polling vs. log-based CDC for message retrieval, designing the outbox table structure, housekeeping, and leveraging database-specific features like pg_logical_emit_message() in Postgres. It also covers format considerations and the management of backfills.
Criticisms of the Outbox pattern
The criticisms of the Outbox Pattern include concerns about database overhead, complexity, and latency. The database overhead concern is about the additional load put onto a service’s database by using it for storing and emitting outbox messages. The complexity argument includes the overall number of moving parts of the solution as well as their interactions.
Alternatives to the Outbox Pattern
Alternatives to the Outbox Pattern include Dapr, Read-yourself, Stream Processing on "Raw" Change Event Streams, 2PC, and Durable Execution. Each alternative has its own trade-offs in terms of consistency, availability, queryability, developer experience, and operational complexity.
Summary
The outbox pattern continues to deserve a very central spot in the toolbox of development teams building microservices architectures. The outbox pattern puts a strong focus on consistency and reliability, availability and letting developers benefit from all the capabilities of their well-known datastore.