Software Architecture
Distributed Transactions and Consistency: 2PC, Sagas, Outbox and CAP
Advanced11 min readUpdated
Once state is spread across services and databases, the single-database transaction is gone and something must replace it. This guide explains what CAP and PACELC actually constrain, why two-phase commit is rarely the answer outside a single trusted stack, how sagas with compensation really behave, and how the outbox and idempotency turn at-least-once delivery into correct business outcomes.
What you will learn
- State precisely what CAP and PACELC do and do not say about a system
- Explain the two-phase commit protocol and the failure modes that make it block
- Design a saga with compensating actions and choose choreography or orchestration
- Implement the transactional outbox so that state changes and messages cannot diverge
- Reason about at-least-once delivery, idempotency and the exactly-once illusion
Inside one database, consistency is something you get by default: begin, do several things, commit, and either all of it happened or none of it did. That guarantee has shaped how most developers think about correctness, and it disappears the moment a business operation spans two databases or two services. Nothing replaces it exactly. What exists instead is a set of protocols and patterns, each of which gives up something specific, and the engineering work is choosing which loss you can live with.
This guide starts with what the theory genuinely constrains, because CAP in particular is more often cited than understood. It then covers the two families of answer: atomic commitment protocols that try to preserve the old guarantee, and saga-style approaches that abandon isolation and rebuild correctness from compensations and idempotency. The pattern catalogue in microservice design patterns introduces these by name; here they are examined for how they actually fail.
ACID, BASE and what isolation buys
ACID describes the local transaction: atomicity (all or nothing), consistency (constraints hold at commit), isolation (concurrent transactions do not observe each other's intermediate states) and durability (committed data survives a crash).
Isolation is the property people miss most when it goes, and it is worth remembering that even locally it is usually weaker than "serializable". Most databases default to read committed, where a transaction can see rows another transaction committed while it was running, so non-repeatable reads and phantoms are already possible. Understanding the anomalies your isolation level permits — dirty read, non-repeatable read, phantom, write skew — matters before you distribute anything, because distribution only widens those windows.
BASE is the deliberate contrast: basically available, soft state, eventually consistent. The system stays available and accepts writes, replicas may disagree for a while, and given no further updates they converge. This is not a lower standard of engineering; it is a different contract, and it requires more design work rather than less, because every consumer must be written to tolerate a value that is briefly out of date.
The business question that decides between them is not "do we need consistency" — everyone answers yes — but "what happens if these two facts disagree for two seconds?" Two accounts in a ledger disagreeing is unacceptable. A product's review count lagging is not. Most real systems need strong consistency in a small core and tolerate eventual consistency everywhere else, which is why aggregates matter: the aggregate is exactly the boundary within which you insist on ACID.
CAP and PACELC, stated carefully
CAP says that a distributed system cannot simultaneously guarantee consistency (every read sees the latest write), availability (every request to a non-failing node receives a response) and partition tolerance (the system continues despite messages between nodes being lost) — and since partitions happen whether you plan for them or not, the real choice is what to do during a partition: refuse requests to preserve consistency (CP), or answer from possibly stale state to preserve availability (AP).
Three clarifications remove most of the confusion:
- The choice applies during a partition only. A healthy system can be both consistent and available, and usually is.
- It is a choice per operation and per data set, not per system. The same platform can refuse a balance transfer during a partition while continuing to serve a product catalogue.
- CAP's "consistency" means linearizability — a specific, strong guarantee. It is not the C in ACID, and the accidental overlap of letters has caused years of confused arguments.
PACELC extends this usefully: if there is a Partition, choose Availability or Consistency; Else, choose Latency or Consistency. The second clause covers the 99.9 % of time when the network is fine, and it names the trade-off you make far more often. Acknowledging a write after one replica has it is fast and risks losing it; waiting for a quorum is slower and safer. Every replicated store exposes this as a tunable — write concern, consistency level, acknowledgements required — and setting it deliberately per operation is more consequential than any label attached to the database.
Two-phase commit and XA
The classical answer to atomicity across resources is two-phase commit (2PC), coordinated by a transaction manager.
Phase 1 — prepare
The coordinator asks every participant whether it can commit. Each does the work, writes it durably to its log, acquires the locks it needs, and votes yes or no. A yes vote is a binding promise: the participant must be able to commit later, so it holds its locks.Phase 2 — commit or abort
If all voted yes, the coordinator writes its decision to its own log and tells everyone to commit. If any voted no, it tells everyone to roll back. Participants apply the decision and release locks.
The protocol is correct, and its failure mode is that it blocks. If the coordinator crashes after collecting yes votes but before broadcasting the decision, participants are stuck: they have promised to be able to commit, so they cannot unilaterally abort, and they do not know the decision, so they cannot commit. Locks are held until the coordinator recovers and reads its log. In-doubt transactions of this kind are a familiar operational problem, and resolving them by hand risks breaking atomicity for real.
The other costs are structural. Every participant must implement the protocol, which excludes most modern HTTP services and many data stores. Availability multiplies downward — the transaction succeeds only if every participant is up. Latency includes at least two round trips plus durable log writes at each participant. And throughput suffers because locks are held across the whole protocol, so hot rows serialise the entire system.
XA is the standard interface for this, implemented on the JVM through JTA with an XAResource per participating resource. It remains genuinely appropriate in a narrow band: a single application server coordinating a database and a JMS provider, both local, both under one operations team, with modest throughput — the case covered in transactions and JTA. It is the wrong tool across independently deployed services, and "distributed transactions do not work over microservices" is shorthand for exactly this.
Three-phase commit and consensus-based commit (using Paxos or Raft to replicate the coordinator's decision) address the blocking problem at the cost of more round trips and considerably more machinery. Systems that need cross-shard atomicity — distributed SQL databases — implement something in this family internally, which is a good reason to let a database do it rather than building it yourself.
Sagas
A saga replaces one distributed transaction with a sequence of local transactions, each committed independently, plus a compensating action for each step that can undo its effect if a later step fails.
The trade is explicit and worth stating plainly: you keep atomicity in the weak sense that the saga either completes or is compensated, and you lose isolation entirely. Intermediate states are visible. Between reserving stock and capturing payment, the stock genuinely is reserved and other transactions can see it. Every saga design must therefore answer what other actors observe mid-flight, and either make that acceptable or hide it behind a status.
Compensation is not rollback. It is a new business action with its own semantics: a refund is not the erasure of a payment, and a cancellation email may already have been read. Some steps cannot be compensated at all — an email sent, a file transmitted to a clearing house — which is why the standard advice is to order steps so that compensatable steps come first, then a pivot step that decides the outcome, then retriable steps that must eventually succeed and are never undone.
Choreography
Each service reacts to events and emits its own. No coordinator exists.
OrderService → OrderCreated
InventoryService → (on OrderCreated) reserve stock → StockReserved | StockUnavailable
PaymentService → (on StockReserved) charge → PaymentCaptured | PaymentDeclined
InventoryService → (on PaymentDeclined) release stock [compensation]
OrderService → (on PaymentCaptured) confirm | (on PaymentDeclined) cancel
Coupling stays low and each team owns its own step. The costs appear as the flow grows: no single place describes the process, so understanding it means reading every service; cyclic event dependencies creep in; and answering "where is order 4711 stuck?" requires correlating events across systems.
Orchestration
A coordinator holds the saga's state and issues commands.
// Orchestrator: explicit state machine, persisted after every transition.
@Component
class PlaceOrderSaga {
@SagaStep(order = 1, compensate = "releaseStock")
void reserveStock(SagaState s) { inventory.reserve(s.orderId(), s.lines()); }
@SagaStep(order = 2, compensate = "refund") // pivot: after this, we go forward
void capturePayment(SagaState s) { payments.capture(s.orderId(), s.total(), s.orderId()); }
@SagaStep(order = 3) // retriable: never compensated
void arrangeShipment(SagaState s) { shipping.schedule(s.orderId(), s.address()); }
void releaseStock(SagaState s) { inventory.release(s.orderId()); }
void refund(SagaState s) { payments.refund(s.orderId(), s.total(), s.orderId() + ":refund"); }
}
The flow is explicit, testable and observable; timeouts and escalation live in one place; a support tool can show every in-flight instance. The costs are a component that must know all participants, a risk of business logic accumulating in the orchestrator rather than in the services, and one more thing to make highly available. Note that both capture and refund pass a stable idempotency key — without it, a retried compensation refunds twice.
| Criterion | Choreography | Orchestration |
|---|---|---|
| Coupling | Low; services know only events | Coordinator knows every participant |
| Visibility of the flow | Implicit, spread across services | Explicit and centrally queryable |
| Suitable size | 3–4 steps | Any, and essential beyond about 5 |
| Timeouts and escalation | Per service, duplicated | Central |
| Testing the whole flow | Hard | Straightforward |
| Failure to watch for | Cyclic event chains | A distributed monolith in the coordinator |
Either way, sagas need timeouts on every step (a step that never answers must eventually be treated as failed), persistence of saga state so a coordinator restart resumes rather than restarts, and a plan for the case where compensation itself fails — which usually means retrying it indefinitely and raising an operational alert, because there is no further fallback.
The dual-write problem and the outbox
A subtler failure sits underneath all of this. A service that updates its database and then publishes an event is performing two writes to two systems without atomicity. If it crashes between them, the state changed and no event was published; if it publishes first and the commit fails, consumers act on something that never happened. Neither is recoverable by retrying blindly.
The transactional outbox removes the problem by making both writes local:
BEGIN;
UPDATE orders SET status = 'CONFIRMED' WHERE id = '8f21c' AND status = 'PENDING';
INSERT INTO outbox (id, aggregate_type, aggregate_id, type, payload, created_at)
VALUES (gen_random_uuid(), 'order', '8f21c', 'OrderConfirmed',
'{"orderId":"8f21c","total":"89.90","currency":"EUR"}'::jsonb, now());
COMMIT;
A separate relay then publishes rows from outbox to the broker and marks or deletes them. If it crashes after publishing but before marking, the message is sent twice — at-least-once, which consumers must already tolerate. The relay can poll the table, or read the database's write-ahead log through change data capture, which avoids polling load and preserves commit order.
The mirror image on the consumer side is the inbox: record each processed message identifier in the same transaction as its effect, and skip anything already recorded. Together, outbox and inbox turn an unreliable pair of systems into a reliable pipeline without any distributed protocol.
An alternative for event-sourced services is the listen-to-yourself pattern: append the event to the event store as the only write, and publish from the store. There is no dual write because the store is the system of record.
At-least-once, idempotency and the exactly-once illusion
Every messaging system offers a choice among at-most-once (fire and forget, may lose), at-least-once (retry until acknowledged, may duplicate) and exactly-once (advertised, and worth reading carefully). Almost all real systems run at-least-once, because losing a payment instruction is worse than processing a duplicate you can detect.
Exactly-once delivery is impossible over a network that can drop messages: a sender that receives no acknowledgement cannot distinguish a lost message from a lost acknowledgement, and must choose to resend (risking duplication) or not (risking loss). What is achievable is exactly-once effect, and there are two routes. The first is idempotent processing: the consumer detects and ignores repeats, using a stored message identifier, a natural business key, or a state-conditional update such as UPDATE ... WHERE status = 'PENDING' that simply affects zero rows the second time. The second is a transactional boundary inside one system — Kafka's transactional producer with read-process-write, for example, atomically commits produced records and consumed offsets, which gives exactly-once semantics within Kafka but says nothing about the external database you also wrote to. The guarantees and their limits are covered in messaging patterns and guarantees.
Practical implications: make every consumer idempotent regardless of what the broker promises; propagate a business-level idempotency key end to end rather than relying on broker message identifiers, which change when a message is republished; and keep deduplication records for longer than the maximum plausible retry window.
Eventual consistency in the design
Choosing eventual consistency is a product decision as much as a technical one, and the designs that work make the lag visible rather than hiding it.
- Show the state honestly. "Payment processing" is a truthful and acceptable status; a balance that silently displays a stale figure is not.
- Read your own writes. A user who just changed their address must see the new one. Route their reads to the primary for a short window, or serve the pending value from the write side.
- Make convergence observable. Monitor replication and projection lag as first-class metrics with alerts, because a stalled projection is silent until someone complains.
- Design for reconciliation. Run a periodic job that compares services' views of the same facts and reports discrepancies. Any eventually consistent system accumulates them; the choice is whether you find them or a customer does.
- Version and order events so a consumer can discard something it has already superseded, since ordering across partitions is not guaranteed by default.
Key takeaways
- Isolation, not atomicity, is what you truly lose when a transaction spans services, so every design must state what other actors may observe mid-flight; CAP constrains behaviour only during a partition and only per data set; PACELC names the trade-off you make constantly — latency against consistency in normal operation.
- Two-phase commit is correct and blocking: a coordinator failure between prepare and decide holds locks across every participant, which is why XA suits a single stack and not independent services.
- A saga is a sequence of local transactions with compensations; order compensatable steps before the pivot, make every step idempotent, persist saga state and put timeouts on everything.
- Choreography suits short flows and keeps coupling low; orchestration is necessary once a flow has several steps or the business needs to see where an instance is stuck.
- The transactional outbox eliminates the dual-write problem locally, and an inbox on the consumer side makes at-least-once delivery safe.
- Exactly-once delivery does not exist; exactly-once effect does, through idempotent consumers and business-level idempotency keys carried end to end.
Sources and further reading
Frequently asked questions
- Does the CAP theorem mean I must choose between consistency and availability?
- Only during a network partition, and only for the data that the partition separates. When the network is healthy — which is nearly all the time — you can have both, and the interesting trade-off then is latency against consistency, which is what PACELC adds. Treating CAP as a permanent architectural choice between two labels leads to systems that give up consistency they never needed to give up.
- Why is two-phase commit considered harmful for microservices?
- It blocks. Between the prepare vote and the commit decision, every participant holds locks and cannot proceed unilaterally, so a coordinator failure at that moment leaves resources locked until it recovers. It also requires every participant to support the protocol, couples their availability together, and scales poorly. Inside one application server against two local resources it is fine; across independently deployed services it is not.
- How do I choose between choreography and orchestration for a saga?
- Count the steps and ask who needs to see the process. Up to three or four steps with simple failure handling, choreography keeps services decoupled and needs no new component. Beyond that, or when the business needs to ask where a given instance is stuck, an orchestrator makes the flow explicit, testable and observable at the price of a component that knows about every participant.
- Is exactly-once delivery possible?
- Exactly-once delivery over an unreliable network is not achievable, because the sender can never distinguish a lost message from a lost acknowledgement. What is achievable is exactly-once effect: at-least-once delivery combined with idempotent processing, or a transactional read-process-write within one system such as Kafka's transactional producer. Systems advertising exactly-once semantics are providing the second, within their own boundary.