Software Architecture
Microservice Design Patterns: Resilience, Discovery, Data and Observability
Advanced9 min readUpdated
Distributed systems fail in ways single processes do not, and the microservice pattern catalogue exists to make those failures survivable. This guide groups the patterns by the problem they solve — routing, resilience, data consistency, deployment and observability — shows configuration and code for the ones you will actually implement, and is explicit about the cost each one adds.
What you will learn
- Select the right routing and discovery pattern for a given deployment platform
- Configure timeouts, retries, circuit breakers and bulkheads so that failure stays contained
- Make an operation idempotent so that retries and redeliveries are safe
- Choose between CQRS, event sourcing and a plain shared read model for a query problem
- Decide whether a service mesh earns its operational complexity for your platform
The microservice pattern catalogue reads like a shopping list, and treating it as one is the fastest way to build something nobody can operate. Every pattern here exists because a specific failure hurt someone badly enough to be written down, and every pattern adds moving parts. The useful question is never "which patterns should we adopt" but "which failure am I currently exposed to, and what is the cheapest thing that removes it".
This guide is organised by problem rather than by name. It assumes you have already decided on the style and drawn your boundaries — see microservices for that — and it defers the deep treatment of consistency to distributed transactions and consistency, covering saga and outbox here only as far as their shape in a design.
Getting a request to the right instance
Edge gateway. Clients should not know your service topology. A single entry point terminates TLS, authenticates, applies rate limits and routes by path or host to the appropriate service. It keeps cross-cutting concerns in one place and lets you move a route to a new implementation without touching clients — which is what makes the strangler fig migration possible at all. The gateway must stay thin: routing, policy, translation of protocols, nothing that encodes business rules. The API management guide covers products and policies in depth.
Backend for frontend (BFF). When a mobile app and a web app need materially different aggregations, one gateway API serving both becomes a compromise that suits neither. A BFF is a small service per client type, owned by the client team, that composes calls to downstream services and shapes responses for one consumer. The risk is duplicated logic across BFFs; keep them to composition and formatting.
Service discovery. Instances come and go, so callers resolve a logical name to an address at call time. There are two shapes. In client-side discovery the caller queries a registry and picks an instance itself, which allows sophisticated selection — least-loaded, zone-aware — at the price of a discovery client in every language you use. In server-side discovery the caller talks to a stable virtual address and a proxy or platform load balancer picks the instance. On Kubernetes this is simply a Service, and it is the reason most new systems do not need a separate registry at all.
| Approach | Where the choice is made | Fits |
|---|---|---|
| Client-side registry | In the caller's process | Mixed platforms, VM deployments, fine-grained routing |
| Server-side proxy | In a load balancer or platform | Container orchestration, polyglot fleets |
| DNS only | Resolver, with TTL caching | Simple topologies; beware stale caches on failover |
Load balancing deserves one caution: round-robin distributes requests evenly, which is the wrong goal when requests differ in cost. A least-outstanding-requests policy sends work to whichever instance has the fewest calls in flight and adapts automatically to a slow instance, so prefer it where your proxy or client supports it.
Keeping failure contained
Every remote call has three bad outcomes: it fails fast, it fails slow, or it lies. Slow failure is the dangerous one, because a caller waiting on a dependency holds a thread, a connection and memory, and a few thousand such callers exhaust the caller's own capacity. The patterns below all address that chain.
Timeouts. The single highest-value control, and the one most often left at a library default measured in minutes. Every outbound call needs a deadline derived from what the caller can afford, not from what the callee usually takes. A useful discipline is a request-level budget: an inbound request allowed 800 ms allocates 300 ms to one dependency and 200 ms to another, and any call that would exceed the remaining budget is not attempted at all.
Retries with backoff and jitter. Retrying a transient failure is worthwhile; retrying immediately, from every client, is how you turn a five-second blip into a thundering herd. Use a small maximum attempt count, exponential backoff, and random jitter so that clients do not synchronise. Never retry a request that was not idempotent, and never retry a response that indicates a client error.
Circuit breaker. Track the recent failure rate of a dependency; when it crosses a threshold, stop calling for a cooling-off period and fail immediately with a fallback. After the window, allow a few probe calls, and close the breaker if they succeed. The value is twofold: the caller stops burning resources on calls that will fail, and the struggling dependency gets a chance to recover instead of being hammered.
Bulkhead. Partition the caller's resources so no single dependency can consume them all. In practice that means a bounded thread pool or a semaphore limiting concurrent calls per dependency, and separate connection pools rather than one shared pool.
resilience4j:
circuitbreaker:
instances:
pricing:
slidingWindowType: COUNT_BASED
slidingWindowSize: 50
failureRateThreshold: 50 # open at 50% failures over the window
slowCallDurationThreshold: 400ms # a slow call counts as a failure
slowCallRateThreshold: 60
waitDurationInOpenState: 10s
permittedNumberOfCallsInHalfOpenState: 5
bulkhead:
instances:
pricing:
maxConcurrentCalls: 20 # never more than 20 in flight
maxWaitDuration: 0 # reject immediately rather than queue
retry:
instances:
pricing:
maxAttempts: 3
waitDuration: 100ms
enableExponentialBackoff: true
exponentialBackoffMultiplier: 2
retryExceptions:
- java.io.IOException
@Service
class PricingClient {
@CircuitBreaker(name = "pricing", fallbackMethod = "listPrice")
@Bulkhead(name = "pricing")
@Retry(name = "pricing")
Money quote(String sku, String customerId) {
return http.get("/quotes/{sku}?customer={c}", Money.class, sku, customerId);
}
// Degrade, do not fail: the catalogue price is worse than a quote but better than an error page.
private Money listPrice(String sku, String customerId, Throwable cause) {
log.warn("pricing degraded for {}: {}", sku, cause.toString());
return catalogue.listPrice(sku);
}
}
Rate limiting and load shedding protect the callee side of the same relationship. A service that accepts every request until it collapses serves nobody; one that rejects excess load quickly with 429 keeps serving the traffic it can handle. Shedding by priority — reject batch and reporting traffic before interactive traffic — is a cheap and effective refinement.
Making operations safe to repeat
Resilience patterns guarantee that some requests will be delivered more than once. At-least-once delivery from a broker guarantees the same. The only defence is idempotency: processing the same logical operation twice has the same effect as processing it once.
Reads and deletes are naturally idempotent, and PUT replaces a whole resource so it usually is too. The problem is creation and mutation. The standard answer is an idempotency key supplied by the client, stored by the server with the result, and replayed on a repeat.
-- The uniqueness constraint, not the application code, is what makes this correct.
CREATE TABLE processed_request (
idempotency_key VARCHAR(64) PRIMARY KEY,
request_hash CHAR(64) NOT NULL,
response_body JSONB NOT NULL,
created_at TIMESTAMPTZ NOT NULL DEFAULT now()
);
The handler inserts the key in the same transaction as the business change. If the insert violates the primary key, the request is a repeat: return the stored response. If the same key arrives with a different payload hash, that is a client bug — reject it with 422 rather than silently returning the old answer. Keep rows for longer than any client's retry window, then expire them.
For message consumers the same idea applies with the message identifier as the key. Where the effect is a state transition, a cheaper alternative works well: make the update conditional on the current state, so UPDATE orders SET status='PAID' WHERE id=? AND status='PENDING' simply affects zero rows on a duplicate.
Data patterns across services
Once each service owns its data, three recurring problems appear: writing to your database and publishing an event atomically, keeping a business process consistent across services, and querying data you do not own.
Transactional outbox. A service cannot write its database and publish to a broker in one transaction, so it writes the event into an outbox table inside the business transaction, and a separate relay reads that table and publishes. Nothing is lost if the process dies between the two steps.
Saga. A business process spanning services becomes a sequence of local transactions, each with a compensating action if a later step fails. Choreography wires services through events; orchestration puts a coordinator in charge of the sequence. Both are covered properly, with failure analysis, in the distributed transactions guide.
CQRS. Separate the model you write from the model you read. The write model enforces invariants and stays normalised; one or more read models are shaped for specific queries and updated from events. Use it where read and write requirements genuinely diverge — high-volume search over data with a complex write model — and not as a default, because it introduces a lag that every consumer must tolerate and every UI must be designed around.
Event sourcing. Persist the sequence of state-changing events instead of the current state; rebuild state by replaying them. You gain a perfect audit trail, temporal queries and the ability to build new projections from history. You pay with schema evolution of events that live forever, snapshotting for long streams, and a mental model unfamiliar to most teams. Adopt it for a bounded context where history is the product — ledgers, trading, compliance — rather than across a whole system.
API composition. The simple case: a caller fans out to several services and joins the results. Perfectly reasonable for small result sets, unworkable for anything needing cross-service filtering or sorting.
Deployment and platform patterns
Sidecar. Deploy a helper container alongside the service in the same unit, sharing its lifecycle and network namespace, to provide a capability the service should not implement itself: log shipping, credential rotation, a proxy. The service stays focused on business behaviour; the sidecar is upgraded independently.
Service mesh. A mesh is sidecars all the way down: every instance gets a proxy, and a control plane configures them to provide mutual TLS, retries, timeouts, traffic splitting, and uniform telemetry without application code. That is genuinely valuable in a large polyglot fleet. It is also a second distributed system with its own upgrade path, failure modes and latency cost, and it duplicates capabilities your libraries may already provide. Decide by counting: how many languages, how many services, how much of the mesh's feature list would you otherwise write yourself.
Externalised configuration. Configuration comes from the environment, not from the artefact, so one image is promoted unchanged through environments. Secrets come from a secret manager and are rotatable without a rebuild.
Health endpoints. Distinguish liveness (is the process broken beyond recovery — restart it) from readiness (can it serve traffic right now — send it none until it can). Conflating them causes restart loops when a dependency is briefly unavailable, which converts a partial outage into a total one.
Blue/green and canary releases. Route a small share of production traffic to the new version, compare error rate and latency against the old, and promote or roll back automatically. This is the pattern that makes independent deployment safe in practice, and it depends on backward-compatible interfaces and database changes.
Seeing what is happening
In a monolith a stack trace explains a failure. In a distributed system nothing does unless you built for it.
Correlation and distributed tracing. Every inbound request gets a trace identifier, propagated on every outbound call using W3C traceparent headers, and each hop records a span. The resulting trace shows exactly where the 4-second checkout went. OpenTelemetry is the vendor-neutral way to instrument this; auto-instrumentation for common frameworks means the baseline costs little effort.
Structured logs. JSON lines with the trace ID, service name, version and tenant on every record, aggregated centrally. Free-text logs across twelve services are unsearchable.
Metrics that describe users. Request rate, error rate and latency percentiles per endpoint, plus saturation of the pools you bulkheaded. Alert on service level objectives — "99 % of checkouts complete in under 1 s over 30 days" — rather than on CPU, which tells you nothing about whether anyone is suffering. Interpreting those percentiles correctly is the subject of application performance.
Aggregate health. A dashboard per service showing the four signals plus circuit-breaker state is enough for most incidents; anything more elaborate goes unread.
Key takeaways
- Adopt a pattern to remove a failure you are actually exposed to; every one of them adds moving parts and operational surface.
- Gateways and discovery keep clients ignorant of topology; on a container platform the platform's own service abstraction usually replaces a separate registry.
- Timeouts, bounded retries with jitter, circuit breakers and bulkheads work as a set — configure retries in exactly one layer and wrap them inside the breaker.
- At-least-once delivery and retries make idempotency mandatory; enforce it with a stored idempotency key or a state-conditional update, not with hopeful application logic.
- Outbox, saga, CQRS and event sourcing solve distinct data problems; use the smallest one that fits and avoid defaulting to event sourcing.
- Distributed tracing, structured logs with a correlation ID and SLO-based alerting are not optional extras — without them a microservice failure has no explanation.
Sources and further reading
Frequently asked questions
- Should retries live in the client, the gateway or the service mesh?
- In exactly one place. Retries configured at three levels multiply: three attempts each becomes twenty-seven requests to a struggling service, which is how a brownout becomes an outage. Put them where the caller knows whether the operation is safe to repeat, which is normally the client library, and disable them elsewhere. If a mesh does the retrying, the application must not.
- What is the difference between a circuit breaker and a bulkhead?
- A circuit breaker stops calling a dependency that is already failing, so the caller fails fast instead of queuing behind timeouts. A bulkhead limits how much of the caller's own resource — threads, connections, in-flight requests — any single dependency may consume, so one slow dependency cannot starve the rest of the application. They address different halves of the same problem and are usually applied together.
- Do I need a service mesh?
- Only when you have enough services in enough languages that implementing mTLS, retries, traffic splitting and telemetry in each client library has become a real burden. A mesh moves that work into sidecar proxies at the price of another distributed control plane to run, upgrade and debug, plus a small latency tax per hop. Below roughly a dozen services, libraries are usually cheaper.
- Is event sourcing required for CQRS?
- No, and conflating them causes a lot of unnecessary complexity. CQRS simply means the write model and the read model are separate, which you can implement with an ordinary database and a projection table. Event sourcing means the event log is the system of record. Each is useful alone; combining them makes sense mainly when you need a full audit history and temporal queries.