Microservices Infrastructure Anti-Patterns: Sync, Idempotency, Coupling, Retries (2026)

Q: What are the four most damaging microservices infrastructure anti-patterns?

Synchronous blocking calls across services (one slow downstream propagates up the call graph and freezes the platform via thread-pool exhaustion); no idempotency on critical operations (network blips trigger client retries that duplicate charges, orders and notifications); tightly coupled services or distributed monolith (services that have to be deployed together in coordinated waves, defeating the independent-deployment promise of microservices); and centralised retry logic (a shared retry library that, in aggregate across thousands of instances, produces a thundering-herd stampede that prevents recovering downstreams from ever recovering). All four account for the majority of P1 incidents in modern back-end platforms; all four have well-understood structural fixes from the positive pattern catalogue. The four are not independent — each one accelerates the others, so a single 30-second downstream blip in a system with all four can become a multi-hour outage with duplicate charges and a release-train freeze.

Q: How do synchronous blocking calls cause cascading outages?

Service A calls Service B synchronously and waits for the response; B calls C synchronously; C is slow today. Service B is now holding open requests for the duration of C's slowness; B's thread pool fills; new requests to B start queueing. Service A is now holding requests for the same duration; A's thread pool fills; A's endpoints that did not even need C are now queueing because A's threads are all busy waiting on B waiting on C. Within minutes the entire request graph that touches C is held in waiting threads even though most of the platform's traffic does not need C. Recovery does not happen until C recovers because every service is waiting on every other service waiting on C. The fix is the resilience pipeline — bulkhead caps concurrency, circuit breaker fails fast on consistent failure, tight timeout converts blocking into bounded degradation, and a fallback returns a degraded response rather than blocking for the full timeout. The graceful degradation classification (critical vs supplementary) determines which calls need the pipeline and which calls can be removed from the synchronous critical path entirely.

Q: Why do duplicate orders happen, and how does idempotency stop them?

A client calls a state-changing endpoint over an unreliable network. The server processes the request successfully but the response is lost in flight — the TCP connection drops, the load balancer times out, the client's HTTP library throws "socket hang up". The client retries because retrying on transient errors is a sensible default. The server processes the retry as a brand-new request because nothing in the protocol distinguishes "this is the same request you already saw" from "this is a fresh request". The user is charged twice; two orders are placed. The fix is the idempotency pattern: every state-changing endpoint accepts an idempotency key (typically a UUID generated by the client per logical operation, sent in an HTTP header); the server stores a mapping from key to response with a TTL of 24 hours or longer; on receipt of a request with a known key, the server returns the stored response without re-executing the operation. The latency overhead is a single Redis lookup per request, well under 5ms in production. A fintech we worked with had 0.3% duplicate-charge rate (six thousand duplicates per day at two million transactions) that disappeared completely after one sprint of idempotency-key rollout.

Q: How do I know if I have built a distributed monolith?

Three signals. Routine feature work requires changes to four or more services that have to be released together in a specific order with coordinated schema migrations. The integration test suite spins up most of the platform together because mocking is impractical given the cross-service coupling. On-call rotation cannot reliably debug any incident without context from at least three teams. The deployment dependency graph (which services have to deploy together for routine changes) shows large connected components. The cause is service boundaries drawn by technical layer (API service, business service, data service) or by team boundary at the moment of migration, rather than by domain bounded context. The fix is to redraw service boundaries around bounded contexts (services that always change together belong inside the same context and should be merged) and to replace synchronous coupling with asynchronous events (Service B publishes events when its data changes; Service A maintains a local read model populated from those events). The combination produces services that can be deployed independently because their compatibility contract is event schemas, not synchronous API contracts.

Q: What is a retry stampede and how do I prevent it?

A downstream service has a brief degradation: it returns 503 errors for thirty seconds while it is restarting or while a dependency recovers. Every caller of that downstream — typically a thousand or more service instances across many services — receives a 503; each instance retries three times because the shared HTTP client library wraps every call with automatic retry. The downstream now receives four times its normal request volume from a thousand instances simultaneously, all targeted at a service that is in the middle of recovering. The recovering service is overwhelmed and stays unavailable; the retries continue; the stampede continues. The thirty-second blip becomes a thirty-minute outage. The fix has three components. First, move retry decisions to the call site so each call expresses its own retry policy based on its semantics (idempotent vs not, internal vs external). Second, every retry policy must include exponential backoff and jitter — the jitter desynchronises retries from many instances. Third, implement a platform-wide retry budget — a token-bucket counter shared across instances (typically Redis-backed) that throttles total retries per unit time, so when a downstream is failing the retries naturally reduce to a sustainable level rather than amplifying the failure. Pair with a circuit breaker so consistent failures stop new retries entirely.

Q: In what order should I unwind the four infrastructure anti-patterns?

Start with idempotency (anti-pattern #2) because it is the cheapest fix and unblocks safe retry behaviour for all the others — you cannot safely retry a non-idempotent write, so until idempotency is in place every other resilience improvement is constrained. Add the resilience pipeline (bulkhead, breaker, timeout, fallback) for synchronous calls (anti-pattern #1) next because it converts unbounded blocking into bounded degradation and gives the platform back its responsiveness under partial failure. Replace shared retry logic with call-site retry plus a retry budget (anti-pattern #4) third because it removes the stampede dynamic without disturbing the resilience pipeline added in step two. Address service coupling and bounded-context redrawing (anti-pattern #3) last because it is the most expensive fix — it requires redrawing service boundaries and converting synchronous calls to events — and the previous three fixes make the system safe enough to do the coupling work without constant fire-fighting. A retail platform we worked with completed this sequence in eight months; zero P1 incidents in the following twelve months were attributable to these causes, down from roughly seventy per cent before.

Q: How do I detect these anti-patterns before they cause incidents?

Four detection patterns, all instrumentable in standard observability stacks. Anti-pattern #1: plot p99 latency by endpoint over time, grouped by service; if p99 latency for unrelated endpoints rises and falls together when a single downstream degrades, the system has synchronous-blocking symptoms. Anti-pattern #2: sample successful operations against domain identifiers and look for duplicates within a short window from the same client; any non-zero rate is evidence of missing idempotency. Anti-pattern #3: build a graph of which services have to be deployed together for routine work; connected components larger than two or three services indicate distributed-monolith symptoms. Anti-pattern #4: measure the ratio of retry requests to fresh requests, both per-instance and platform-wide; a healthy platform has under 5% in steady state with a sharply bounded ratio under failure (capped by the retry budget); an unhealthy platform shows 200-400% spikes when a downstream degrades. The variance metrics make the anti-patterns visible early enough to unwind without rebuilding the entire system.

Q: When are these anti-patterns acceptable trade-offs?

Three scenarios. Genuinely small systems with three services, one team, and low traffic do not need a retry budget, do not need bounded-context boundary work, and may not need idempotency keys if the operational cost of the occasional duplicate is lower than the engineering cost of implementing the keys. The anti-patterns are anti-patterns at scale; at small scale the simpler architecture is correct. The graduation point is when the team grows beyond about ten engineers, traffic exceeds a few hundred requests per second, or the cost of a single duplicate operation becomes unacceptable. Internal-only or low-stakes deployments (internal admin tool, twenty employees) can tolerate synchronous coupling and missing idempotency because the failure modes do not scale; public-facing or high-volume systems cannot. Deliberate transitional states are also acceptable — a platform mid-migration may have synchronous coupling between an old monolith and a new microservice during the strangler-fig phase; the anti-pattern is keeping the transitional state indefinitely, not having it during the migration.

Q: How do these anti-patterns compound on each other?

Each anti-pattern accelerates the others. Synchronous blocking (#1) means a slow downstream propagates up the call graph; missing idempotency (#2) means the resulting client retries duplicate the operation; tight coupling (#3) means the cascade reaches services that had no business being involved; centralised retries (#4) amplify the cascade across thousands of instances. A single 30-second downstream blip in a system with all four can produce a multi-hour platform outage with duplicate charges, data inconsistencies, and a release-train freeze while engineering investigates. The same blip in a system that has fixed all four would produce a few hundred milliseconds of degradation on the affected endpoints and zero side effects on unrelated traffic. The compounding effect is why fixing the anti-patterns in the correct order matters — fixing one in isolation produces marginal improvement; fixing all four in sequence produces order-of-magnitude reductions in incident frequency, blast radius, and operational burden.

Q: What real-world results come from unwinding these four anti-patterns?

A retail platform we worked with exhibited all four anti-patterns at the start of the engagement. Pre-unwinding state: a four-hour outage every six to eight weeks traced back to synchronous-call cascades; about four thousand duplicate orders per month from missing idempotency; release trains involving twelve services per month with thirty-minute change windows; a shared retry library that had caused two stampede outages in the prior twelve months. The team's perception was that microservices had been a mistake. The unwinding sequence took eight months in the correct order (idempotency, resilience pipeline, retry budget, bounded-context boundaries). Post-unwinding outcomes: zero P1 incidents in the following twelve months attributable to these causes (down from roughly seventy per cent of P1s); duplicate-order rate of zero; release cadence increased from monthly to weekly per service; on-call burden reduced because incidents were now bounded to single services rather than cascading; the team's perception of microservices reversed because the architecture was finally delivering the promised benefits. Every team that has unwound these anti-patterns in production wishes they had started sooner; no team regrets doing the work.

Satyam Kumar

返回博客

ai-architecture

Microservices Infrastructure Anti-Patterns: Synchronous Blocking, Missing Idempotency, Tight Coupling, Centralised Retries (2026)

By Satyam KumarApril 25, 202626 min read

microservices anti-patterns infrastructure anti-patterns synchronous blocking calls idempotency keys distributed monolith centralised retry logic retry stampede circuit breaker bulkhead isolation graceful degradation event-driven architecture bounded contexts thread pool exhaustion duplicate operations cascading failures production incident analysis distributed systems failures release coupling

Microservices Infrastructure Anti-Patterns: Synchronous Blocking, Missing Idempotency, Tight Coupling, Centralised Retries (2026)

Frequently Asked Questions

分享这篇文章

Twitter LinkedIn WhatsApp

Satyam Kumar

Founder & AI Architect, AppScale LLP

人工智能和云架构师。帮助团队构建可扩展到数百万的系统。

LinkedIn GitHub

Microservices Infrastructure Anti-Patterns: Synchronous Blocking, Missing Idempotency, Tight Coupling, Centralised Retries (2026)

Frequently Asked Questions

分享这篇文章

Comments

Leave a comment