In microservice architectures, dependencies fail. Networks experience partitions, third-party payment gateways slow down, and databases lock up. When a downstream service slows from 50ms to 10 seconds, upstream services quickly exhaust connection pools and thread workers, triggering cascading failures across the entire platform.
The Circuit Breaker Pattern is the premier resilience pattern to isolate outages and ensure graceful degradation.
1. The Anatomy of Cascading Failures
Imagine an e-commerce platform where the Order Service calls the Payment Gateway:
Payment Gatewaysuffers an outage and hangs on incoming connections.- Every request to
Order Serviceties up an HTTP connection socket and Node.js event-loop resources for 30-second timeouts. - Soon,
Order Serviceruns out of free sockets, memory fills up, and health checks fail. - Now, the
Frontend Web Appcannot contactOrder Service, and the entire store crashes.
A Circuit Breaker stops this chain reaction by failing fast as soon as degradation is detected.
2. The 3-State Finite State Machine
The Circuit Breaker pattern is modeled as a finite state machine:
[Normal Operations]
|
v
+------------+
| CLOSED | <-------------------------+
+------------+ |
| |
Failure Threshold Exceeded |
| |
v |
+------------+ Trial Request
| OPEN | Succeeded
+------------+ |
| |
Reset Timeout Elapsed |
| |
v |
+------------+ |
| HALF-OPEN | --------------------------+
+------------+
|
Trial Request Failed (Trip back to OPEN)1. CLOSED (Normal Operation)
failureThreshold, the circuit trips to OPEN.2. OPEN (Fail-Fast)
resetTimeoutMs before testing recovery.3. HALF-OPEN (Canary Probe)
resetTimeoutMs expires, the breaker enters HALF-OPEN.
- It allows a single trial request through:
- If the trial succeeds: The downstream service has recovered. The circuit resets to CLOSED, and normal traffic resumes.
- If the trial fails: The service is still sick. The circuit trips back to OPEN and the reset timer restarts.3. Implementation in TypeScript / Node.js
Here is a production-grade Circuit Breaker wrapper with state machine mechanics:
type CircuitState = "CLOSED" | "OPEN" | "HALF_OPEN";
interface CircuitBreakerOptions {
failureThreshold?: number; // e.g. 3 consecutive failures
resetTimeoutMs?: number; // e.g. 5000ms cool-down
fallback?: (...args: any[]) => any;
}
interface CircuitBreaker<T extends (...args: any[]) => Promise<any>> {
(...args: Parameters<T>): Promise<ReturnType<T>>;
getState(): CircuitState;
reset(): void;
}
function createCircuitBreaker<T extends (...args: any[]) => Promise<any>>(
fn: T,
options: CircuitBreakerOptions = {}
): CircuitBreaker<T> {
const failureThreshold = options.failureThreshold ?? 3;
const resetTimeoutMs = options.resetTimeoutMs ?? 5000;
const fallback = options.fallback;
let state: CircuitState = "CLOSED";
let failures = 0;
let nextAttempt = 0;
const breaker: any = async function (...args: Parameters<T>): Promise<ReturnType<T>> {
const now = Date.now();
// Check if cool-down timer elapsed while OPEN
if (state === "OPEN") {
if (now >= nextAttempt) {
state = "HALF_OPEN";
} else {
// Fail-fast immediately
if (typeof fallback === "function") {
return await fallback(...args);
}
throw new Error("Circuit breaker is OPEN");
}
}
try {
const result = await fn(...args);
// Success: reset failures and restore CLOSED state
failures = 0;
state = "CLOSED";
return result;
} catch (err) {
failures++;
// If we failed in HALF-OPEN or hit failureThreshold in CLOSED -> Trip OPEN
if (state === "HALF_OPEN" || failures >= failureThreshold) {
state = "OPEN";
nextAttempt = Date.now() + resetTimeoutMs;
}
if (typeof fallback === "function" && state === "OPEN") {
return await fallback(...args);
}
throw err;
}
};
breaker.getState = (): CircuitState => {
if (state === "OPEN" && Date.now() >= nextAttempt) {
state = "HALF_OPEN";
}
return state;
};
breaker.reset = (): void => {
state = "CLOSED";
failures = 0;
nextAttempt = 0;
};
return breaker;
}4. Fallback Strategies in Production
When a circuit breaker trips open, what should happen to the end user?
- Stale Cache Fallback: Return cached product information or user profile data from Redis with an indicator header (
X-Cache-Status: Stale). - Degraded Default: E.g. If the recommendation engine is down, return top 10 globally popular items instead of personal recommendations.
- Async Queueing: For non-blocking write operations (e.g. audit logs, notification dispatches), enqueue messages to a background dead-letter queue (DLQ) to retry once the downstream recovers.
5. Senior System Design Evaluation Checklist
In technical interviews, top candidates stand out by addressing:
- •Metrics & Observability: Exporting metrics like
circuit_state{name="payment"}andcircuit_trips_totalinto Prometheus / Datadog dashboards. - •Consecutive Failures vs. Failure Rate Percentage: Simple consecutive counters can trip prematurely during intermittent spikes; production libraries (like Netflix Hystrix or Opossum) track percentage error rates over rolling 10-second windows.
- •Distributed Coordination: Ensuring each containerized Node.js instance manages its own local breaker to prevent central Redis bottlenecks.
Practice Circuit Breaker Live on Pairlet
Frequently Asked Questions
How does a Circuit Breaker differ from retry with exponential backoff?
Retries protect against transient blips by attempting the call again. However, if a downstream service has crashed completely, retrying worsens the outage by generating retry storms. A Circuit Breaker detects that the dependency is down and fails fast immediately without placing additional load on the failing service.
What are the three states of a Circuit Breaker?
CLOSED (normal operation, tracking failures), OPEN (service is down; calls fail-fast immediately without invoking dependency), and HALF-OPEN (testing recovery with a single canary trial request).
What happens when a canary trial request fails in HALF-OPEN state?
If the trial request fails, the circuit immediately returns to the OPEN state and restarts the reset timeout timer, preventing traffic from flooding the still-struggling downstream system.
What fallback strategies should be considered when a circuit trips OPEN?
Options include serving stale cached data from Redis, returning graceful default placeholders (e.g. empty recommendations), or queueing mutations to an asynchronous background dead-letter queue.
Conduct Live Coding Interviews in the Browser
No candidate sign-up required. Create a room, share the link, and code together in real time with shared code execution.