Building Resilient Systems

Overview

Resilience is the ability of a system to handle and recover from failures gracefully, continuing to provide acceptable service levels even when components fail. Unlike fault tolerance - which aims to mask failures entirely through redundancy - resilience acknowledges that failures will happen and focuses on minimizing blast radius, recovering quickly, and degrading gracefully.

A resilient system doesn't promise zero failures. Instead, it promises predictable behavior under failure conditions. Users may experience degraded functionality - slower responses, fewer features, stale data - but the system remains usable rather than collapsing entirely.

Resilience vs. Fault Tolerance

Why It Matters in Production

In distributed systems, partial failures are the norm. A single unresponsive downstream service can cascade into full system outage. Network partitions, overloaded services, and resource exhaustion are daily realities. Resilience patterns protect against these by isolating failures, shedding load, and providing predictable degradation paths.

Consider a typical e-commerce checkout flow that depends on payment processing, inventory checks, shipping calculation, and fraud detection. Without resilience patterns, a 2-second delay in the fraud detection service causes thread pool exhaustion in the checkout service, which cascades to the API gateway, ultimately making the entire platform unresponsive - even product browsing, which has no dependency on fraud detection. This is the cascading failure problem that resilience engineering solves.

---

Resilience Patterns Overview

---

Retry Pattern

When to Retry

Retry only for transient failures: network timeouts, HTTP 503, connection resets, temporary resource unavailability. Never retry for deterministic errors (400 Bad Request, 401 Unauthorized, business logic violations). The distinction is critical - retrying a 400 wastes resources and will never succeed.

Retryable errors: Connection refused, DNS resolution timeout, HTTP 429 (with Retry-After), HTTP 500/502/503/504, socket timeout, TLS handshake failure.

Non-retryable errors: HTTP 400/401/403/404/409/422, validation failures, authentication errors, resource not found, business rule violations.

Key Concepts

• Exponential Backoff: Wait time doubles with each attempt (1s, 2s, 4s, 8s...). This gives the failing service progressively more time to recover between attempts.
• Jitter: Add randomness to prevent thundering herd (retry storms). Full jitter: random(0, baseDelay * 2^attempt). Decorrelated jitter: min(cap, random(base, previousDelay * 3)).
• Max Retries: Cap attempts to avoid infinite loops (typically 3-5). Beyond 3 retries, the probability of success drops dramatically for most transient failures.
• Retry Budget: An alternative to max retries - limit total retry percentage across all requests (e.g., "retries should not exceed 10% of total traffic").
• Idempotency Requirement: The operation MUST be safe to repeat. Non-idempotent operations (e.g., charging a credit card) require idempotency keys. The server uses the key to deduplicate requests, returning the original response for repeated calls.

Pseudocode

function retryWithBackoff(operation, maxRetries = 3, baseDelay = 1000):
    for attempt in 0..maxRetries:
        try:
            return operation()
        catch TransientError as e:
            if attempt == maxRetries:
                throw e
            delay = baseDelay * (2 ^ attempt)
            jitter = random(0, delay * 0.1)
            sleep(delay + jitter)

---

Circuit Breaker Pattern

First popularized by Michael Nygard in Release It! (2007), the circuit breaker prevents an application from repeatedly calling a failing service, giving it time to recover. The analogy is electrical: when current exceeds safe levels, the breaker trips to prevent damage.

Without a circuit breaker, a failing dependency causes callers to accumulate blocked threads waiting for timeouts. The circuit breaker "fails fast" - immediately returning an error without attempting the call - preserving resources for requests that can succeed.

States

• Closed: Requests flow normally. Failures are counted.
• Open: Requests are immediately rejected. A timeout timer starts.
• Half-Open: A limited number of probe requests are allowed through to test recovery.

ASCII State Diagram

                    failure threshold exceeded
         ┌──────────────────────────────────────────┐
         │                                          ▼
    ┌─────────┐                              ┌──────────┐
    │ CLOSED  │◄─── probe succeeds ──────────│HALF-OPEN │
    └─────────┘                              └──────────┘
         │                                        ▲
         │         timeout expires                │
         │    ┌──────────┐                        │
         └───►│   OPEN   │────────────────────────┘
              └──────────┘
              (requests rejected)

Configuration

• Failure Threshold: Number/percentage of failures to trip (e.g., 5 failures in 60s)
• Timeout Duration: How long to stay open before probing (e.g., 30s)
• Probe Count: Number of requests allowed in half-open state

Pseudocode

class CircuitBreaker:
    state = CLOSED
    failureCount = 0
    lastFailureTime = null
    threshold = 5
    timeout = 30_seconds

    function call(operation):
        if state == OPEN:
            if now() - lastFailureTime > timeout:
                state = HALF_OPEN
            else:
                throw CircuitOpenException()

        try:
            result = operation()
            onSuccess()
            return result
        catch Exception as e:
            onFailure()
            throw e

    function onSuccess():
        failureCount = 0
        state = CLOSED

    function onFailure():
        failureCount++
        lastFailureTime = now()
        if failureCount >= threshold:
            state = OPEN

---

Timeout Pattern

Why Timeouts Are Critical

Without timeouts, a slow downstream service holds threads/connections indefinitely, eventually exhausting the caller's resources. Every network call MUST have a timeout. This is the single most important resilience rule.

Common timeout categories:

• Connection timeout: Time to establish TCP connection (typically 1-5s)
• Read/Socket timeout: Time to receive response after connection is established (varies by operation)
• Overall/Request timeout: Total time budget for the entire operation including retries

A missing timeout is a latency leak. In production, "hanging" requests are often worse than failed requests because they silently consume resources while providing no value.

Cascading Timeout Calculation

In a call chain A → B → C, timeouts must decrease at each hop:

A's timeout > B's timeout > C's timeout

Example:
  A timeout = 5000ms
  B timeout = 3000ms  (leaves 2000ms for A's own processing)
  C timeout = 1500ms  (leaves 1500ms for B's own processing)

Pseudocode

function callWithTimeout(operation, timeoutMs):
    future = executeAsync(operation)
    result = future.await(timeoutMs)
    if result == TIMEOUT:
        future.cancel()
        throw TimeoutException("Operation exceeded " + timeoutMs + "ms")
    return result.value

---

Bulkhead Pattern

Named after ship bulkheads that contain flooding to one compartment, this pattern isolates resources so that failure in one area doesn't sink the entire system. In software, this means partitioning thread pools, connection pools, or memory allocations per dependency or per functionality.

Isolation Strategies

• Thread Pool Isolation: Each dependency gets its own thread pool. If one pool is exhausted, others continue functioning. Provides true isolation with timeout enforcement on the calling thread. Higher overhead due to thread context switching.
• Semaphore Isolation: Limits concurrent calls to a dependency without dedicated threads. Lower overhead but no timeout control on the calling thread. Best for high-volume, low-latency calls where thread overhead is unacceptable.

ASCII Diagram

┌─────────────────────────────────────────────────┐
│                  Application                     │
│                                                  │
│  ┌─────────────┐  ┌─────────────┐  ┌─────────┐ │
│  │ Payment Svc │  │ Inventory   │  │ Email   │ │
│  │ Pool: 10    │  │ Pool: 20    │  │ Pool: 5 │ │
│  │ ██████░░░░  │  │ ████░░░░░░░ │  │ ██░░░   │ │
│  └──────┬──────┘  └──────┬──────┘  └────┬────┘ │
│         │                 │              │       │
└─────────┼─────────────────┼──────────────┼───────┘
          ▼                 ▼              ▼
    [Payment API]    [Inventory API]  [Email API]

If Payment API hangs → only its 10 threads are blocked.
Inventory and Email continue operating normally.

Pseudocode

class Bulkhead:
    semaphore = Semaphore(maxConcurrent)

    function execute(operation):
        if not semaphore.tryAcquire(timeoutMs):
            throw BulkheadFullException()
        try:
            return operation()
        finally:
            semaphore.release()

---

Fallback Pattern

When the primary operation fails, provide a degraded but acceptable alternative response. The key insight is that for many features, some response is far better than an error. Users would rather see yesterday's recommendations than a 500 error page.

Strategies

• Cache Fallback: Return stale cached data when the live service is unavailable. Works well for data that changes infrequently (product catalogs, user profiles).
• Default Values: Return safe defaults (e.g., empty recommendations list, standard shipping estimate). The UI should handle these gracefully.
• Static Response: Pre-computed response for critical paths. Generated during deployment and stored locally.
• Alternative Service: Route to a backup provider (e.g., secondary payment processor, different CDN).

Pseudocode

function callWithFallback(primaryOp, fallbackOp):
    try:
        return primaryOp()
    catch Exception as e:
        log.warn("Primary failed, using fallback", e)
        return fallbackOp()

// Usage
result = callWithFallback(
    () -> recommendationService.getPersonalized(userId),
    () -> cache.get("popular_items")  // stale but acceptable
)

---

Rate Limiting

Rate limiting protects services from being overwhelmed by controlling request throughput. It serves dual purposes: protecting backend services from overload and ensuring fair resource allocation among clients.

Rate limiting can be applied at multiple layers: API gateway (global), service level (per-service), and client level (per-user/API-key). Each layer serves a different purpose - gateway limiting prevents infrastructure overload while per-user limiting ensures fairness.

Algorithm Comparison

Token Bucket Pseudocode

class TokenBucket:
    capacity = 100
    tokens = 100
    refillRate = 10  // tokens per second
    lastRefill = now()

    function allowRequest():
        refill()
        if tokens > 0:
            tokens--
            return true
        return false

    function refill():
        elapsed = now() - lastRefill
        newTokens = elapsed * refillRate
        tokens = min(capacity, tokens + newTokens)
        lastRefill = now()

---

Health Checks

Liveness vs. Readiness

A service can be live but not ready (e.g., still warming caches, waiting for DB connection pool, loading ML models). Kubernetes uses this distinction to avoid sending traffic to pods that are alive but not yet capable of serving requests.

Deep Health Checks

Go beyond "process is running" to verify actual dependencies. Shallow health checks (just returning 200) hide real problems. Deep checks verify the service can actually do its job:

function deepHealthCheck():
    checks = {}
    checks["database"] = pingDB(timeout=2s)
    checks["cache"] = pingRedis(timeout=1s)
    checks["downstream_api"] = httpGet(healthUrl, timeout=3s)

    status = all(checks.values()) ? "HEALTHY" : "DEGRADED"
    return { status, checks, timestamp: now() }

---

Combining Patterns

Resilience patterns are most powerful when composed together. No single pattern handles all failure modes - retries handle transient blips, circuit breakers handle sustained failures, timeouts prevent resource exhaustion, and fallbacks ensure user experience. The typical composition order:

Request
  │
  ▼
┌──────────┐    trip threshold    ┌──────────────┐
│ Timeout  │───────────────────►  │   Fallback   │
└────┬─────┘                      └──────────────┘
     │                                   ▲
     ▼                                   │
┌──────────────┐   circuit open          │
│Circuit Breaker├────────────────────────►┘
└────┬─────────┘
     │
     ▼
┌──────────┐
│  Retry   │──── retries exhausted ──────►┘
└────┬─────┘
     │
     ▼
┌──────────┐
│ Bulkhead │──── pool full ──────────────►┘
└────┬─────┘
     │
     ▼
[Downstream Service Call]

Execution order (outermost to innermost): Timeout → Circuit Breaker → Retry → Bulkhead → Actual Call. If any layer fails, control passes to the Fallback.

Composed Pseudocode

function resilientCall(operation):
    return withTimeout(5000,
        withCircuitBreaker("payment-svc",
            withRetry(3,
                withBulkhead("payment-pool",
                    operation
                )
            )
        )
    ).orFallback(() -> cachedResponse())

---

Real-World Examples

Netflix Hystrix (Archived)

Pioneer of resilience patterns in microservices. Introduced thread pool isolation, circuit breakers, and real-time metrics dashboards. Key innovations included request collapsing (batching multiple requests), request caching, and a real-time dashboard for monitoring circuit states across hundreds of services. Now in maintenance mode but its concepts live on in every modern resilience library.

Resilience4j (Java)

Lightweight, functional-style library designed as a Hystrix replacement. Modular design - use only what you need. Supports circuit breaker, retry, rate limiter, bulkhead, and time limiter as composable decorators. Built for Java 8+ with functional interfaces, making it natural to compose with lambdas.

CircuitBreaker cb = CircuitBreaker.ofDefaults("backendService");
Retry retry = Retry.ofDefaults("backendService");
Supplier<String> decorated = Decorators.ofSupplier(() -> service.call())
    .withCircuitBreaker(cb)
    .withRetry(retry)
    .decorate();

Polly (.NET)

Policy-based resilience for .NET. Fluent API for defining retry, circuit breaker, timeout, bulkhead, and fallback policies. Supports policy wrapping for composition. Integrates with HttpClientFactory for HTTP resilience out of the box. Polly v8 introduced a unified ResiliencePipeline concept for cleaner composition.

AWS SDK Retry Policies

AWS SDKs implement adaptive retry with token buckets. Three modes available:

• Legacy: Simple exponential backoff
• Standard: Exponential backoff with jitter, token bucket to limit retries during sustained errors
• Adaptive: Dynamically adjusts retry rates based on throttling responses, using a client-side token bucket that drains faster when receiving throttling errors

---

Advantages & Disadvantages

---

Constraints & Edge Cases

Retry Storms

When many clients retry simultaneously, they amplify load on an already struggling service. A service handling 1000 req/s that starts failing will face 3000 req/s if every client retries 3 times - exactly when it can least handle extra load. Mitigations:

• Add jitter to spread retries across time
• Use circuit breakers to stop retries entirely when failure rate is high
• Implement server-side backpressure (HTTP 429 with Retry-After header)
• Use retry budgets: "no more than 10% of requests should be retries"
• Implement adaptive concurrency limiting on the server side

Circuit Breaker in Distributed Systems

Each instance maintains its own circuit state. This creates interesting dynamics:

• Instance A may have an open circuit while Instance B's is closed (different failure observations)
• Solution: Shared state via Redis/distributed cache, or accept per-instance behavior
• Half-open probes from multiple instances can overwhelm a recovering service
• Count-based thresholds behave differently under varying load - prefer rate-based (e.g., "50% failure rate over 10s window")
• Sliding window counters (count-based or time-based) provide more stable behavior than simple counters

Timeout Propagation

Timeouts must be propagated through the call chain via deadline/context:

• Pass absolute deadline, not relative timeout (avoids clock drift issues between hops)
• Each layer subtracts its own processing time before passing the remaining budget downstream
• If remaining budget ≤ 0, fail fast without making the call
• gRPC implements this natively with deadline propagation in metadata
• In HTTP, use custom headers (e.g., X-Request-Deadline) to propagate deadlines

Bulkhead Sizing

• Too small: legitimate requests rejected under normal load (false positives)
• Too large: doesn't provide meaningful isolation (defeats the purpose)
• Guideline: Size based on p99 latency × expected throughput, with 20-30% headroom
• Monitor rejection rates - sustained rejections indicate undersizing
• Consider dynamic sizing based on observed latency (adaptive bulkheads)
• Account for burst traffic patterns; steady-state sizing may be insufficient during peaks

---

Interview Follow-ups

Q1: How would you handle a circuit breaker in a multi-region deployment?

A: Each region maintains independent circuit state. A failure in us-east-1's downstream shouldn't open circuits in eu-west-1. However, for shared dependencies (global DB), you might use a distributed circuit state with region-aware thresholds. Cross-region failover should be triggered by health checks, not circuit breakers.

Q2: What happens if your retry policy conflicts with your timeout policy?

A: The timeout is the outer boundary. If timeout = 5s and each retry takes 2s with 3 max retries (potentially 6s total), the timeout will kill the operation mid-retry. Solution: Calculate max retry budget as timeout / (maxRetries + 1) and set per-attempt timeouts accordingly.

Q3: How do you test resilience patterns?

A: Use chaos engineering tools (Chaos Monkey, Gremlin, Toxiproxy) to inject failures: latency, connection drops, error responses. Test in staging with production-like traffic. Verify circuit breakers trip at expected thresholds, retries don't amplify load, and fallbacks return acceptable data.

Q4: Your service has 10 downstream dependencies. How do you size bulkheads?

Hint: Consider the relative importance of each dependency, their latency profiles, and what happens to the user experience if each one is isolated. Think about whether all dependencies are equally critical.

Q5: A retry storm brought down your payment service during a flash sale. How do you prevent this?

Hint: Think about client-side vs. server-side solutions. Consider adaptive mechanisms that respond to current system load rather than static configurations.

---

Counter Questions to Ask Interviewer

1. "What's the acceptable degradation level? Should the system return stale data, empty results, or an error page?"
2. "Are the downstream services idempotent, or do we need to implement idempotency keys?"
3. "What's the expected failure mode - transient blips or prolonged outages? This affects whether retry or circuit breaker is the primary pattern."
4. "Is there a shared infrastructure (service mesh, API gateway) that already provides some of these patterns?"
5. "What's the SLA requirement? 99.9% vs 99.99% significantly changes the resilience investment."

---

References & Whitepapers

• Release It! - Michael Nygard (2007, 2nd ed. 2018). The foundational text on stability patterns including circuit breakers, bulkheads, and timeouts.
• Netflix Tech Blog - Making the Netflix API More Resilient. Real-world application of Hystrix patterns at scale.
• Microsoft Azure Architecture - Resilience Patterns. Comprehensive catalog with implementation guidance.
• Hystrix Wiki - How It Works. Detailed explanation of thread isolation, circuit breaking, and request collapsing.
• AWS Architecture Blog - Exponential Backoff and Jitter. Analysis of jitter strategies for distributed retries.
• Google SRE Book - Chapter on handling overload and cascading failures in distributed systems.

---

Related Topics

• Error Handling Strategies
• Distributed Systems Fundamentals
• Load Balancing
• Monitoring & Observability
• Chaos Engineering
• Service Mesh Patterns