Metastable Failures: Why Your System Won't Recover After the Problem Is Gone

• The outage that outlived its cause
• Three words
• The engine is almost always retries
• Watch it happen, and not recover
• The shape of the fix
• Conclusion
• References

The outage that outlived its cause

I’ve hit this same outage several times, at different companies, and it almost never gets diagnosed. A brief spike, thirty seconds of something, then it passes. Load returns to normal. And the service stays down.

Not degraded. Down. Full errors for twenty minutes, with traffic well under what it served fine an hour ago and CPU sitting half-idle. Every dashboard says there’s headroom. Nothing is broken. Someone restarts the fleet and it comes straight back.

That restart is why it rarely gets understood. It works, the pages stop, everyone moves on, and the incident gets logged as a fluke or waved off as some mystery deadlock. But the restart is also the tell. If it clears an outage where nothing was actually broken, your system had two stable states, and the spike knocked it into the bad one. That is a metastable failure, and once you’ve seen one you start seeing them everywhere.

Three words

The whole thing fits in three words: trigger, sustaining effect, metastable state.

A healthy system sits in a stable state. It absorbs the usual bumps and settles back to normal. A trigger (a load spike, a deploy, a dependency blip) shoves it into a degraded state. Normally it climbs back out. Sometimes a sustaining effect kicks in instead: a feedback loop that generates enough extra load to hold the system in the bad state long after the trigger is gone.

        \                             /
         \         trigger           /
          \    ( ● )───────▶        /
           \______________/\_______/
             good state        bad state
            (stable, low)    (metastable: a feedback
                              loop holds it here)

Capacity is not the problem. The system has plenty. The problem is that the load pinning it down is being manufactured by the failure itself.

The engine is almost always retries

The sustaining effect is almost always a retry.

Picture the backend slowing down. Requests start crossing the client timeout, and the client does the sensible thing: it retries. Now the backend is handling the original requests plus the retries, so it’s slower still, so more requests time out, so more retries fire. The load crushing the server is generated by clients reacting to the server being crushed.

It’s a closed loop, and it no longer needs the trigger. You removed the match, but the fire is feeding on itself.

The retries aren’t even the only waste. A request that already timed out is often still sitting in the server’s queue, and the server has no idea the caller gave up, so it burns a full slot producing a response nobody is waiting for. At the worst possible moment, capacity goes to answers that get thrown away the instant they’re ready.

Watch it happen, and not recover

You can reproduce the whole arc in about a hundred lines of Python. It’s a minimal skeleton of the kind of faulty service you’d actually hit in prod: an open-loop client, a server with fixed concurrency and a per-request cost, a client timeout, and retries. That short list is all it takes to get metastable behavior.

The demo runs in two modes: baseline, with no protection, and fixed, with the two guards from the next section. Everything else is identical between them:

• Capacity: 2500 req/s (50 concurrent slots, 20ms per request).
• Steady load: 1800 req/s, a comfortable 72% of capacity.
• The trigger: at t=20s, load jumps to 6000 req/s for ten seconds, then drops straight back to 1800.
• Clients: time out at 200ms and retry up to twice.

Goodput is the metric to watch: requests actually served before the client gives up.

The model is one time-stepped loop. Every millisecond:

1. New requests arrive into a FIFO queue.
2. Finished requests free their slot. If the client is still waiting, that one counts as goodput.
3. Free slots pull the next request off the queue.
4. Anything past its 200ms deadline that hasn’t finished is a client timeout, which fires a retry.

The collapse really comes down to two properties this model shares with real systems. The server keeps no deadline of its own, so it will spend a whole slot finishing a request whose client already walked away, draining capacity into responses nobody reads. And a timeout doesn’t replace the original request, it adds a new one, so load climbs exactly when the server is weakest.

The two modes differ by exactly two guards, both in admit(). fixed adds a retry budget (retries capped at 10% of first-attempt traffic each second) and load shedding (reject at once when the queue is deeper than QUEUE_MAX); baseline has neither.

The full model: ~130 lines of stdlib Python, no dependencies

import sys
import heapq
from collections import deque

# ---- system parameters (a small service) --------------------------------
DT            = 0.001    # simulation step, seconds (1 ms)
DURATION      = 90.0     # total simulated seconds
SLOTS         = 50       # concurrent requests the server can process
SERVICE_TIME  = 0.020    # seconds of work per request  -> capacity 2500 req/s
TIMEOUT       = 0.200    # client gives up after this long
MAX_ATTEMPTS  = 3        # 1 original try + 2 retries

# ---- load profile --------------------------------------------------------
BASE_RATE     = 1800     # steady offered load, req/s  (72% of capacity)
TRIGGER_RATE  = 6000     # offered load during the trigger window
TRIGGER_START = 20.0
TRIGGER_END   = 30.0     # a 10-second shove, then back to BASE_RATE

# ---- protections (only active in "fixed" mode) ---------------------------
RETRY_RATIO   = 0.10     # retries may be at most 10% of first-attempt traffic
QUEUE_MAX     = 250      # shed (reject fast) when the queue is deeper than this

CAPACITY = SLOTS / SERVICE_TIME  # 2500 req/s


class Req:
    __slots__ = ("deadline", "attempts", "done", "failed")
    def __init__(self, deadline, attempts):
        self.deadline = deadline
        self.attempts = attempts
        self.done = False     # completed in time (good)
        self.failed = False   # client already gave up (timed out or shed)


def offered_rate(t):
    return TRIGGER_RATE if TRIGGER_START <= t < TRIGGER_END else BASE_RATE


def run(mode):
    shed_on   = (mode == "fixed")
    budget_on = (mode == "fixed")

    queue = deque()          # waiting requests (FIFO)
    slots = []               # min-heap of finish_time for in-service requests
    timeouts = []            # min-heap of (deadline, seq, Req) for client timeouts
    seq = 0                  # tie-breaker so heap never compares Req objects

    win_firsts = 0           # per-second window counters for the retry budget
    win_retries = 0
    sec_offered = 0          # per-second reporting counters
    sec_goodput = 0

    rows = []
    frac = 0.0               # fractional-arrival accumulator
    steps = int(DURATION / DT)

    def admit(req, is_retry):
        # returns True if the request entered the queue, False if shed
        nonlocal win_firsts, win_retries, sec_offered
        if is_retry:
            if budget_on and win_retries >= RETRY_RATIO * (win_firsts + 1):
                return False          # retry budget exhausted -> give up
            win_retries += 1
        else:
            win_firsts += 1
        if shed_on and len(queue) >= QUEUE_MAX:
            return False              # load shed -> reject fast
        sec_offered += 1
        queue.append(req)
        heapq.heappush(timeouts, (req.deadline, next_seq(), req))
        return True

    def next_seq():
        nonlocal seq
        seq += 1
        return seq

    for step in range(steps):
        t = step * DT

        # 1. new first-attempt arrivals this step (rate -> count via accumulator)
        frac += offered_rate(t) * DT
        n = int(frac)
        frac -= n
        for _ in range(n):
            admit(Req(deadline=t + TIMEOUT, attempts=1), is_retry=False)

        # 2. complete any in-service requests whose service time has elapsed
        while slots and slots[0][0] <= t:
            _, _, req = heapq.heappop(slots)
            if not req.failed:
                req.done = True
                sec_goodput += 1      # completed before the client gave up
            # else: wasted work -- the client already retried elsewhere

        # 3. fill free slots from the queue head (server keeps no deadline;
        #    it will happily burn a slot on a request nobody is waiting for)
        while len(slots) < SLOTS and queue:
            req = queue.popleft()
            heapq.heappush(slots, (t + SERVICE_TIME, next_seq(), req))

        # 4. fire client timeouts; each one may spawn a retry
        while timeouts and timeouts[0][0] <= t:
            _, _, req = heapq.heappop(timeouts)
            if req.done:
                continue              # it finished in time; no timeout
            req.failed = True
            if req.attempts < MAX_ATTEMPTS:
                admit(Req(deadline=t + TIMEOUT, attempts=req.attempts + 1),
                      is_retry=True)

        # 5. once per simulated second, record and reset the windows
        if (step + 1) % int(1.0 / DT) == 0:
            rows.append((round(t + 1), sec_offered, sec_goodput, len(queue)))
            sec_offered = sec_goodput = 0
            win_firsts = win_retries = 0

    return rows


def summarize(mode, rows):
    with open(f"{mode}.csv", "w") as f:
        f.write("t,offered,goodput,queue\n")
        for r in rows:
            f.write(",".join(str(x) for x in r) + "\n")

    def row(sec):
        return next(r for r in rows if r[0] == sec)

    pre, mid, after, settled = row(19), row(25), row(60), row(89)
    print(f"[{mode}] capacity            : {CAPACITY:.0f} req/s")
    print(f"[{mode}] steady goodput      : {pre[2]} req/s (offered {pre[1]})")
    print(f"[{mode}] goodput mid-trigger : {mid[2]} req/s (offered {mid[1]})")
    print(f"[{mode}] @ t=60s goodput     : {after[2]} req/s (offered {after[1]})  <- trigger gone 30s ago")
    print(f"[{mode}] @ t=89s goodput     : {settled[2]} req/s (offered {settled[1]})")
    print(f"[{mode}] recovered on its own: {'yes' if settled[2] >= 0.95 * BASE_RATE else 'NO'}")


def main():
    mode = sys.argv[1] if len(sys.argv) > 1 else "baseline"
    if mode not in ("baseline", "fixed"):
        print("usage: python3 sim.py [baseline|fixed]", file=sys.stderr)
        sys.exit(2)
    summarize(mode, run(mode))


if __name__ == "__main__":
    main()

python3 sim.py baseline

[baseline] capacity            : 2500 req/s
[baseline] steady goodput      : 1800 req/s (offered 1800)
[baseline] goodput mid-trigger : 0 req/s (offered 18000)
[baseline] @ t=60s goodput     : 0 req/s (offered 5400)  <- trigger gone 30s ago
[baseline] @ t=89s goodput     : 0 req/s (offered 5400)
[baseline] recovered on its own: NO

Three numbers here are worth keeping straight. Goodput is what the server finishes in time. Offered, shown in parentheses on each line, is everything clients send: real requests plus retries. Capacity is the ceiling the server can never beat, a flat 2500 req/s.

During the trigger, clients offer 18000 req/s (6000 real requests, each retried twice) at a server that tops out at 2500. Goodput: zero.

Then the trigger ends and real demand drops back to 1800, comfortably under 2500. It still never recovers. Look at the t=60s and t=89s lines: goodput is 0, and offered load is stuck at 5400 req/s. Real demand (1800) sits well below capacity (2500), and yet retries alone push offered load to 5400, more than double what the server can serve. So the queue never drains, so every request keeps timing out, so the retries keep coming. Nobody is sending abusive traffic. The system is manufacturing its own overload and holding itself underwater.

The shape of the fix

The fix is not more capacity. You cannot outscale a feedback loop; doubling the servers just moves the cliff. You have to break the loop, and two changes do most of the work.

Cap the retries. A retry budget limits retries to a fraction of live traffic, say 10%, instead of letting every failure multiply. When the budget is spent, clients fail fast rather than piling on. This is the change that matters most: it puts a ceiling on the amplification, so a struggling server can never be handed 3x its load by its own clients.

Shed load at the door. When the queue is deeper than the server can drain within the timeout, reject new work immediately with a fast error. A cheap rejection beats accepting a request, spending 20ms on it, and delivering the answer after the caller has already left. Shedding keeps the queue short enough that whatever you accept, you finish in time.

Plain exponential backoff, worth saying, is not enough on its own. It slows the loop but doesn’t cap it, and without jitter it just re-synchronizes every client to retry on the same beat. Backoff is table stakes. The budget and the shed are what actually bound the system.

python3 sim.py fixed

[fixed] capacity            : 2500 req/s
[fixed] steady goodput      : 1800 req/s (offered 1800)
[fixed] goodput mid-trigger : 2500 req/s (offered 2500)
[fixed] @ t=60s goodput     : 1800 req/s (offered 1800)  <- trigger gone 30s ago
[fixed] @ t=89s goodput     : 1800 req/s (offered 1800)
[fixed] recovered on its own: yes

Same trigger, everything else identical. During the spike the server pins at 2500 req/s, its real ceiling, and sheds the rest. It still can’t serve 6000. But it serves everything it accepts, so goodput holds at capacity instead of collapsing to zero. The moment the trigger passes it is back to 1800, because there was never a storm to sustain.

The retry budget and load shedding are two sides of one idea: bound the work the system will attempt. The client bounds what it sends; the server bounds what it accepts. Either alone helps; together they close the loop from both ends.

Conclusion

Metastability is not about the trigger. You will never predict or prevent every spike, deploy, or dependency blip, so stop trying to. What you control is whether your recovery mechanisms are self-damping or self-amplifying. Build the loops so a struggling system sheds load instead of manufacturing more, and the spike that used to mean a twenty-minute outage becomes a ten-second dip nobody pages you for.

References

• Metastable Failures in Distributed Systems (Bronson et al., HotOS 2021)
• Marc Brooker: Metastability and Distributed Systems
• Timeouts, retries, and backoff with jitter (Amazon Builders’ Library)