Nobody pressed cancel: timeouts, retries and the query that outlived its caller

Load a report and a chain of programs waits on each other: your app calls a service through an SDK, and the service runs a SQL query on a database. Each link sets its own timeout and retries, and many SDKs retry on their own, some on timeouts too. Nothing makes the layers’ settings agree.

Toy numbers: the app gives each request 10 seconds, its SDK makes up to 3 attempts, and the query needs 25 seconds of database time. At 10 s the app times out and the SDK resends. The database starts a second copy, and the first keeps running, because nothing told it anyone had given up. At 20 s a third copy starts. At 25 s the first finishes, 15 seconds after anyone was listening. At 30 s the app shows its error while the database spends another 15 seconds finishing the other two: 75 seconds of work that reached nobody.

A timeout ends the caller’s wait, and nothing about it travels down the chain. I’ll call work still running after everyone who wanted its result has left an orphan. Orphans become outages two ways. Retries multiply across layers: 3 attempts in your code around an SDK call that also makes 3 put 9 copies on the database. And copies slow each other down, so more attempts time out and start more copies, a loop that past some load outlives the burst that started it.

The chain behind this post: a workflow engine (a scheduler that starts jobs on a timetable and retries failures on its list) starts a cloud function, the wrapper, which the platform runs in a sandbox and kills at a time limit. The wrapper invokes a second function, the worker, and waits; the worker runs one SQL query on Postgres.

In the figure, hover to move through time: a lamp lights on each copy still turning after its caller gave up. Click to wire the brake, a cancel signal from the caller.

The model

Timeouts end waits, not work

To stop the work below a timeout, a signal has to reach the callee: a closed connection it notices, an explicit cancel, or a deadline (a moment after which the answer is useless) it was handed up front. Sending that signal is cancellation. Google’s SRE book says cancellations “should be propagated throughout the entire stack” (Addressing Cascading Failures). In my experience most stacks propagate them as far as the database driver and stop.

Postgres serves each connection with its own process (a backend), which notices a departed client only “at the next interaction with the socket, when it waits for, receives or sends data” (connection settings). A backend mid-scan does none of those until it has rows to send. Two settings change that, both off by default:

• client_connection_check_interval polls the socket while the query runs and aborts if the kernel reports the connection closed (Postgres 14 on Linux; Postgres 15 added macOS, illumos and the BSDs).
• statement_timeout aborts any statement that runs past a wall-clock limit, whatever the client is doing (client settings).

A layer’s budget is the time it has left before its own caller stops waiting.

The field chain broke both halves. The wrapper’s read timeout equalled its own lifetime, so the runtime’s kill always came first, and nothing below learned the wrapper had died. Every layer also treated retries as free, since reads are idempotent, but each copy is another full scan of the same disk.

Reproduce it

A real orphan in Postgres

orphan_pg.py boots a throwaway Postgres 16.2 and starts a 600-million-row cross join. Two seconds in, it sends SIGKILL to the psql client, as a function runtime reaps a sandbox, and polls pg_stat_activity (the live view of every connection) every 50 ms until the backend disappears. One run:

The caller leaves at 2 s; undisturbed, the query ran 20.67 s.

• A, the default, is the field note in miniature: the query finished for nobody and logged nothing, so grepping logs undercounts orphans.
• B stops within one polling interval (up to 1 s by construction), C at the wall-clock limit whatever the client did.
• D is the case people forget. psql stays connected while its caller has given up, so the socket is healthy and the check has nothing to report. That is any middle layer that outlives its caller, and only a deadline handed down, or statement_timeout, stops it.

B’s check also needs the kernel to know the connection closed. A killed local process closes its socket at once; a vanished host (a frozen sandbox, a dropped NAT entry) may never send a FIN, and the kernel learns nothing until TCP gives up. I read the field log’s Connection timed out as that case. The docs suggest tuning tcp_keepalives_idle, tcp_keepalives_interval and tcp_keepalives_count alongside the check; I did not reproduce this case.

The amplification

The retry ladder

Retries hide in places you didn’t configure. boto3, AWS’s Python SDK, defaults to legacy mode, 5 attempts, with ReadTimeoutError on its retry list (boto3 retries guide). sdk_retry_probe.py confirms it: against a local fake endpoint that takes 3 s, a real boto3 Lambda client with a 1 s read timeout sent 5 requests per invoke() by default, 3 in standard mode and 1 with total_max_attempts=1, and the fake function ran to completion every time.

The widget runs the field chain on one time axis; red hatching is database time after the caller gave up. Click the presets in order, each a fix proposed in good faith.

1. “As deployed”: one copy, 17.5 orphaned minutes, one failure.
2. “Read timeout 14 min”, the standard advice, lets the SDK retry. One retry fits in the wrapper’s last minute: 2 copies, 50 orphaned minutes.
3. ”+ retry the timeout ×6” adds the timeout to the workflow’s retry list with MaxAttempts: 6, which counts retries, not attempts (Step Functions error handling): seven wrapper runs, 14 copies, 6 at once, 350 of 455 database minutes spent for callers that had left. It still fails, because a 32-minute query does not fit in a 15-minute budget.
4. “Deadline + one retry layer” turns SDK retries off, has the worker set statement_timeout from its budget, and lets the scheduler retry twice: three copies, never two at once, zero orphaned minutes, 42 database minutes. It still fails; a deadline makes failure cheap, and success takes a smaller scan (fix 5).

The model is generous: each copy takes 32.5 minutes however many share the disk.

Ignoring timing, one scheduled run can start the product of every layer’s attempts. boto3’s default 5, the field workflow’s 7, and one asynchronous hop (Lambda runs a failed async invocation two more times by default, per its async error handling) give 5 × 7 × 3 = 105. Time bounds trim the product (only 2 SDK attempts fit in a 15-minute wrapper with a 14-minute read timeout, so the field chain tops out at 14), but nothing enforces those bounds on purpose.

The math

When a spike becomes permanent

Bronson et al. call this loop a metastable failure: “a trigger causes the system to enter a bad state that persists even when the trigger is removed”, and request retries are “one of the most common failure-sustaining mechanisms” (HotOS 2021). A scheduler is the “uncontrolled source of load” they describe: it adds a run every period whether or not the last one finished.

In metastable_sim.py a run starts every 10 minutes and needs 5 minutes of the disk alone, so the disk is 50% utilized; everything in flight shares the disk equally. Clients give up at 15 minutes and retry up to 3 times. At 1:00 a backfill (catch-up runs) drops 4 extra runs, 20 minutes of work, and the trigger is over. The four policies differ only in what happens on give-up. Bars are goodput (runs that succeed) per half hour; the line counts scans in flight, orphans shaded.

one backfill, four retry policies · goodput and scans in flightsim · seed 7

a · orphan + retries b · cancel + retries c · orphan, no retries d · orphan + 10% budget

backfill at 1:00

runs succeedednone succeededscans in flightof which orphans

 

Without the backfill all four run 6 an hour, one scan in flight at most. With it, (a) orphans plus retries never recovers, climbing past 140 scans in flight by the end of the eighth hour, almost all orphans. (b) cancel on give-up never recovers either, settling at 6 in flight. (c) orphans without retries is back to full throughput in the next 30-minute bucket. (d) orphans with a token-bucket retry budget (one retry per ten first attempts) sent a single retry and recovered.

Write ρ for utilization (the disk share the scheduled runs need on a normal day) and R for retries per run. In the bad state every attempt times out, so the offered load is (R + 1)ρ: at ρ = 0.5 and R = 3 that is 2.0, twice what the disk can do. The bad state feeds itself whenever (R + 1)ρ > 1, so with three retries any utilization above 0.25 is in Bronson et al.’s vulnerable state: healthy, possibly for months, until a trigger knocks it over.

Cancellation bounds the pile without ending the loop. Little’s law (Little, 1961): the average number in a system equals arrival rate times time spent. Runs arrive at λ = one per 10 minutes with R + 1 = 4 attempts each, and a cancelled attempt lives T = 15 minutes, so λ(R + 1)T = 6 attempts are in flight. Each gets a sixth of the disk for 15 minutes, 2.5 minutes of work against the 5 it needs, so every attempt fails. Without cancellation the backlog grows by (R + 1)ρ − 1 = 1.0 disk-minute every minute: the climbing line in (a). A retry budget ends the loop: capping retries at a fraction b of first attempts caps the offered load at (1 + b)ρ, and b = 10% moves the vulnerable range up to ρ above 0.91.

Rerunning the backfill at other utilizations, (a) and (b) stayed collapsed at every ρ from 0.35 to 0.9, (c) never did, and (d) collapsed only at 0.9, at its predicted edge. At 0.3 and below no run passed 15 minutes, so no retry fired: the trigger must be big enough to reach the bad state, and the size needed shrinks as utilization grows, which is why these outages look like bad luck.

Field numbers

What one copy costs

Provisioned IOPS is a paid-for ceiling on a disk’s I/O operations per second; queue depth is reads sent to the disk and not yet completed.

Before the increase, read IOPS flat-topped at the limit with a deep queue: saturation. After it the queue was still deep. Little’s law tells the two apart: a deep queue means fast arrivals or slow reads, and latency says which. Pick a peak sample and compare the depth the law predicts with the depth the volume reported; they match within 7%:

little's law · is a deep disk queue saturation?calculator

peak sample 1 peak sample 2

reads / s latency, ms measured queue depth

predicted, L = λW measured

That is throughput, not saturation: reads averaged under 3 ms, there were just many of them. More IOPS would not have made the scan read less; the bottleneck was how much it read, and how many times.

The rewrite got 1.52× against the test runs’ 2 to 3×. Two hypotheses for the gap, neither verified: the test runs were warm-cache repeats, and production had three overlapping scans contending for the same buffers (consistent with the drop in IPC:BufferIO). Fewer columns shrink what the sort carries, not which heap pages the scan reads, and the 3× topology and 21× over-read were untouched.

Prior art

Where this sits in the literature

Huang et al., Metastable Failures in the Wild (OSDI 2022), report that “at least 4 out of 15 major outages in the last decade at Amazon Web Services were caused by metastable failures”.

The SRE book’s Handling Overload caps retries at three attempts per request and 10% of a client’s requests, and says requests “should only be retried at the layer immediately above the layer that is rejecting them”. Addressing Cascading Failures gives the product rule and asks that every RPC in a request tree share one propagated deadline. Marc Brooker’s Timeouts, retries, and backoff with jitter recommends retrying “at a single point in the stack” and limiting retries with a token bucket. gRPC deadlines cancel the server-side call when the client’s deadline passes, then add the sentence this post is about: “the server application is responsible for stopping any activity it has spawned to service the RPC.”

None of them reaches the last hop: Postgres ships its cancellation settings off, and its docs advise against a global statement_timeout, which is why fix 2 sets it per role.

What to do

Fixes, ranked

fix	what it removes	cost
1. propagate a deadline	orphans at every layer, including the database	a parameter in every call; clock margins
2. make the database cancel	orphans whose client died or ran out of time	per-role settings; keepalive tuning
3. retry at one layer, with a budget	retry amplification and the metastable loop	slower recovery from genuine blips
4. single-flight per job and window	a retry stacking on an orphan	a lock; breaks under transaction pooling
5. scan once, fan out, use watermarks	the 3× topology and silent gaps	a state table and a little orchestration

1. Propagate a deadline end to end

Each layer derives its timeout from what is left of its caller’s budget, minus a margin to report a clean error, and passes the remainder down as a duration (as gRPC’s grpc-timeout header does, wire protocol) so clock skew doesn’t matter. At the bottom, the remainder becomes statement_timeout for that one transaction:

import json, time
import boto3
from botocore.config import Config

MARGIN_S = 10  # time to fail cleanly before the platform kills us

class DeadlineExceeded(Exception):
    pass

def wrapper_handler(event, context):
    budget = context.get_remaining_time_in_millis() / 1000 - MARGIN_S
    lam = boto3.client("lambda", config=Config(
        read_timeout=budget,                # inner wait < outer lifetime
        retries={"total_max_attempts": 1},  # retries belong to one layer, and it isn't this one
    ))
    resp = lam.invoke(FunctionName=WORKER, InvocationType="RequestResponse",
                      Payload=json.dumps({**event, "budget_s": budget}))
    return json.loads(resp["Payload"].read())

def worker_handler(event, context):
    deadline = time.monotonic() + min(event["budget_s"],
                                      context.get_remaining_time_in_millis() / 1000) - MARGIN_S
    ms = int((deadline - time.monotonic()) * 1000)
    if ms <= 0:
        raise DeadlineExceeded("no budget left; not starting the scan")
    with conn.cursor() as cur:  # inside a transaction, so is_local=true scopes it to this transaction
        cur.execute("SELECT set_config('statement_timeout', %s, true)", (f"{ms}ms",))
        cur.execute(SCAN_SQL, params)
        return cur.fetchall()

set_config() takes a bind parameter, which SET doesn’t. The worker’s statement timeout now fires before the wrapper’s read timeout, so the wrapper reports an error it chose and the database stops at the same moment.

2. Make the database cancel on its own

A backstop for every client that doesn’t do fix 1, set per role rather than globally:

ALTER ROLE batch_reader SET statement_timeout = '14min';                -- below every caller's budget
ALTER ROLE batch_reader SET client_connection_check_interval = '10s';   -- notice dead clients mid-query
ALTER ROLE batch_reader SET idle_in_transaction_session_timeout = '1min';
ALTER ROLE batch_reader SET tcp_keepalives_idle = 60;                    -- find hosts that vanished
ALTER ROLE batch_reader SET tcp_keepalives_interval = 10;
ALTER ROLE batch_reader SET tcp_keepalives_count = 3;

All six are user-settable (I checked pg_settings.context on 16.2) and take effect on the role’s next connection. Postgres 17 adds transaction_timeout for whole transactions (release notes).

3. Retry at exactly one layer, with a budget

Retry only at the layer that has the whole budget and knows whether the result still matters, usually the scheduler; every layer below fails fast with an error that says why. Give that layer the token bucket from policy (d): each first attempt adds a tenth of a token, each retry spends a whole one, and an empty bucket means no retry. In a managed workflow engine, use a single retrier on your own deadline error with a small MaxAttempts, a long IntervalSeconds and "JitterStrategy": "FULL" (randomized waits, so runs that failed together don’t retry together), and alarm when it fires. Never retry a timeout at a layer that cannot cancel the attempt it is abandoning.

4. Single-flight per job and window

A retry should join or skip the run already happening, not start another. A session-level advisory lock (a lock Postgres keeps on a number you choose, held as long as the session) keyed on the job and its window does this, and stays held while an orphaned backend is still scanning:

-- first statement of the worker's session; false means a copy is already running
SELECT pg_try_advisory_lock(hashtext('scan:' || $1));  -- $1 = job name + window start

It returns at once instead of waiting (docs). A transaction-pooling proxy (which hands your connection to another client after each transaction) breaks session-level locks, and the key must name the window or a slow run blocks the next hour’s.

5. Scan once, fan out, and keep a watermark

One scan per window that hands rows to every consumer removes the 3× multiplier, and a watermark makes failures heal instead of leaving gaps:

start = load_watermark("threads")                    # last fully processed instant
end = min(now() - SETTLE, start + MAX_WINDOW)        # catch up in bounded chunks
rows = scan(start, end)                              # one scan for every consumer
for consumer in consumers:
    consumer.handle([r for r in rows if consumer.wants(r)])
save_watermark("threads", end)                       # only after every consumer succeeded

A failed run leaves the watermark in place, so the next run covers the gap; bounded chunks keep each scan inside the caller’s budget.

Detect it

Catch it on day one

Poll for orphans directly: anything running longer than your longest caller timeout is work nobody is waiting for, including run A’s silent kind:

SELECT pid, application_name, client_addr,
       now() - query_start AS running_for,
       wait_event_type, wait_event, left(query, 80) AS query
FROM pg_stat_activity
WHERE backend_type = 'client backend'
  AND state = 'active'
  AND now() - query_start > interval '15 minutes'  -- your longest caller timeout
ORDER BY running_for DESC;

Alarm when it returns rows; pg_cancel_backend(pid) is the manual remedy.

Count copies per job: set application_name to 'scan:<job>:<window>' and alarm when a key has more than one active backend in pg_stat_activity.

Trend log lines like connection to client lost, knowing they miss orphans like run A.

Test the cancel path in CI, as orphan_pg.py does: kill a client mid-query and assert the backend is gone within seconds.

Alarm when maximum duration approaches the caller’s budget: the first field note’s job gave days of warning before it failed.

Try it

Try it

Scripts are in experiments/nobody-pressed-cancel/, stdlib-only except the SDK probe; each merges its results into results.json, which the widgets read.

cd experiments/nobody-pressed-cancel
python3 -I amplification.py      # ladder presets, 540-point grid, attempt products
node check_ladder.mjs            # the widget's JS model matches the Python twin
python3 -I metastable_sim.py     # four policies, spike or calm, utilization sweep
python3 -I field_math.py         # Little's law, the scan's I/O, timeline, topology

# real Postgres (14+ on Linux, 15+ on macOS) without Docker: the pgserver wheel ships binaries
python3 -m venv /tmp/pgvenv && /tmp/pgvenv/bin/pip install pgserver boto3
PGBIN=$(/tmp/pgvenv/bin/python -c "import pgserver,os;print(os.path.join(os.path.dirname(pgserver.__file__),'pginstall','bin'))")
python3 -I orphan_pg.py --pgbin "$PGBIN"     # scenarios A to D on a throwaway cluster
/tmp/pgvenv/bin/python -I sdk_retry_probe.py  # how many requests one invoke() sends

Then point the orphan query at your own database and see which row of the table you’re in.