Architecture
What happens when a call has no timeout?
CircleCI's July 2 outage came from connections that never timed out. A slow dependency became an unavailable product, and the review question is which of your calls wait forever.
By TIS Partners · · 3 min
Every system we review contains a set of calls nobody configured. They inherited a client library's default, and a surprising number of those defaults mean "wait".
CircleCI's post-incident report for July 2 is a short, honest account of what that costs. Two ordinary pieces of work collided. An internal maintenance job began deleting data at 13:30 UTC. From 14:40, customer-initiated project deletions arrived in volume. Neither is remarkable on its own. Together they slowed a data service, and by 15:12 the slowness had become an outage: customers could not start pipelines, and some could not log in. Normal service returned at 16:44.
The cause is one sentence, and it is the sentence worth reading twice. Connections between the API layer and that data service were not configured to time out slow responses.
What does a missing timeout actually do?
It converts a latency problem into an availability problem, and it moves the failure one hop upstream, into the component that was healthy.
The data service here did not fail. It got slow. A caller that waits indefinitely on a slow response holds a connection, a thread, or a slot in a pool, and holds it for as long as the slowness lasts. Arrivals continue at their usual rate. The pool drains. Requests that have nothing to do with deleting projects now queue behind requests that do, and the login page, which shares the pool, stops answering.
This is why the visible symptom rarely names the cause. Saturation (a resource with no capacity left to give) propagates toward the customer, while the degradation that started it sits two layers down and never raises an alert of its own.
What is a timeout worth, and what does it cost?
The price on each side, because a recommendation without a cost is marketing.
A timeout buys back the resource. It converts an unbounded wait into a fast, countable error, which is the difference between one degraded feature and a dark product. It costs correctness at the margin: a request that would have succeeded at eleven seconds is now a failure at ten, and if the caller retries without restraint, the timeout you added becomes the load amplifier that finishes the job.
Timeouts are therefore never a single setting. They are a budget, allocated down a call chain, with the outermost value smaller than the sum of the inner ones, and paired with a retry policy that has a cap. The signal a timeout produces is the same signal backpressure carries: the system stating its real capacity, early enough to act on.
What we ask on a review
Three questions, and we have yet to run a review where all three had answers.
Which calls in the request path have no timeout? Not "what is our standard timeout", which is a policy; which connections actually carry one, which is a fact about configuration.
What runs on a schedule against the same dependency as customer traffic? Maintenance jobs, backfills and bulk deletes are load. They are usually invisible in capacity models because nobody counts them as requests.
What is the largest deletion a single customer can request, and does it have a rate limit? The July 2 incident needed both halves. The remediation was both halves too: timeouts, and resource limits on background tasks.
None of these needs a rewrite. They need someone to go and look.