Scaling
What happens when an optimization skips the capacity check?
Google Cloud Storage lost writes in us-central1 for hours in October. The final report reads as a briefing on safeguards that live in a code path.
By TIS Partners · · 2 min
On 8 October, writes to Google Cloud Storage in us-central1 failed or timed out for about twelve and a half hours, then failed again the next morning in us-central1-b for close to six more. Google's incident report, published on 14 October, is unusually plain about the cause. A configuration update to the storage load balancing layer, meant to relieve localized capacity constraints, sent traffic around the capacity-aware routing logic. A disproportionate share of writes landed on a few small storage nodes, which saturated and answered with 503s.
Three details in that report deserve an architect's attention.
Where did the safeguard live?
Capacity-aware routing was a path, and the optimization took a different path. That is the general shape: a safeguard enforced by the normal route protects only the traffic that takes the normal route. Google's own preventative action says as much, promising to enforce capacity constraints so that traffic cannot bypass them.
The question we ask in a review: is your capacity limit an invariant the system checks wherever load is placed, or a property of one code path that a later, well-meant change can step around? Most teams find the second, usually in the code that was added to make things faster.
Why didn't more capacity help?
Engineers tried to add emergency capacity in the region and were blocked by regional capacity limits. Recovery came from draining traffic off the overloaded nodes and reverting changes. Scaling out is a plan only if the capacity to scale into exists at the moment of the incident, in that region, for that tier. We covered the assumptions behind that plan in how capacity models fail; this report adds a clean example of the price.
What did the rollback leave behind?
The second day's failure came from an incomplete rollback of an upload planner feature, which broke load balancing again and overloaded smaller nodes. A rollback is a change. It deserves the same verification as the deploy it reverses, including a check that the system is back in the state everyone believes it is in.
The tradeoff is real on both sides. Enforcing capacity as an invariant costs latency on every placement decision and makes clever optimizations harder to ship. Leaving it in one path costs nothing until the day a config change finds the other path, and then it costs eighteen hours.
Google also plans to validate load balancing changes in smaller regions before large ones. That is staged rollout, and it works when small regions fail the way large ones do. Whether they do is a question about your topology, and one worth answering before the change rather than after.