Scaling

What does an availability zone promise?

A London power event in one zone became eight hours of region-wide errors. A briefing on what a zone isolates, and where that isolation quietly leaks.

By TIS Partners · · 2 min

On October 20 a power event in one London availability zone, europe-west2-b, turned into roughly eight hours of regional service errors across Google Cloud, with Lloyds, HSBC, Revolut and around thirty-eight other services degraded (The Register's account). Our question is not why the power failed. Power fails. The question a review would ask is narrower: why did the failure of one zone read as the failure of the region?

Two-color illustration of a grid of cells with a single cell gone dark

What is a zone supposed to isolate?

A zone is sold as a failure domain. The premise you buy is that power, cooling and network inside one zone are independent of the others, so a fault stays put and the region keeps serving from the two that remain. That premise is the whole reason multi-zone deployment costs what it costs. When a single zone's power event surfaces as region-wide errors, the isolation you paid for did not hold, and the interesting work is finding where it leaked.

Where does zonal isolation usually leak?

In our reviews the leak is rarely the compute. It is the thing underneath that was quietly regional: a control plane that schedules across zones and stalls when one stops answering, a regional endpoint fronting zonal capacity, a quorum that loses its majority when a third of its nodes vanish at once. A managed service can be marketed as regional and still carry a zonal dependency its customers never see. The zone boundary held for your pods. It did not hold for the API that places them.

What should a review check before the next one?

Three things, and teams check one. Which of your dependencies are genuinely spread across zones, and which only inherit the label from the region they sit in. Whether your health checks trigger on degraded-but-up, not only on hard down, because a zone browning out for eight hours never flips the binary. And whether you have ever actually run from the surviving zones, or only assumed you could. Multi-region is the expensive answer to this question; multi-zone done honestly is the cheap one, and most estates have bought neither while believing they bought one.

Two-color illustration of a grid of cells with a single cell gone dark

The price of real zonal isolation is duplicated capacity you pay for while it sits idle. The price of skipping it is the outage above, inherited whole from a provider you cannot debug. Pick deliberately. A zone is a promise, and a promise you have never tested is a diagram.