Diligence

How deep does your dependency graph go?

On June 12 a Google Cloud failure took down Cloudflare services, which took down their customers. Three levels of transitivity, one afternoon. A briefing.

By TIS Partners · · 3 min

On June 12, Google Cloud suffered a global outage. Within minutes, a large set of Cloudflare services failed with it, and Cloudflare's customers, some of whom had chosen Cloudflare precisely to reduce their dependence on any single cloud, failed in turn. Cloudflare's postmortem is admirably direct: their Workers KV service kept its source of truth in storage "backed by a third-party cloud provider", the provider went down, and KV was a critical dependency for Access, WARP, Gateway and a dozen other products. Three levels of transitivity, one afternoon. Independent analyses traced the blast radius across the wider internet.

We take no pleasure in the example; Cloudflare's engineering reputation is earned, and their postmortem candor is the reason the lesson is teachable at all. That is precisely what should worry a reader: this happened to a company whose core competence is infrastructure. The question for everyone else follows.

do you know your level-two dependencies?

Every diligence exercise we run includes a dependency inventory, and every inventory stops, by default, at level one: the vendors you pay. The June 12 pattern lived at level two, the vendors your vendors depend on, and level two is where the modern concentration actually sits. Three hyperscalers host a remarkable share of the SaaS industry's control planes. Your five independent vendors may be one region failure from being one vendor.

Level two is discoverable, imperfectly but usefully. Some vendors publish their subprocessors for privacy reasons; the same lists read as dependency documents. Status-page archaeology works too: when a hyperscaler has a bad day, note which of your vendors have one simultaneously. June 12 generated a fresh dataset. We suggest an hour with your vendor list and that day's status pages. The correlations you find are your real topology, observed rather than claimed.

what would have to be true for redundancy to be real?

Redundancy claims deserve the practice's standing question. Two vendors are redundant only if they fail independently, and independence is the thing level-two concentration quietly removes. A failover DNS provider hosted on the same cloud as your primary CDN's control plane is a diagram, not a redundancy. The check is not expensive: for each pair of components you believe redundant, name the shared dependencies that would have to be absent. If nobody can name them either way, the redundancy is unverified, which for planning purposes is the same as absent.

what did the recoveries have in common?

The second lesson of June 12 is about failure modes rather than graphs. Cloudflare's affected services did not merely slow; the postmortem describes cold reads and writes failing outright, because the dependency was a source of truth rather than an accelerant. Compare the services across the industry that degraded gracefully that day: they had treated remote state as cache, kept local capability to serve stale data, and shed the features that needed freshness. Same afternoon, same upstream failure, different architecture, different customer experience.

The distinction to carry into your own review: for each external dependency, is it load-bearing or load-lightening? A dependency you can serve stale from is a performance feature; a dependency you cannot answer without is a component of your availability math, and it inherits its own level-two graph into yours. Most estates have never sorted their dependencies into those two piles. The sorting is a whiteboard afternoon and changes the architecture roadmap more reliably than most quarters of feature planning.

what do we actually recommend?

Three actions, sized. This month: the vendor-correlation exercise above, one hour, plus the load-bearing/load-lightening sort for your top ten dependencies, one afternoon. This quarter: for the load-bearing set, a written answer to "what do we serve when this is down", with the honest answer "nothing" recorded where true, because a recorded nothing is a decision and an unrecorded one is a surprise. This year: for the one or two dependencies whose honest answer was nothing and whose failure is intolerable, fund the stale-serving path, and test it by turning the dependency off on purpose, in daylight, on a schedule you chose.

The June 12 outage lasted under three hours. The interesting number is different: how many affected companies could have drawn, in advance, the path by which a Google Cloud storage failure would reach their customers? On the evidence of that afternoon, very few had tried. The graph exists whether or not you draw it. Drawing it is the cheap part, and June supplied the tracing paper.