Does your health check use the same path as your traffic?

A health check is a claim, and the claim is only as good as the path the check travels to make it.

Google Cloud's incident report for July 14 states the failure plainly. A configuration update to the foundational network control plane exposed a gap in its routing logic. Hosts misconfigured their routing tables and began dropping traffic bound for the private address space that GCVE Stretched Clusters use to talk between zones. Stretched clusters in australia-southeast2 and europe-west3 lost inter-site communication for ten hours and forty minutes, and the loss triggered VMware HA failovers.

Then the sentence we would put on the first page of a review. Standard health-checking protocols remained functional, because they ran on separate address spaces, and that is what prevented automated detection of the degradation.

What was the check measuring?

Reachability of a control path, which was fine, offered as evidence about a data path, which was not.

The distinction matters because the two paths are almost never the same by accident. Management traffic runs on its own addresses, its own VLANs, sometimes its own hardware, for reasons that are good in isolation: isolation of blast radius, simpler firewall rules, a management plane that survives a saturated data plane. Every one of those reasons also buys the failure mode above.

Why does the drift go unnoticed?

Because green is not an alarm. A probe on a separate path is not broken; it answers its question correctly. It is the inference that breaks, and inferences do not have owners.

The cost of fixing it is real, and we state it rather than skip it. Synthetic checks that traverse the production data path consume the production data path, carry credentials, and can themselves cause the saturation they look for. That is the trade: a probe that can lie to you cheaply, or a probe that tells the truth and costs capacity, credentials and a rate limit.

What we ask

For each health check, name the address space, route and dependencies it uses, then name the ones the customer request uses. Where the two lists differ, that difference is the set of failures the check cannot see.

Then ask what the failover does when the check is wrong. In this incident the automation behaved reasonably given what it believed, and what it believed came from the wrong probe. A failover driven by a mistaken signal is not a safety mechanism. It is a second incident, timed to arrive during the first.