Scaling
What does a black start teach a software architect?
In April, the Iberian grid went dark and was rebuilt generator by generator in a day. The restart discipline is one most software estates have never priced.
By TIS Partners · · 2 min
On April 28, the Iberian peninsula lost its power grid in seconds and spent the rest of the day performing something grid engineers have rehearsed for decades: a black start. Restore a seed generator with no external power, energize an island of grid around it, synchronize the islands, grow. By late evening most of the peninsula was lit. The event will be studied for years for its causes. We keep thinking about the recovery.
What would a black start of your estate look like? Not a failover; a start from nothing: every service down, every cache cold, every queue empty or, worse, full. The question sounds theatrical until an incident hands it to you. Regional cloud events, expired root credentials and bad global config pushes have each, in recent memory, put estates into states their operators had never once practiced leaving.
Grid operators know three things about restart that software teams mostly do not know about their own systems. Which components can start with no dependencies (their seed generators). The exact order in which the rest must follow, because energizing in the wrong order trips the protection again. And how much load each island can accept before synchronizing with the next, because demand arriving faster than generation is what caused the trip.
Every one of those has a software translation, and almost no estate can answer the translated questions. Which of our services starts clean with zero peers up? What is the dependency order, not as drawn, but as actually enforced by startup checks? And what happens when the retry storm of every client we have hits the first service to wake, which is the software spelling of demand outrunning generation?
The tradeoff is stated easily. A restart runbook with an enforced order and load shedding at the front door costs a few weeks to build and a day a year to rehearse. Not having one costs nothing until the day it costs everything, and that pricing structure, small certain premium against rare total loss, is one the grid people resolved in favor of the premium long ago. They rehearse black starts on schedule, in daylight, with the same crews who will run the real one.
The peninsula was relit in a day because the order existed on paper and in muscle. Ask the question of your own estate this quarter: if everything stopped, what starts first? If the room disagrees, that is the finding.