What happens when the system is presumed correct?

The British Post Office scandal is usually filed under miscarriage of justice, which it is: hundreds of subpostmasters prosecuted over shortfalls that were, in large part, defects in the Horizon accounting system, with the public inquiry's reporting laying out the human cost in detail. Software people should also file it somewhere closer to home. It is the definitive case study in what happens when an institution grants a system the presumption of correctness, and every estate we review contains smaller versions of the same grant.

Strip the case to its system properties and three failures compound. The system's word was treated as evidence without corroboration: a Horizon balance was, institutionally, a fact. The people best positioned to dispute it, operators reporting impossible numbers, were structurally outranked by it: each report was handled as an isolated competence question rather than aggregated into a signal about the system. And the information that would have settled the dispute, known defects, remote-access capabilities, error records, lived on one side of the argument. The victims argued against a database from memory.

Now run our standing question against your own estate: what would have to be true for your system's word to deserve the authority it carries? Three conditions, none of which Horizon met, most of which your critical systems do not either.

Its claims are reconstructible. A balance, a bill, a fraud score can be traced to inputs and rules, by someone outside the team that built it, in bounded time. If reconstruction requires the vendor's goodwill, the system's word is testimony from an interested party.

Disputes are aggregated as telemetry. One operator disputing the system is an anomaly; forty are a defect signal, and the architecture question is whether anyone can see the forty. Horizon's institution processed disputes retail, one career at a time. A dispute register with trend visibility is not bureaucracy; it is the system's error bar, kept by the only population that observes the errors.

And the defect record travels with the output. Every mature system has known issues; the question is whether the people relying on its outputs can see them. An output consumed three departments away from the known-defects list is carrying more authority than its own engineers would grant it, and the gap widens with every reorganization.

The counterargument writes itself: systems are usually right, and institutions that re-litigate every automated number will grind to a halt. Quite so. The presumption of correctness is efficient, which is exactly why it accumulates unexamined. The engineering answer is not less automation; it is calibrated authority, the same design question we put to agents and to every system whose confidence outruns its verification. Authority sized to evidence, disputes as telemetry, reconstruction as a tested property.

Horizon's numbers sent people to prison. Your system's numbers deny claims, price risk, flag fraud, fire alerts at employees. The difference is degree, and degree is not a design principle. The presumption of correctness should be the most reviewed component in any estate, and it is, reliably, the only one that never appears on the diagram.