How do you read a latency histogram?

The most expensive number in observability is the average latency, because it is the one number guaranteed to describe no actual request. Distributions are what systems produce; the histogram is the system's handwriting, and shapes are legible once you have read a few hundred of them.

What do the shapes say?

A clean single peak with a short tail is a healthy service doing one kind of work. Two peaks is two populations wearing one metric: cache hits and misses, two query plans, two customer tiers, one endpoint serving both a lookup and a report. The review question for bimodality is always "which population is which," because averaging them produces a number that will misdirect a capacity model for a year.

A long right tail is queueing somewhere: a pool, a lock, a GC pause, a retry storm amplifying its own delay. Tails are where users live. At any real fan-out, the tail stops being rare: a page that touches 100 services experiences someone's p99 on essentially every load, the arithmetic Dean and Barroso made canonical in The Tail at Scale. This is why we read p50, p95, p99 and p999 as different instruments: p50 is the experience you designed; p99 is the experience you actually sell; p999 at fan-out is the experience your best customer gets at 2 p.m. on your busiest day.

What lies do histograms tell?

Three, recurring. Bucket boundaries: a histogram with a final bucket of "1s+" hides the difference between 1.1 seconds and 30, and pre-aggregated percentile math on coarse buckets can invent improvements no user felt; check the bucket layout before trusting any p99 chart. Survivorship: timeouts that kill requests at 10 seconds remove them from the latency data entirely, so the distribution improves as the failures worsen; read latency next to error rate or read fiction. And window averaging: a p99 averaged over a day flattens the 4 p.m. incident into a rounding error, while the pager remembers it fine.

What would we have the team do?

Three habits, cheap ones. Plot distributions, not lines of percentiles, for the five endpoints that matter; shapes carry diagnoses that percentile lines smooth away. Segment before averaging: per customer tier, per region, per query class, because every interesting system is several systems sharing a name. And when a latency number appears in a review document, attach its shape: "p99 340ms, unimodal, short tail" and "p99 340ms, bimodal, second peak at 2s" describe different machines that happen to share a percentile.

The histogram is the rare instrument that gets more honest under load. Learn its handwriting, and the system will tell you where it hurts before the customers volunteer to.