The honest metric: why most dashboards lie by omission.
A metric is a sentence with most of the words taken out. "Revenue: ↑ 14%" looks like information. It is closer to a headline. A compressed claim that depends entirely on the words somebody chose not to show you. Dashboards rarely lie by stating something false. They lie by omission, and the omissions are predictable enough to turn into a checklist.
The four missing words
Nearly every metric that gives you that vague uneasy feeling is missing one of four things. Train the instinct to ask for them and most dashboards become readable again.
1. The denominator
"500 new signups this month." Five hundred out of what? Out of 600 last month, that is a slump. Out of 300, that is nearly double. The raw count is theatre. The rate is the fact. A number with no denominator is a number that does not want to be compared to anything.
2. The baseline
"Conversion is 3.2%." Compared to what? Last quarter, the industry, your own target, or nothing at all. A metric with no baseline is asking you to supply one from memory, and memory will reach for a flattering one. Every live number should carry its comparison point on its back.
3. The interval
This is the one technical people skip, and it is the most important. "Churn went from 4.0% to 4.6%." Real change, or just Tuesday? With a few hundred users, half a percentage point sits inside the noise. The honest version of that sentence has a band on it: 4.6% ± 0.7%. If the band overlaps the old number, you are reporting weather and calling it climate.
A metric without a confidence interval is a press release. The interval is the part that is honest about how much the metric does not know.
4. The counter-metric
Every metric can be "improved" by sacrificing something it does not measure. Push average session length and you might be making the product harder to leave. Push tickets-closed-per-day and quality slips out the back door. An honest dashboard pairs each metric with the thing it is most likely to cannibalise. A metric standing alone is, more often than not, a metric being gamed.
The checklist
| Before you trust a metric, check… | If it is missing, the failure mode is |
|---|---|
| It has a denominator. A rate, not a raw count. | Mistaking scale for progress. |
| It shows a baseline or comparison. | Grading yourself on a curve you drew. |
| It shows variability. An interval. | Reacting to noise like it is signal. |
| It is paired with a counter-metric. | Fixing one number by breaking another. |
Why this keeps happening
Not malice. Three duller forces, worth naming because you cannot fight a force you have mistaken for a villain.
Dashboards optimise for glance-ability. A confidence interval is hard to render as a big green number. So the interval gets cut, and the metric's honesty about its own ignorance goes with it.
Incentives point at the number, not the truth. Measure someone on a metric and they will improve the metric. Whether they improved the underlying reality is a separate question, and one the dashboard is not asking.
Good news compresses; caveats do not. "Up 14%" fits in a tile. "Up 14%, but it is almost all one enterprise deal, and the band is ±9%" does not. The format selects for the misleading version. Nobody decided that. It just happens.
The honest version
An honest metric tile is uglier and longer. That is the whole problem with it. Something like: "Activation 38% (of 1,240 signups), vs 35% last month against a 40% target; ±4pp; watch alongside 30-day retention, which slipped 2pp." Nobody designs that tile because it does not look like winning. It is the only version you can base a decision on.
If you would rather practise the interval instinct on research papers than dashboards, the same idea aimed at p-values and effect sizes lives in the paper sanity check.