“Customer renewal increased by 50%.”
That could mean renewal went from 2% to 3%. Or it could mean renewal went from 40% to 60%. Same sentence, same calculation, very different practical meaning. So if the statistic is accurate in both cases, what does calling it “true” actually tell us?
Numbers can mislead without being fabricated, because accuracy is only one part of statistical meaning. This article, a companion to the Reclaiming Rhetoric video, looks at how a statistic can be completely accurate and still create an impression the evidence does not fully justify, and at a short audit you can run on any number that arrives with a conclusion attached.
Foundations Series · Volume 2
How to Analyze Any Argument
A visual guide to claims, evidence, assumptions, and reasoning.
Three layers of a statistical claim
It helps to separate three layers of a statistical claim:
- Arithmetic accuracy. Was the number calculated correctly?
- Representational adequacy. Does the number fairly summarize what happened?
- Inferential adequacy. Does it support the conclusion the audience is being encouraged to draw?
A correct calculation does not automatically produce a complete representation, and a complete representation does not automatically prove the conclusion attached to it. Most public arguments about whether a statistic is “true” are stuck on layer one, while the real problem lives in layers two and three.
None of this makes statistics uniquely deceptive. Ordinary language also selects, compresses, and frames. Statistics simply carry an additional aura of precision. A number looks like a measurement of the world rather than a choice about how to describe it.
A statistic does not merely report a quantity. It selects a relationship.
Relative vs. absolute: four true descriptions of one result
Take a fictional subscription company that announces: “Our new onboarding system increased customer renewal by 50%.” Say renewal actually went from 20% to 30%. Every one of these statements is true:
- It's a 50% relative increase.
- It's an increase of 10 percentage points.
- It's 10 additional renewals per 100 customers.
- Seven out of ten customers still did not renew.
None, by itself, is the complete meaning of the result. A 50% increase tells you how the new number relates to the old number. It does not tell you whether either number was large.
Relative figures are genuinely useful. They answer a legitimate question: how big was the change compared with where things started? And neither the percent change nor the percentage-point change is “the real number.” Each answers a different question.
But research on risk communication has repeatedly found that relative and absolute formats can create different impressions of the same result. A Cochrane systematic review by Akl and colleagues (2011) on presenting health risks found that people tend to perceive a risk reduction as larger, and are more persuaded by it, when it is presented as a relative reduction rather than as an absolute reduction or a “number needed to treat.” Risk-communication researchers widely recommend giving absolute numbers and baselines alongside relative changes for exactly this reason.
Without the baseline, the percentage may be precise but not yet informative.
The denominator: 30% of whom?
Next, the company reports: “30% of customers renewed.” Thirty percent of whom? Perhaps “customers” means only the users who completed every onboarding step. What happened to the people who stopped halfway? Were they excluded? Were they different from the people who stayed?
Picture a narrowing funnel:
- 10,000 people register.
- 6,000 begin onboarding.
- 3,000 complete it.
- 900 renew.
“30% renewed” is accurate among the people who finished: 900 of 3,000. Among everyone who registered, it's 9%: 900 of 10,000.
Neither denominator is automatically correct. The right one depends on the question. If you're asking whether onboarding works for the people who complete it, completers make sense. If you're asking how the business is doing overall, they leave a lot out. And since the people who finish onboarding are probably more motivated to begin with, even the completer figure may say as much about who finishes as about what the onboarding does.
The rhetorical problem is letting the audience assume a broader denominator than the analysis actually used.
The numerator gets the headline. The denominator decides what population the headline is really about.
The selected comparison
Now suppose the company compared its strongest month under the new system with an unusually weak month under the old one. Both values may be genuine. But:
- What happens across the entire quarter?
- How does the new system compare with a typical month?
- Did anything else change, such as pricing, seasonality, or a marketing push?
- Was this comparison chosen before or after the favorable result appeared?
The statistic doesn't become false because the comparison was selected. But a selected comparison can make an ordinary fluctuation look like a decisive trend.
A comparison does not merely reveal a difference. It chooses which difference the audience will see.
People often choose comparisons because they're available, familiar, flattering, or easy to explain. Selection can be innocent, habitual, or strategic. The analytical task is the same regardless of motive: ask what the comparison would look like if a different, equally reasonable baseline had been chosen. Researchers guard against this by specifying comparisons in advance; when that hasn't happened, treat the chosen comparison as one possibility among several.
The chart: the scale is part of the argument
Then there's the chart. Imagine two bar charts showing the same values: 20% and 30%. The first y-axis begins at zero. The second begins at 18%. Now the second chart's taller bar looks several times taller than the shorter one, instead of one and a half times. Nothing in the data has changed. The apparent magnitude has.
In a set of experiments presented at CHI 2020, Michael Correll, Enrico Bertini, and Steven Franconeri found that truncated y-axes led viewers to judge differences as larger, and that the exaggeration persisted even when the truncation was visually signaled to viewers.
But not every non-zero baseline is illegitimate. The researchers themselves caution against treating zero as a universal rule: a narrow scale can reveal meaningful variation that a zero-based axis would flatten. Think of body temperature, where a chart starting at zero would make a serious fever look like nothing. The question is whether the scale matches the size of difference that matters for the conclusion.
A chart does not only show the numbers. It gives the numbers visual proportions.
Charts are a form of visual rhetoric, and like screenshots, they can feel like direct evidence even when they embed a choice about how to see.
Rebuild the claim
So take away the headline and rebuild the claim with its context included:
During the strongest month of the pilot, renewal among customers who completed the new onboarding process rose from 20% to 30%, compared with the selected earlier month.
That is less exciting. It is also far more informative. Every phrase that was added (strongest month, pilot, who completed, from 20% to 30%, selected earlier month) is a piece of information that changes what the audience should conclude.
You could also rebuild it for a different audience. A board asking about the whole business might need: “Across all 10,000 registrants, 9% renewed; among the 3,000 who completed onboarding, 30% did.” A product team asking whether the new flow helps might need the full quarter, not the best month. The point is not that there is one honest sentence. It is that an honest sentence matches the denominator and comparison to the question actually being asked, and says which ones it used.
A five-question statistic audit
From there, you can run a quick audit on any number that arrives with a conclusion attached:
- What is the baseline? A relative change needs a starting point.
- What are the raw numbers? How many people or events does the percentage represent?
- What is the denominator, and who was excluded?
- What comparison and time period were selected? Would a typical period tell the same story?
- Does the visual scale exaggerate or minimize the difference?
Underneath it all sits a practical question: what would I still need to know before drawing the conclusion this statistic seems to invite? And one deeper question: what conclusion does the statistic support, and what stronger conclusion is it merely being used to suggest? That gap between support and suggestion is where most misleading-but-true statistics live. It's the statistical version of a move discussed in why a story can outweigh a statistic: evidence that is accurate for what it is, asked to carry more than it can.
More true-but-misleading patterns
Baselines, denominators, comparisons, and scales are the four big ones in the video, but the same logic applies to several other familiar patterns.
- Average vs. typical. A mean can be pulled far from the typical case by a few extreme values. “Average customer spend rose” might reflect three very large accounts while most customers spent less. Ask for the median or the distribution.
- Cumulative totals. A running total almost always goes up. A chart of “total renewals since launch” can look like steady success even if monthly renewals are falling.
- Small samples. A 50% jump from 4 to 6 renewals is arithmetically identical to a jump from 400 to 600, but far less likely to reflect a stable pattern.
- Survivors only. Studying only the customers still around a year later can make any program look effective, because the people it failed are no longer in the data. This is a denominator problem in disguise.
Each of these can be entirely accurate. Each can also leave the audience with an impression that a fuller description would correct.
Where you'll meet these numbers
You will run into these patterns constantly: in company announcements and investor updates, product marketing (“2x faster!”), health headlines about risk doubling, political claims about crime or employment, and dashboards at work. The context changes, but the audit does not. In each case, ask what the number was compared to, out of how many, over what period, and drawn on what scale. A thirty-second check is often enough to tell whether a headline number means what it seems to mean. And when the information you need simply isn't provided, that absence is itself worth noticing.
Compression is not the enemy
None of this means rejecting numbers on sight. A dramatic percentage can represent a dramatic effect. Averages and summaries are necessary. And no single presentation can include every possible detail.
The target is not compression. Communication always compresses. The target is compression that hides information the audience needs to evaluate the invited conclusion. The goal is not to demand every data point before believing anything. It is to identify the missing context that could materially change what the number means.
If you present numbers yourself, the same principle becomes a practice: give the baseline with the relative change, name the denominator, say how the comparison was chosen, and use a scale that fits the difference you want people to judge. Your claims will be less dramatic and much harder to dismiss. For the broader habits that make careful audiences resistant to persuasive-but-thin evidence, see why smart people believe bad arguments.
The takeaway
Return to the original line: renewal increased by 50%. It may be true. But until we know the baseline, denominator, comparison, sample, and frame, we do not yet know what conclusion it deserves.
Statistics do not become rhetorical only when someone falsifies them. The rhetoric begins when someone decides what to count, what to compare, what to leave out, and how large the result should appear. So do not ask only whether the number is accurate. Ask whether the frame is adequate. Because a statistic can be perfectly correct and still leave you with the wrong idea.
To keep going, browse all articles, explore structured lessons on the Courses page, or see the illustrated guides on the Guides page.
Foundations Series · Volume 2
How to Analyze Any Argument
A visual guide to claims, evidence, assumptions, and reasoning. Stop reacting. Start diagnosing.
A visual method for seeing where an argument actually works, and where it does not. Name the claim, test the evidence, and find the assumption holding it all together.
Sources and further reading
- Akl, E. A., Oxman, A. D., Herrin, J., et al. (2011). Using alternative statistical formats for presenting risks and risk reductions. Cochrane Database of Systematic Reviews, CD006776
- Correll, M., Bertini, E., & Franconeri, S. (2020). Truncating the y-axis: Threat or menace? Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems, 1–12
- Reclaiming Rhetoric, Why a Story Can Outweigh a Statistic
- Reclaiming Rhetoric, Visual Rhetoric
- Reclaiming Rhetoric, How Framing Changes What You Believe
Frequently Asked Questions
How can a true statistic be misleading?
A number can be calculated correctly yet summarize the situation incompletely or support a stronger conclusion than the evidence justifies. Baselines, denominators, comparisons, and scales all shape the impression.
What's the difference between percent change and percentage points?
Going from 20% to 30% is a 50% relative increase but a 10 percentage point absolute increase. Both are accurate; they answer different questions, and the baseline is needed to interpret either.
Why do relative risks sound bigger than absolute risks?
A Cochrane review (Akl et al., 2011) found people perceive risk reductions as larger and more persuasive in relative form. Giving absolute numbers and baselines improves understanding.
What is a denominator problem in statistics?
When an audience assumes a broader population than the one analyzed. "30% renewed" might mean 30% of completers but only 9% of all registrants.
Do truncated y-axes really mislead people?
Correll, Bertini, and Franconeri (CHI 2020) found truncated axes make differences look larger even when the truncation is signaled. A non-zero baseline isn't always wrong, but it is part of the argument.
What questions should I ask about any statistic?
What is the baseline? What are the raw numbers? What is the denominator and who was excluded? What comparison and time period were chosen? Does the visual scale exaggerate or minimize the difference?
Should I distrust all statistics?
No. Summaries and averages are necessary, and dramatic numbers can reflect dramatic effects. The aim is to find missing context that would materially change what the number means.
Reclaiming Rhetoric is independent and reader-supported.
If this helped, you can help keep it free.
