What the expert-labeled data revealed
When our expert reviewers labeled a dataset of 1,598 statements for cognitive bias, we expected disagreement at the margins. What we found instead reshaped how we think detection should work: 93.4 percent of statements supported more than one plausible bias label. The average statement carried 2.93 candidate biases. Across 82 distinct bias labels in 12 families, the single-label statement, the clean case where one bias explains the wording and nothing else fits, was the rare exception. Ambiguity was not a defect in the labeling process. It was the most consistent empirical fact about the material.
A concrete example shows why. Take the phrase "many financial advisors recommend this allocation." An expert reading that sentence cannot honestly assign it a single label. The appeal to advisors is an authority cue. The word "many" is a bandwagon cue. If the allocation carries a specific number, that number can anchor every estimate the reader makes afterward. All of these readings are legitimate because they describe the same words operating through different psychological channels, and which channel dominates depends on the reader, the context, and what the surrounding questions have already primed. Language underdetermines mechanism. A sentence is evidence for a bias in the way a symptom is evidence for a disease: real evidence, and rarely conclusive on its own.
Medicine is the right analogy, and its practitioners settled this methodological question long ago. A clinician presented with chest pain does not announce a single diagnosis and stop. She writes a differential, ranks it by likelihood, and orders the test that discriminates between the leading candidates. The differential is what makes the reasoning auditable; a colleague can look at the list and argue with the ranking. A diagnosis delivered without one is an assertion, whatever its confidence.
Why disagreement is meaningful
The machine-learning field has been arriving at the same conclusion from another direction. A growing body of NLP research on subjective annotation tasks argues that annotator disagreement is signal rather than noise: when trained labelers diverge on a hate-speech judgment or a sentiment rating, the divergence often encodes real properties of the text and real variation in how populations read it and collapsing it into a majority vote destroys information. Some of that work goes further, showing cases where the minority reading is the correct one and majority-vote aggregation systematically erases it. Our expert data says the same thing about bias. When two reviewers assign different labels to a statement, the productive question is what each of them saw, and the answer is usually that both cues are present.
This finding has sharp consequences for how detection tools should behave, including ours. A system that returns one bias label with one confidence score is presenting a differential-shaped problem in a verdict-shaped box, and the mismatch misrepresents how the experts who trained it actually reason. So, we hold our own output to requirements derived from the labeled data. A detection must cite the exact span of wording that supports it, because evidence that cannot point at anything is opinion. It must surface the competing labels consistent with the same cue, because 93.4 percent of the time those competitors exist. It must actively search for counter-evidence, including the plausible benign reading of the same sentence, because a tool built to detect confirmation bias should not itself exhibit confirmation bias. And its confidence language must stay calibrated: may, can, tends to. A wording choice can make a bias more likely to fire in a population of readers. Certainty about any single reader is not on offer, and a tool that claims it is overclaiming.
Analyzing effects instead of intentions
One more regularity in the expert data is worth reporting. Across all 1,598 statements, every expert rationale, without exception, described the effect of the wording on the person receiving it: the reader may feel implicit pressure to agree, the phrasing prompts the respondent to weigh prior investments. None of them made claims about what the author intended. This was not an instruction we gave the reviewers; it is how careful analysts naturally talk about bias when they are being precise, because intent is unobservable from text and effect is what matters for measurement. The same sentence can arrive in a survey through carelessness, convention, or design, and its effect on respondents is identical in each case.
For teams that rely on survey and research data, the practical takeaway is a change in what to expect from analysis, human or automated. Distrust the single confident verdict, whoever produces it, and ask for the differential instead: which cues are present in this wording, which biases could each cue activate, what would distinguish between them, and what the case against the flag looks like. An analysis that can answer those questions is one you can argue with, and an analysis you can argue with is the only kind worth paying for. The statements in our dataset did not become ambiguous when the experts labeled them. They were ambiguous all along. The labeling just made it visible.
Better questions start here
Make bias work for you.
CogBias helps teams identify and improve biased survey questions before they affect the results.
Request a Demo!