Beyond Single Ground Truth: Reference Monism as Epistemic Injustice in ASR Evaluation

Anna Seo Gyeong Choi, Maria Teleki, James Caverlee, Miguel del Rio, Corey Miller, Hoon ChoiarXiv 2026 · arXiv

TL;DR

Standard Automatic Speech Recognition (ASR) evaluation relies on a single "ground truth" transcript—a practice the authors term reference monism—which treats human transcription conventions as objective facts rather than interpretive choices. This paper argues that reference monism commits epistemic injustice by systematically penalizing speakers with non-standard speech (such as aphasia) when evaluated against "clean" references that erase meaningful disfluencies. To fix this, the authors introduce a framework to measure this structural bias and propose WER-Range, a reporting practice that evaluates ASR systems across multiple legitimate transcription conventions.

What Problem Does This Paper Address?

When we evaluate ASR systems, we typically compute the Word Error Rate (WER) by comparing the system's hypothesis against a single human-annotated "ground truth" transcript. However, human transcribers don't just record objective facts; they follow specific formatting conventions. A verbatim transcript preserves every "um," "uh," and false start; a non-verbatim transcript cleans them up for readability; a legal transcript preserves hedges but normalizes pronunciation errors.

Enforcing a single convention as the absolute ground truth creates a hermeneutical gap for speakers whose natural speech patterns diverge from that standard. For instance, individuals with aphasia produce clinically meaningful disfluencies. If an ASR system correctly captures these disfluencies, but the evaluation infrastructure uses a "clean" non-verbatim reference, the system is penalized with a higher WER for being accurate. The authors argue that this isn't just a technical glitch; it is a form of institutionalized epistemic injustice that misrepresents how well a system actually understands marginalized speaker groups.

Figure 1: Reference monism versus pluralism, and their metric consequences. Top: under reference monism, a single enforced convention r(p⋆) produces one WER score (9.81%), naturalizing one interpretive choice as ground truth and collapsing legitimate plurality into a single number. Bottom: under reference pluralism, the same utterance yields three transcripts under distinct conventions – non-verbatim r(pN) removes all disfluencies; verbatim r(pV ) preserves fillers, fragments, and repairs exactly as spoken; legal r(pL) retains hesitation markers but normalizes repairs – which together feed into WER-Range, producing an interval [9.81%, 17.38%] that makes the convention-dependence of “accuracy” visible. The same ASR hypothesis h(x) and audio x underlie both computations; only the evaluative infrastructure differs.

Methods

The paper bridges philosophy and engineering by formalizing the cost of reference monism:

  1. Epistemic Injustice Distance (EID): A metric that quantifies the extra penalty a speaker group incurs when evaluated under a single enforced monist policy $p^\star$, compared to the best-case legitimate policy for that system-utterance pair.
  2. Comparative Injustice ($\Delta$EID): Measures the difference in EID between a marginalized group and a control group, isolating the structural evaluation burden imposed purely by convention choice.
  3. WER-Range: Instead of collapsing performance into a single number, this proposed reporting metric presents an interval $[ \min_{p \in P} \text{WER}_p, \max_{p \in P} \text{WER}_p ]$ across a set of legitimate transcription policies $P$.

The authors evaluate seven ASR configurations, including commercial systems with explicit convention controls (Rev AI v2 and v3) and open-source models with implicit biases (Whisper-large-v3 and CrisperWhisper).

Datasets

The authors construct a stratified test set of 8 hours and 58 minutes from the AphasiaBank English Protocol dataset, a corpus of semi-structured interviews. The test set comprises:

  • 29 neurotypical control speakers (3.13 hours)
  • 30 speakers with aphasia (18 fluent, 12 non-fluent; totaling 5.85 hours)

Each audio segment was transcribed by trained human annotators under three distinct professional conventions: Verbatim ($p_V$), Non-Verbatim ($p_N$), and Legal ($p_L$).

Key Insights and Ideas

  • Accuracy is Relational: An ASR system's "accuracy" is not an intrinsic property of the model; it depends entirely on the alignment between the system's output style and the enforced evaluation reference.
  • Error Profiles Shift Systematically: Mismatches between system optimization and reference conventions cause predictable error spikes. Evaluating a verbatim-optimized system against a non-verbatim reference causes insertions to dominate the error profile, as the system is penalized for transcribing actual spoken words that the clean reference chose to omit.
  • Fairness Metrics are Distorted: Standard group fairness audits take the ground truth as a given. This paper reveals that the measured "fairness gap" between speaker groups is highly volatile and dictated by the choice of transcription standard.

Key Results

  • Substantial Convention-Dependence: For the Rev AI v2 (verbatim) system, the overall WER swung from 9.81% (under a verbatim reference) to 17.38% (under a non-verbatim reference)—a 1.8× difference based purely on evaluation infrastructure.
  • Quantifying the Injustice ($\Delta$EID): Under the commonly enforced non-verbatim standard, control speakers had an EID of 4.73 percentage points (pp), while non-fluent aphasic speakers suffered an EID of 12.91 pp. This resulted in a $\Delta$EID of 8.18 pp, proving that the evaluation standard itself inflicts a disproportionate penalty on disfluent speakers.
  • Volatile Fairness Gaps: The performance gap between control and non-fluent aphasia groups varied by over 60% depending on the reference used, ranging from 13.50 pp (verbatim reference) to 21.68 pp (non-verbatim reference).

Limitations and Future Work

While the theoretical framework is broadly applicable, the empirical validation in this paper is limited to a single clinical condition (aphasia), one corpus (AphasiaBank), and three transcription conventions. The authors note that future work must validate EID and $\Delta$EID across other populations vulnerable to transcription bias, including child language, non-native speech, and dialectal variations like African American English (AAE). Additionally, larger speaker sample sizes are needed to apply mixed-effects modeling to ensure statistical robustness.

Summary updated

Abstract

Automatic speech recognition (ASR) evaluation compares system output to ground truth transcripts, with Word Error Rate (WER) quantifying the distance between them. But ground truth transcripts are not discovered - they are produced by human annotators following conventions that encode normative assumptions about which speech features matter. Different conventions (verbatim, non-verbatim, legal) produce different transcripts of identical speech and judge the same ASR output differently. This paper argues that reference monism - enforcing a single transcription convention as ground truth - commits epistemic injustice. Speakers with aphasia, whose speech includes clinically meaningful disfluencies, are systematically disadvantaged when evaluated against "clean" references that treat those disfluencies as errors. The harm is not merely differential performance, but that evaluative infrastructure lacks interpretive resources to recognize their contributions as legitimate. We develop a philosophical framework introducing the hermeneutical gap, formalize Epistemic Injustice Distance (EID) to measure reference monism's cost, and demonstrate empirically using AphasiaBank that WER varies depending on which convention defines ground truth. We propose WER-Range: reporting performance across legitimate conventions rather than assuming a single correct answer.
BibTeX
@misc{choi2026beyond,
  author = {Anna Seo Gyeong Choi and Maria Teleki and James Caverlee and Miguel del Rio and Corey Miller and Hoon Choi},
  title = {{Beyond Single Ground Truth: Reference Monism as Epistemic Injustice in ASR Evaluation}},
  year = {2026},
  eprint = {2605.07084},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2605.07084},
}
Figure