Severity-Weighted Word Error Rate: Letting an LLM Judge Price Each ASR Error

Word error rate (WER) is the standard score for speech recognition (ASR) systems. It aligns the system's transcript against a human reference and counts substituted, deleted and inserted words. Every error costs the same. "A" for "an" weighs as much as "cancel" for "set", or a dropped "not". LLM-as-a-judge, where a large language model reads the reference and the hypothesis and decides which is better, already tracks human preference far more closely. On the French HATS benchmark, GPT-4.1 agrees with human consensus in 94% of pairwise choices, against 63% for WER (Generative vs. Encoder Large Language Models for ASR Evaluation). Whole-sentence judging has a cost, though. It throws away what made WER useful: a corpus-level number that adds up error by error, says where the errors are, and supports comparing two systems.

We propose keeping WER's alignment and asking the LLM judge only one narrow question per aligned error: how badly does this edit, in this context, damage the message? The judge's answer is a severity class from a published linguistic taxonomy. Each class has a cost, and the costs sum into a severity-weighted error rate. A small human-labelled calibration set then corrects for the judge's per-class mistakes, so that claims like "system B beats system A" come with valid error bars. The bet is that judging one error at a time is easier, and less biased, than judging a whole sentence. If it is, the result keeps the human alignment of LLM judges and the transparency of WER.

Core Claim

Define the severity-weighted error rate over a test set as

$$\text{SWER} = \frac{1}{N}\sum_{e \in \mathcal{E}} w_{c(e)},$$

where:

  • $\mathcal{E}$ is the set of edits found by the standard WER alignment;
  • $c(e)$ is the severity class an LLM judge assigns to edit $e$, given the reference sentence, the hypothesis sentence and the edit position;
  • $w_c$ is a non-negative cost per class;
  • $N$ is the number of reference words.

With all $w_c = 1$ this is exactly WER, so the ablation that isolates the judge is built in.

The classes come from the four-tier taxonomy of A Benchmark of French ASR Systems Based on Error Severity. That paper defines the tiers as follows:

  • Lex: recognizable distortion of a word.
  • Gram: grammatical error that does not block understanding.
  • Cotx: error that context is needed to resolve.
  • Fail: miscommunication.

It applied them by hand, with one expert, to about 120 errors per system.

The claim has three testable parts:

  1. Per-edit judging matches or beats sentence-level LLM judging in agreement with human preferences, especially on the low-consensus items where the sentence judge is weakest.
  2. Per-edit judging reduces the documented blurring between "Useful" and "Bad" categories.
  3. A misclassification correction fitted on a few hundred human-labelled edits makes system-level comparisons statistically valid, which no current LLM-based ASR metric offers.

How this relates to prior work

Motivating prior work: the HATS study and two semantic-metric lines. The closest work is the HATS study (Generative vs. Encoder LLMs for ASR Evaluation). It judges whole hypotheses, either pairwise or by direct four-way classification. It reports that direct classification blurs "Useful" and "Bad", and it leaves cost and distillation open. We differ in two ways: the unit of judgment is the aligned edit rather than the sentence, and the output is an additive, decomposable corpus score rather than a preference.

Two further lines motivate the move away from flat counting. SemDist (Semantic Distance; Evaluating User Perception with Semantic Distance) replaced WER with embedding cosine distance. It raised correlation with user ratings from 0.36 to 0.59. However, it is symmetric and cannot say which error caused the damage. The Integrated Score on dysarthric speech (Aligning ASR Evaluation with Human and LLM Judgments) combines three separate models by linear regression: Soundex phonetic similarity, BERTScore, and a RoBERTa natural language inference model. It reaches Pearson 0.890 against human intelligibility on 100 pairs. That paper also shows that repetition insertions can push WER above 200% for fully intelligible speech, a case per-edit pricing handles directly: a repeated word is Lex or zero cost.

Supplying the method: the severity taxonomy and Noisy HT. The taxonomy comes from Wottawa et al., cited above. The statistical machinery comes from Noisy but Valid. That paper calibrates an imperfect LLM judge's true- and false-positive rates on a small human set and keeps Type-I error at 5% even for biased judges. It handles only binary labels. We extend it to a four-class confusion matrix applied to error counts.

Constraining the design: downstream utility and judge biases. An approach to measuring ASR performance in LLM-powered applications shows that WER rankings and downstream answer error rate (AER) diverge on VoxPopuli. So agreement with downstream use is a target, not an assumption. Known judge failures shape the design:

Scoring each hypothesis independently against the reference removes the pairwise setup that position bias and intransitivity need.

Research Program

Building the per-edit judge. We take a standard WER alignment and group adjacent edits into error spans, so that a split compound counts as one span. For each span, the judge receives:

  • the full reference;
  • the full hypothesis;
  • the span marked in both;
  • the four-tier rubric with examples.

It returns one class and a one-sentence rationale. We run an open-weight judge (Qwen3.5-35B, the strongest open model on HATS) and GPT-4.1, with and without a short chain-of-thought.

Setting the class costs. The costs $w_c$ are set two ways. The first is fixed and interpretable: Lex 0, Gram 0.25, Cotx 0.5, Fail 1. The second fits the costs on a human-preference training split by maximizing pairwise agreement. Only that split is used, and it is disclosed.

Statistical correction. An estimate $\hat{M}$ of the judge's confusion matrix over the four classes comes from a few hundred expert-labelled edits. We invert $\hat{M}$ on the observed class counts of each system. Bootstrap intervals then propagate both calibration and test-set uncertainty into the difference in SWER between two systems.

Cost reduction and transfer.

  • Distillation. The judge's per-edit labels on a large unlabelled pool train a small encoder classifier, a CamemBERT- or RoBERTa-sized model, to show that the scheme survives without a large model at test time.
  • Caching. Many systems share identical edits in identical contexts, so judge calls can be cached across systems.
  • Other languages and impaired speech. SAP tests whether the same rubric transfers beyond French broadcast speech.

Baselines

Naive reference. The do-nothing reference is plain WER, with CER alongside. On the HATS 100%-consensus subset they agree with humans 63% and 77% of the time (Generative vs. Encoder LLMs). They are computed on identical alignments, and SWER with uniform weights must reproduce WER exactly as a sanity check.

Strongest existing method. The main competitor is sentence-level LLM judging as in the HATS study, both pairwise and direct four-class. The reported strongest numbers are 94% pairwise agreement for GPT-4.1 and 92% for Qwen3.5-35B on the 100%-consensus subset. We rerun it ourselves on the same subsets, with the same judge models and comparable prompt-engineering budget (the same number of prompt iterations on the same development split). We also randomize candidate order, so it is not handicapped by position bias we would otherwise avoid. Because 94% leaves little headroom, beating it means one of two things:

  • matching it within one point on the 100% subset while exceeding it on the ≥70% and full subsets, where no numbers are reported in the file;
  • or matching agreement while showing clearly less Useful/Bad confusion in direct classification.

Encoder-based metrics. These are SemDist with Sentence-CamemBERT-large, the best embedding model on HATS, though no numeric value is reported in the file. On SAP the comparison is the Integrated Score at Pearson 0.890. That score is refit on our splits with the same training data our learned costs see. Beating it on SAP means a higher correlation on a held-out set larger than the original 100 pairs.

Ablation. The key ingredient is per-edit decomposition. The decisive ablation therefore gives the same judge, the same taxonomy text and the same chain-of-thought setting the whole sentence, asking for an overall class, versus the per-edit procedure. A second ablation removes sentence context from the per-edit prompt and shows only the span. It tests whether context, rather than decomposition, drives any gain.

Why not just…?

Why not just use the sentence-level LLM judge that already gets 94%? This is the real threat. Our evidence for an advantage is indirect: the HATS paper's own report of Useful/Bad boundary overlap, and the fact that per-edit scoring avoids pairwise biases. The whole-sentence-versus-per-edit ablation settles it on the same model and prompts. If per-edit judging shows no gain in agreement or in boundary separation, the remaining contributions narrow to three things:

  • decomposability, meaning which error types a system makes;
  • the calibrated system comparison;
  • caching across systems.

We would report that narrowed claim plainly.

Why not just use SemDist or the Integrated Score, which are cheap and deterministic? On HATS, embedding quality did not predict judging quality. The phonetic-semantic-NLI score was fit on only 100 pairs from one population. Neither metric points to the offending edit. Still, a refit Integrated Score could plausibly match us on SAP. The head-to-head comparison on held-out SAP data and on HATS decides whether the LLM's per-edit reasoning adds anything over three encoders.

Why not just distill the sentence-level judge into a small model, as the HATS authors suggest? We test this directly. We distill both the sentence-level and the per-edit judge into the same encoder, using the same unlabelled pool and the same compute. Per-edit labels are short, local and more numerous, so we expect them to be easier to learn. If the distilled sentence judge does as well, the per-edit route offers only interpretability and the statistical correction, not a cost advantage.

Proposed Experiments

Human-preference agreement on HATS. We report pairwise agreement on all three consensus subsets, with confusion matrices for the Useful/Bad boundary in direct classification. Both are compared against WER, CER, SemDist and sentence-level judging.

Per-edit validity on REPERE. We report Cohen's kappa between judge labels and the expert's four-tier labels on REPERE. No reported inter-annotator figure was found in the file, so a second human annotator on a subset gives the ceiling. We also check whether SWER reproduces the paper's qualitative system ordering: Kaldi hybrids fewest Lex errors, LeBenchmark models fewest Fail errors.

Statistical validity of system comparisons. We simulate system comparisons by resampling. The test is whether the corrected SWER difference keeps a nominal 5% false "B is better" rate when the systems are in truth equal under human labels, and how much power it keeps compared with the uncorrected judge.

Downstream alignment. Using Whisper Tiny through Large and wav2vec 2.0 on LibriSpeech, Fleurs and VoxPopuli, we compute Spearman rank correlation between each metric's system ranking and the downstream AER ranking. No per-model AER numbers are reported on Lacuna, so we recompute AER ourselves.

Transfer to impaired speech. On SAP dysarthric speech we test whether stutter-type insertions are priced near zero while dropped negations are priced as Fail.

Decisive check. The decisive check is the sentence-versus-edit ablation. If it shows a gain on the low-consensus subsets and on the Useful/Bad boundary, per-edit pricing earns its place as a WER replacement.