Beyond Word Error Rate: A Failure Taxonomy of Streaming Dictation ASR

Harras Mansoor, Muhammad Wisal, Hassan Bhatti, Masood ShakirarXiv 2026 · arXiv

TL;DR

Standard Word Error Rate (WER) is a blunt instrument that hides critical failures in real-world streaming dictation. This paper introduces a multi-level failure taxonomy and demonstrates that ASR systems with nearly identical baseline performance can diverge wildly—sometimes catastrophically—when faced with whispers, background noise, or mixed-language speech.

What Problem Does This Paper Address?

Most Automatic Speech Recognition (ASR) benchmarks evaluate systems using pre-recorded files and "normalized" text (stripping punctuation and casing). However, in streaming dictation—like sending a text or taking a note—the raw output is the product.

A simple WER score doesn't tell you if a system missed a single word or dropped an entire sentence. It also ignores "app-level" failures, such as a Voice Activity Detector (VAD) accidentally silencing a user's whisper before the ASR model even hears it. The authors aim to move beyond a single number to a detailed understanding of where and how these systems break.

Methods

The researchers evaluated 11 streaming ASR systems (including commercial APIs like Gemini, Deepgram, and Zoom, alongside open models like Whisper and Qwen) using a novel failure taxonomy. Errors were categorized into four levels:

  1. Recognition: Standard word-level errors (substitutions, deletions, insertions).
  2. Output Formatting: Issues with punctuation, capitalization, and Inverse Text Normalization (e.g., writing "7" vs "seven").
  3. Language Handling: How systems manage code-switching or recordings where the user speaks a language other than the one selected.
  4. Before the Model: Failures in the audio pipeline, specifically "VAD gating" where audio is dropped before reaching the model.

The evaluation used a streaming protocol, sending audio in small chunks (typically 32ms) to mimic real-world microphone input.

The proposed failure taxonomy categorizes errors by where they occur in the pipeline.

Datasets

The study utilized a mix of in-house and public datasets:

  • In-house English: 106 scripted dictations from a single speaker recorded in three conditions: quiet, with a fan on, and whispered.
  • Multilingual/Dialect Sets: 20 clips each for 11 language/dialect sets (e.g., Arabic dialects, Hindi, Urdu, Mandarin) sourced from FLEURS, SADA, and others.
  • Two-language Recordings: 400 recordings created by joining clips of different languages to test code-switching.
  • External Corpora: The EARS dataset (107 speakers) was used to validate whispered speech findings across many voices, and VOiCES was used for far-field and noisy environments.

Key Insights and Ideas

  • The "Whisper Gap": Systems that look identical on quiet speech can be miles apart on whispers. For example, Zoom and OpenAI’s GPT-Live-Transcribe both have ~7.4% WER on quiet English. On whispered English, GPT-Live stays at 8.8%, while Zoom’s error rate explodes to 34.7%.
  • VADs are Silent Killers: A Voice Activity Detector tuned for normal speech often misidentifies whispers as silence. Running Whisper behind a standard Silero VAD caused the whispered WER to jump from 10.1% to a staggering 51.4%.
  • Language "Filters" vs. "Preferences": When a user selects "English" but speaks another language, some systems (like Gemini and Deepgram) act as a filter, deleting the non-English speech entirely (86–99% loss). Others (like Meta and GPT-Live) act as a preference, still attempting to transcribe the "wrong" language.
  • Names are Harder: Across all systems, names were written correctly less often than ordinary words, with some systems showing a nearly 10-point gap in accuracy.

Key Results

The overall results (Table 3) highlight that Gemini Smart achieved the lowest quiet-English WER at 3.9%. However, the breakdown of error types (Table 4) shows that for many systems, "extra words" (insertions) are a major source of error, often caused by the system transcribing disfluencies or self-corrections that a "smart" system should ideally filter out.

Overall WER and Formatted error rates across different English conditions.
Breakdown of WER into wrong, missing, and extra words.

Limitations and Future Work

  • Speaker Diversity: The primary English results rely on a single speaker. While the EARS dataset helped validate whispered speech findings, more diverse spontaneous dictation data is needed.
  • Spontaneous vs. Read Speech: The test sets were largely scripted. Real-world dictation involves more "ums," "ahs," and mid-sentence changes of heart, which significantly impact "clean-up" features.
  • Dataset Size: Some language sets were small (20 clips), making it difficult to draw definitive conclusions for specific low-resource dialects.

The authors have released their code, recordings, and every transcript to allow the community to replicate and extend this more nuanced approach to ASR evaluation.

Summary updated

Abstract

Speech recognition (ASR) systems are usually compared by word error rate (WER) on recorded files, after punctuation, capital letters and number format are removed. In dictation, the audio is streamed live and the text goes into the user's message as written, so what matters is which mistakes the user has to fix. The test set, streamed to eleven ASR systems, comprises 106 scripted English dictations from one speaker (read in a quiet room, with a fan on, and whispered), 20 clips in each of eleven other language and dialect sets, and 400 recordings that join clips in two languages. Every error was sorted by a fixed rule into four levels: recognition, output formatting, language handling and, for the self-hosted models, audio dropped before the model. Systems with similar WER failed in different ways. Zoom Scribe and OpenAI's GPT-Live-Transcribe have almost the same WER on quiet English (7.4% and 7.3%), but 34.7% and 8.8% on whispered English. Zoom and both AssemblyAI models left out 26-28% of whispered words, and still 21-26% with their end-of-turn detection relaxed or off; seven other systems left out at most 3%. Qwen3-ASR's 32% came from the speech detection in our own deployment (6% when relaxed). Across 107 EARS speakers, the same four systems lost 7-31% of whispered words, while six others lost under 2%. With only English selected, Gemini, Deepgram and our self-hosted Whisper lost the non-English part of a two-language recording in 86-99% of recordings, against 33-46% with both languages selected, while GPT-Live-Transcribe, OpenAI's realtime Whisper and Meta lost it in at most 20%. Running Whisper behind a Silero voice activity detector that drops audio judged to be silence raised whispered WER from 10.1% to 51.4%. Most English results come from one read speaker. The code, the recordings, both references and every transcript are released.
BibTeX
@misc{mansoor2026beyond,
  author = {Harras Mansoor and Muhammad Wisal and Hassan Bhatti and Masood Shakir},
  title = {{Beyond Word Error Rate: A Failure Taxonomy of Streaming Dictation ASR}},
  year = {2026},
  eprint = {2611.16919},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.16919},
}
Figure