Training on Noisy Transcripts for Robust Spoken Understanding

This direction is merged into another direction: synthetic error injection (ASR error simulation). Its last version stays readable and citable here.

Version 1 of 129 papers38 conceptsfrozen 2026-10-04

Most natural language processing models are trained on "gold-standard" text—clean, well-formatted prose from books or Wikipedia. However, in real-world applications like voice assistants or digital medical scribes, the input is rarely clean. It is usually the output of an Automatic Speech Recognition (ASR) or Optical Character Recognition (OCR) system, which introduces systematic errors: word substitutions, deletions, and phonetic misinterpretations.

A growing body of research suggests that the best way to handle this "noise" is not to fix it with a separate denoising step, but to embrace it during training. By exposing models to the specific failure modes of ASR and OCR systems, researchers are building "inherently robust" models that outperform those trained on perfect text.

The Signal in the Noise

The most intuitive approach to noisy data is to clean it. However, Kuluhan Binici and colleagues argue that intermediate correction layers often introduce their own "hallucinations," especially in low-resource domains like medicine. Instead, training a model directly on noisy inputs allows it to develop a native resilience to transcription errors.

Surprisingly, this noise can sometimes be a feature rather than a bug. In the context of medical diagnostics, Frank Rudzicz found that machine learning models trained on error-prone automated transcripts achieved higher accuracy in diagnosing primary progressive aphasia than those trained on perfect human transcripts. The reasoning is that the specific ways an ASR system fails when listening to a patient—struggling with certain phonemes or rhythms—capture unique pathological characteristics of the speaker's voice that a clean transcript would erase.

Learning the Error Distribution

To bridge the gap between clean benchmarks and messy reality, researchers are moving away from training on ground-truth text. Anthony Kum Hoe Tung demonstrated that training machine reading models on imperfect ASR transcripts instead of clean text improved $F_1$-scores by over 10 points. This works because the model learns the "error distribution" of the speech recognizer, becoming familiar with common phonetic substitutions (e.g., "recognize speech" vs. "wreck a nice beach").

This strategy is being applied across various domains:

  • Information Extraction: Animesh Prasad showed that training sequence labeling models on the imperfect outputs of OCR systems improves robustness for historical handwritten records.
  • Spoken Language Understanding: Jin Cao and researchers at Amazon AI utilized "style-attuned" pre-training on actual conversational data, including ASR errors, to make models more effective in real-world deployments.
  • Slot Filling: Guangzhi Sun fine-tuned Large Language Models (LLMs) on alternative hypotheses generated by ASR systems, teaching the model to anticipate and compensate for common acoustic corruptions.

Simulating Corruption at Scale

One challenge with this approach is the need for large amounts of paired audio and "noisy" transcripts. To solve this, researchers use data augmentation to simulate ASR errors on clean text. Ido Cohn and others have shown that intentionally corrupting text with simulated word substitutions, deletions, and casing inconsistencies significantly enhances the robustness of downstream entity recognition.

In the medical field, the MEDSAGE framework developed by Kuluhan Binici uses LLMs to generate synthetic "noisy" dialogues. These dialogues realistically mimic ASR errors, allowing summarization models to practice on "dirty" data before they ever encounter a real patient recording. Similarly, Merve Ünlü Menevse developed a pipeline to automatically generate spoken question-answering data, ensuring that models are trained on the same types of recognition failures they will face during inference.

Robustness through Loss Functions

Beyond data augmentation, the mathematical objectives used to train these models are also evolving. Standard loss functions often treat all errors equally, but in noisy environments, some examples are much harder than others. Jiangyan Yi applied Focal Loss to punctuation prediction, which prioritizes "hard" examples during training. This encourages the model to learn resilient features that persist even when the input text is corrupted.

For the ASR systems themselves, Vladimir Bataev introduced novel loss functions like Target-Robust-Transducer (TRT). These modify the underlying graph representation of the loss to account for missing or extra words in the training targets. This allows models to be trained on weakly supervised, noisy data without losing performance, making them more compatible with the "wild" datasets found in real-world production pipelines.

Papers in this version

History & citationVersion 1 · 2026-10-04

Merged into another direction · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_training_on_noisy_transcripts_for_robust_fab77ce5,
  title = {Training on Noisy Transcripts for Robust Spoken Understanding},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/training-on-noisy-transcripts-for-robust-spoken-understanding/rdr_fab77ce570c8454f86ac4569966faa52},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_training_on_noisy_transcripts_for_robust_fab77ce5_v1,
  title = {Training on Noisy Transcripts for Robust Spoken Understanding},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/training-on-noisy-transcripts-for-robust-spoken-understanding/rdr_fab77ce570c8454f86ac4569966faa52/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}