Most natural language processing models are trained on "gold-standard" text—clean, well-formatted prose from books or Wikipedia. However, in real-world applications like voice assistants or digital medical scribes, the input is rarely clean. It is usually the output of an Automatic Speech Recognition (ASR) or Optical Character Recognition (OCR) system, which introduces systematic errors: word substitutions, deletions, and phonetic misinterpretations.
A growing body of research suggests that the best way to handle this "noise" is not to fix it with a separate denoising step, but to embrace it during training. By exposing models to the specific failure modes of ASR and OCR systems, researchers are building "inherently robust" models that outperform those trained on perfect text.
The Signal in the Noise
The most intuitive approach to noisy data is to clean it. However, Kuluhan Binici and colleagues argue that intermediate correction layers often introduce their own "hallucinations," especially in low-resource domains like medicine. Instead, training a model directly on noisy inputs allows it to develop a native resilience to transcription errors.
Surprisingly, this noise can sometimes be a feature rather than a bug. In the context of medical diagnostics, Frank Rudzicz found that machine learning models trained on error-prone automated transcripts achieved higher accuracy in diagnosing primary progressive aphasia than those trained on perfect human transcripts. The reasoning is that the specific ways an ASR system fails when listening to a patient—struggling with certain phonemes or rhythms—capture unique pathological characteristics of the speaker's voice that a clean transcript would erase.
Learning the Error Distribution
To bridge the gap between clean benchmarks and messy reality, researchers are moving away from training on ground-truth text. Anthony Kum Hoe Tung demonstrated that training machine reading models on imperfect ASR transcripts instead of clean text improved $F_1$-scores by over 10 points. This works because the model learns the "error distribution" of the speech recognizer, becoming familiar with common phonetic substitutions (e.g., "recognize speech" vs. "wreck a nice beach").
This strategy is being applied across various domains:
- Information Extraction: Animesh Prasad showed that training sequence labeling models on the imperfect outputs of OCR systems improves robustness for historical handwritten records.
- Spoken Language Understanding: Jin Cao and researchers at Amazon AI utilized "style-attuned" pre-training on actual conversational data, including ASR errors, to make models more effective in real-world deployments.
- Slot Filling: Guangzhi Sun fine-tuned Large Language Models (LLMs) on alternative hypotheses generated by ASR systems, teaching the model to anticipate and compensate for common acoustic corruptions.
Simulating Corruption at Scale
One challenge with this approach is the need for large amounts of paired audio and "noisy" transcripts. To solve this, researchers use data augmentation to simulate ASR errors on clean text. Ido Cohn and others have shown that intentionally corrupting text with simulated word substitutions, deletions, and casing inconsistencies significantly enhances the robustness of downstream entity recognition.
In the medical field, the MEDSAGE framework developed by Kuluhan Binici uses LLMs to generate synthetic "noisy" dialogues. These dialogues realistically mimic ASR errors, allowing summarization models to practice on "dirty" data before they ever encounter a real patient recording. Similarly, Merve Ünlü Menevse developed a pipeline to automatically generate spoken question-answering data, ensuring that models are trained on the same types of recognition failures they will face during inference.
Robustness through Loss Functions
Beyond data augmentation, the mathematical objectives used to train these models are also evolving. Standard loss functions often treat all errors equally, but in noisy environments, some examples are much harder than others. Jiangyan Yi applied Focal Loss to punctuation prediction, which prioritizes "hard" examples during training. This encourages the model to learn resilient features that persist even when the input text is corrupted.
For the ASR systems themselves, Vladimir Bataev introduced novel loss functions like Target-Robust-Transducer (TRT). These modify the underlying graph representation of the loss to account for missing or extra words in the training targets. This allows models to be trained on weakly supervised, noisy data without losing performance, making them more compatible with the "wild" datasets found in real-world production pipelines.