Abstract
This paper introduces a "back transcription" framework designed to evaluate the robustness of Natural Language Understanding (NLU) models to Automatic Speech Recognition (ASR) errors without requiring expensive human-annotated audio. By utilizing Text-to-Speech (TTS) to generate synthetic audio from reference text and then transcribing it back via ASR, the authors create a high-fidelity proxy for real-world spoken language data. Evaluation on the MASSIVE and SLURP datasets demonstrates that synthetic speech errors effectively simulate human speech patterns, revealing significant performance drops in NLU tasks, particularly in character-sensitive tasks like slot filling.
Introduction
In voice-controlled systems such as virtual assistants, a "cascading error" problem often degrades user experience. First, an ASR engine converts voice to text, and subsequently, an NLU model interprets that text to identify user intent and extract specific entities (slots). Even high-accuracy ASR systems introduce errors that can be catastrophic for downstream NLU performance. Traditionally, assessing this robustness requires expensive, human-recorded spoken corpora. This research addresses the need for a low-cost, text-only evaluation method by asking: Can synthetic speech simulation effectively evaluate NLU models in place of real human recordings?
Methodology: The Back Transcription Pipeline
The proposed evaluation framework consists of three distinct stages designed to simulate and analyze the impact of speech recognition errors on NLU models.
- Back Transcription: A reference text is passed through a TTS engine (e.g., FastSpeech 2 or Tacotron 2) to generate audio, which is then processed by an ASR engine (Whisper) to produce a "hypothesis" text. This hypothesis contains phonetic and structural errors typical of real-world voice interfaces.
- Robustness Assessment: The framework categorizes NLU output changes between the reference and hypothesis texts into three criteria: Correct to Incorrect ($C \to I$), where ASR errors break the NLU; Incorrect to Incorrect ($I \to I$), where the NLU remains wrong but the error type changes; and Incorrect to Correct ($I \to C$), where ASR errors inadvertently "fix" a sample.
- Error Analysis via Edit Operations: Using the Ratcliff-Obershelp algorithm, the system aligns original and hypothesis texts to identify specific edit operations (e.g., deletions, substitutions, or splits). A logistic regression model then identifies which error types most significantly impact NLU performance.
Experimental Results
The researchers validated their method using the MASSIVE NLU dataset and the SLURP spoken language resource. A key finding was that robustness scores derived from synthetic TTS-generated audio were nearly identical to those from real human recordings, with differences ranging only between 0.03 and 0.08. This confirms that synthetic speech is a valid and highly effective proxy for testing voice-robustness using only text datasets.
As shown in the results, NLU models exhibit a significant performance decline when moving from gold-standard text to ASR-processed text. Notably, Slot Filling F1 scores dropped from 0.80 to 0.61 when using FastSpeech-generated errors, while Intent classification proved more robust. This suggests that tasks requiring high character-level precision are far more vulnerable to the phonetic substitutions common in ASR.
Discussion and Insights
Beyond simple accuracy, the authors introduced consistency metrics like $R_{123}$, which penalize any change in NLU output caused by ASR noise. This is vital for industrial applications where predictable behavior is a priority. The error analysis further revealed that not all errors are equal; homophone substitutions (e.g., "macs" vs "max") and missing terminal punctuation were found to deteriorate NLU performance more severely than simple typographical errors.
Conclusion
Back transcription provides a scalable, model-agnostic tool for developers to stress-test NLU systems. While the current method relies on the specific error profiles of the chosen TTS and ASR models—potentially missing nuances like heavy accents or background noise—it offers a powerful baseline for robustness evaluation. Future work will focus on expanding the error models to include more diverse acoustic conditions and developing more granular metrics for slot-specific robustness.
Summary updated