Improving RNN-Transducers with Acoustic LookAhead

Vinit S. Unni, Ashish R. Mittal, Preethi Jyothi, Sunita Sarawagi

Abstract

Streaming Automatic Speech Recognition (ASR) often relies on the Recurrent Neural Network-Transducer (RNN-T) architecture. While effective, the RNN-T's internal language model can frequently override acoustic evidence, leading to "hallucinations"—grammatically plausible but phonetically incorrect transcriptions. This paper discusses LOOKAHEAD, a novel architectural modification that grounds the text encoder in the acoustic signal by allowing it to "peek" at future acoustic tokens. Our results demonstrate a 5% to 20% relative reduction in Word Error Rate (WER) across various benchmarks, including out-of-domain accented speech, while preserving real-time streaming capabilities.

1. Introduction

The Recurrent Neural Network-Transducer (RNN-T) has become the industry standard for low-latency, streaming automatic speech recognition. Its modular design consists of an Acoustic Encoder (AE) and a Language Encoder (LE), which are integrated via a joint network. However, this separation often leads to a phenomenon where the LE dominates the prediction process. When the acoustic signal is ambiguous—due to noise, accents, or rare vocabulary—the LE may predict words based solely on linguistic probability, ignoring the actual audio. This research introduces LOOKAHEAD, a mechanism designed to mitigate these hallucinations by providing the text encoder with immediate acoustic context.

2. The Hallucination Problem

The core issue in standard RNN-T models is the "blindness" of the Language Encoder to the acoustic input. Because the LE is trained on vast amounts of text, it becomes a powerful language model. In scenarios where the audio is unclear, the model defaults to the most likely linguistic sequence. For instance, a speaker saying "villeroy took" might be transcribed as "villar I took" because the latter is more frequent in common English datasets. This bias is particularly problematic for rare words and out-of-domain (OOD) speech, where the acoustic signal is the only reliable source of truth.

3. The LOOKAHEAD Architecture

The proposed LOOKAHEAD mechanism "grounds" text representations in the acoustic signal before they reach the joint network. This is achieved through two primary components:

3.1 Implicit Acoustic Model (IAM)

The IAM is a simplified version of the RNN-T that predicts tokens based exclusively on the current acoustic frame, without any preceding text context. For every frame of audio, the IAM identifies the most likely next few non-blank tokens. This provides a "raw" acoustic preview of what is likely to be spoken next.

3.2 Acoustic-Text Fusion

Instead of passing the standard text representation ($g_u$) directly to the joint network, LOOKAHEAD utilizes a small Feed-Forward Network (FFN) to fuse $g_u$ with the lookahead tokens extracted by the IAM. This results in a "time-aware, acoustically informed" text representation. By limiting the lookahead window (typically to $w=3$ tokens), the model maintains its streaming nature without introducing the high latency associated with non-causal models that require the entire audio sequence.

4. Experimental Setup

The effectiveness of LOOKAHEAD was evaluated using two major datasets:

  • LibriSpeech: A standard ASR benchmark using 100, 360, and 960-hour subsets.
  • Mozilla Common Voice (MCV): A dataset featuring diverse English accents (e.g., Indian, Scottish, Filipino). This served as a critical test for OOD robustness and the model's ability to handle speech that deviates from the training distribution.

Beyond Word Error Rate (WER), the researchers employed phonetic metrics such as Phone Error Rate (PER) and Dolgopolsky Error Rate (DER) to measure how closely the model's predictions matched the actual sounds of the speech, regardless of whether the final word was correct.

5. Results and Analysis

The LOOKAHEAD model demonstrated significant improvements across all tested dimensions:

  • WER Reduction: The model achieved a 5% to 20% relative reduction in WER compared to baseline RNN-T models. These gains were consistent across both in-domain (LibriSpeech) and OOD (MCV) datasets.
  • Rare Word Recognition: On the LibriSpeech 100-hour set, the model showed marked improvement in identifying words that appeared fewer than 20 times in the training data—words that are typically misrecognized due to LM bias.
  • Phonetic Faithfulness: Improvements in PER and DER confirmed that LOOKAHEAD's predictions were more phonetically grounded. A qualitative example showed the baseline model misrecognizing "la valliere" as "the valet," while the LOOKAHEAD model correctly identified "lavalier."
  • Optimal Window Size: Ablation studies indicated that a lookahead window of 3 tokens ($w=3$) provides the optimal balance between recognition accuracy and computational efficiency.

6. Conclusion and Future Work

LOOKAHEAD offers a practical solution to the hallucination problem in streaming ASR. By providing the language encoder with a brief acoustic preview, the model can self-correct linguistic biases in favor of phonetic evidence. While the additional FFN and IAM layers introduce modest computational overhead, the benefits in accuracy—especially for accented and rare speech—are substantial. Future research may focus on making the lookahead window dynamic, allowing the model to adjust its "peek" based on the clarity of the acoustic signal.

Summary updated

BibTeX
@inproceedings{unni2023improving,
  author = {Vinit S. Unni and Ashish R. Mittal and Preethi Jyothi and Sunita Sarawagi},
  title = {{Improving RNN-Transducers with Acoustic LookAhead}},
  booktitle = {Interspeech},
  year = {2023},
  doi = {10.21437/interspeech.2023-2354},
}
Figure