Contextual-Utterance Training for Automatic Speech Recognition

Alejandro Gómez Alanís, Lukas Drude, Andreas Schwarz, Rupak Vignesh Swaminathan, Simon Wiesler

Abstract

Modern Automatic Speech Recognition (ASR) systems typically process audio utterances as independent events, ignoring the rich acoustic and speaker-specific context available in multi-turn interactions. This paper explores a novel "Contextual-Utterance" training framework that leverages surrounding audio—both past and future—to improve model performance. By employing a dual-mode distillation strategy, we transfer knowledge from a non-causal teacher model (with access to future context) to a streaming student model (limited to past context). Experimental results on a large-scale far-field dataset demonstrate a 6.3% relative reduction in Word Error Rate (WER) and a latency reduction of over 40ms, proving that implicit acoustic adaptation significantly enhances real-time ASR.

Introduction

In the context of voice-controlled assistants, low latency is a critical requirement, leading most production systems to utilize streaming ASR models. These models process audio in real-time, meaning they are inherently causal and cannot look ahead at future audio frames. Traditionally, these systems treat each user request as an isolated segment. However, human communication is naturally contextual; a speaker's voice characteristics, the background noise of the environment, and the specific room acoustics remain relatively consistent across a single session.

While previous attempts to capture this context have relied on explicit speaker-identification modules or complex "i-vector" embeddings, this research proposes an implicit approach. By training models on sequences of utterances rather than isolated clips, the system learns to adapt to the speaker and environment dynamically.

Methodology

The proposed framework is built upon a Conformer-Transducer architecture and introduces two key innovations: Contextual-Utterance Training and Dual-Mode Distillation.

Contextual-Utterance Training

Instead of the standard approach of feeding the encoder a single audio segment, the model is provided with a sequence of utterances. For example, the input might consist of two previous utterances, the current target utterance, and one future utterance. This allows the encoder to model the relationships between segments, effectively performing implicit adaptation to the acoustic environment and the speaker’s vocal profile without requiring explicit metadata.

Dual-Mode Distillation

To bridge the gap between training-time context and inference-time constraints, the authors utilize a "dual-mode" distillation approach. A single model is trained to operate in two distinct modes:

  1. Teacher (Non-streaming): This mode has access to the full context window, including future utterances. It acts as the performance upper bound.
  2. Student (Streaming): This mode is restricted to the current and past utterances, mimicking the conditions of real-time inference.

During the training phase, an in-place distillation loss is applied. This loss functions by forcing the student’s probability distributions to match those of the teacher. Consequently, the student model learns to approximate the benefits of seeing future context, even though that context is unavailable during actual deployment.

Experimental Setup and Results

The framework was evaluated using an internal Amazon dataset comprising approximately 50,000 hours of de-identified far-field recordings. The dataset was highly contextual, with over 95% of samples containing at least two prior utterances.

Accuracy Improvements

The inclusion of context yielded substantial improvements in transcription accuracy. The non-streaming teacher model achieved a 12.1% relative reduction in WER. More importantly, the streaming student model—the version intended for production—realized a 6.3% relative WER reduction compared to a baseline model trained without contextual utterances.

Latency Gains

Beyond accuracy, the distillation process significantly improved the model's confidence. This increased confidence allowed the system to emit tokens (words) earlier in the streaming process. The researchers reported a reduction in the "average last token emission latency" of over 40ms, a meaningful improvement for maintaining the fluidity of human-to-AI interactions.

Discussion and Key Insights

Several critical findings emerged from this study:

  • Implicit vs. Explicit Adaptation: The model successfully learned to adapt to speakers and environments purely from raw audio context, suggesting that explicit speaker-modeling modules may be unnecessary in the presence of sufficient contextual training.
  • The Value of the "Future": Training against a teacher that sees the future makes the streaming student more robust. The distillation process effectively "prepares" the student for the likely trajectory of the audio.
  • Optimization Strategy: The researchers noted that "warm-starting" the model—pre-training on single utterances before introducing contextual sequences—was more effective than training on sequences from scratch.

Conclusion

Contextual-Utterance training represents a significant step forward in making ASR systems more aware of their environment. By leveraging surrounding audio through dual-mode distillation, the researchers achieved a rare "win-win" in machine learning: a model that is both more accurate and faster than its predecessors. Future work will likely explore expanding the context window and integrating textual conversation history to further refine the system's predictive capabilities.

Summary updated

Abstract

Recent studies of streaming automatic speech recognition (ASR) recurrent neural network transducer (RNN-T)-based systems have fed the encoder with past contextual information in order to improve its word error rate (WER) performance. In this paper, we first propose a contextual-utterance training technique which makes use of the previous and future contextual utterances in order to do an implicit adaptation to the speaker, topic and acoustic environment. Also, we propose a dual-mode contextual-utterance training technique for streaming ASR systems. This proposed approach allows to make a better use of the available acoustic context in streaming models by distilling “in-place” the knowledge of a teacher (non-streaming mode), which is able to see both past and future contextual utterances, to the student (streaming mode) which can only see the current and past contextual utterances. The experimental results show that a state-of-the-art conformer-transducer system trained with the proposed techniques outperforms the same system trained with the classical RNN-T loss. Specifically, the proposed technique is able to reduce both the WER and the average last token emission latency by more than 6% and 40 ms relative, respectively.
BibTeX
@inproceedings{alanis2022contextual,
  author = {Alejandro Gómez Alanís and Lukas Drude and Andreas Schwarz and Rupak Vignesh Swaminathan and Simon Wiesler},
  title = {{Contextual-Utterance Training for Automatic Speech Recognition}},
  booktitle = {IberSPEECH},
  year = {2022},
  url = {https://openreview.net/forum?id=BnGEfhE7Jy},
}
Figure