Abstract
Modern Automatic Speech Recognition (ASR) systems typically process audio utterances as independent events, ignoring the rich acoustic and speaker-specific context available in multi-turn interactions. This paper explores a novel "Contextual-Utterance" training framework that leverages surrounding audio—both past and future—to improve model performance. By employing a dual-mode distillation strategy, we transfer knowledge from a non-causal teacher model (with access to future context) to a streaming student model (limited to past context). Experimental results on a large-scale far-field dataset demonstrate a 6.3% relative reduction in Word Error Rate (WER) and a latency reduction of over 40ms, proving that implicit acoustic adaptation significantly enhances real-time ASR.
Introduction
In the context of voice-controlled assistants, low latency is a critical requirement, leading most production systems to utilize streaming ASR models. These models process audio in real-time, meaning they are inherently causal and cannot look ahead at future audio frames. Traditionally, these systems treat each user request as an isolated segment. However, human communication is naturally contextual; a speaker's voice characteristics, the background noise of the environment, and the specific room acoustics remain relatively consistent across a single session.
While previous attempts to capture this context have relied on explicit speaker-identification modules or complex "i-vector" embeddings, this research proposes an implicit approach. By training models on sequences of utterances rather than isolated clips, the system learns to adapt to the speaker and environment dynamically.
Methodology
The proposed framework is built upon a Conformer-Transducer architecture and introduces two key innovations: Contextual-Utterance Training and Dual-Mode Distillation.
Contextual-Utterance Training
Instead of the standard approach of feeding the encoder a single audio segment, the model is provided with a sequence of utterances. For example, the input might consist of two previous utterances, the current target utterance, and one future utterance. This allows the encoder to model the relationships between segments, effectively performing implicit adaptation to the acoustic environment and the speaker’s vocal profile without requiring explicit metadata.
Dual-Mode Distillation
To bridge the gap between training-time context and inference-time constraints, the authors utilize a "dual-mode" distillation approach. A single model is trained to operate in two distinct modes:
- Teacher (Non-streaming): This mode has access to the full context window, including future utterances. It acts as the performance upper bound.
- Student (Streaming): This mode is restricted to the current and past utterances, mimicking the conditions of real-time inference.
During the training phase, an in-place distillation loss is applied. This loss functions by forcing the student’s probability distributions to match those of the teacher. Consequently, the student model learns to approximate the benefits of seeing future context, even though that context is unavailable during actual deployment.
Experimental Setup and Results
The framework was evaluated using an internal Amazon dataset comprising approximately 50,000 hours of de-identified far-field recordings. The dataset was highly contextual, with over 95% of samples containing at least two prior utterances.
Accuracy Improvements
The inclusion of context yielded substantial improvements in transcription accuracy. The non-streaming teacher model achieved a 12.1% relative reduction in WER. More importantly, the streaming student model—the version intended for production—realized a 6.3% relative WER reduction compared to a baseline model trained without contextual utterances.
Latency Gains
Beyond accuracy, the distillation process significantly improved the model's confidence. This increased confidence allowed the system to emit tokens (words) earlier in the streaming process. The researchers reported a reduction in the "average last token emission latency" of over 40ms, a meaningful improvement for maintaining the fluidity of human-to-AI interactions.
Discussion and Key Insights
Several critical findings emerged from this study:
- Implicit vs. Explicit Adaptation: The model successfully learned to adapt to speakers and environments purely from raw audio context, suggesting that explicit speaker-modeling modules may be unnecessary in the presence of sufficient contextual training.
- The Value of the "Future": Training against a teacher that sees the future makes the streaming student more robust. The distillation process effectively "prepares" the student for the likely trajectory of the audio.
- Optimization Strategy: The researchers noted that "warm-starting" the model—pre-training on single utterances before introducing contextual sequences—was more effective than training on sequences from scratch.
Conclusion
Contextual-Utterance training represents a significant step forward in making ASR systems more aware of their environment. By leveraging surrounding audio through dual-mode distillation, the researchers achieved a rare "win-win" in machine learning: a model that is both more accurate and faster than its predecessors. Future work will likely explore expanding the context window and integrating textual conversation history to further refine the system's predictive capabilities.
Summary updated