Introduction
Distant Speech Recognition (DSR) remains one of the most challenging frontiers in audio processing. Unlike close-talking microphones, distant sensors—such as those in smart speakers—must contend with significant reverberation and background noise. While the industry has developed robust methods for speaker adaptation (e.g., x-vectors), a standardized approach for adapting models to the specific acoustic environment has been missing. This paper introduces R-vectors, a novel self-supervised embedding technique designed to characterize room acoustics and improve Automatic Speech Recognition (ASR) performance.
The R-Vector Methodology
The core innovation lies in treating a room's acoustic profile as a unique "identity." The researchers developed an R-vector extractor using the following steps:
- Self-Supervised Training: Utilizing a room simulator, the authors generated thousands of unique Room Impulse Responses (RIRs). Each RIR was treated as a distinct "class."
- Architecture: A Time-Delay Neural Network (TDNN) was trained to identify which RIR was applied to a clean speech sample. This architecture, similar to those used for speaker recognition, outputs a 512-dimensional embedding.
- Integration: These R-vectors are extracted from input audio and concatenated with standard features like MFCCs or Filterbanks. For models using Convolutional Neural Networks (CNNs), the R-vector is transformed via an affine layer to maintain spatial feature patterns.
Experimental Results and Insights
The effectiveness of R-vectors was evaluated across three diverse datasets: VOiCES19, the AMI Meeting Corpus, and CHiME5.
VOiCES19: Additive Gains
On the VOiCES19 dataset, the R-vector adapted system achieved a 14% relative Word Error Rate (WER) reduction over the baseline. Crucially, the study demonstrated that R-vectors provide complementary gains. When used alongside Weighted Prediction Error (WPE) dereverberation and x-vector speaker adaptation, R-vectors still provided an additional 6.37% relative improvement, suggesting they capture unique acoustic information.
AMI and CHiME5: Real-World Challenges
In the AMI Meeting Corpus, the researchers observed a performance gap between simulated and real-world data. While simulated distant data saw a 14.6% relative improvement, real distant data (SDM) showed a more modest gain of approximately 1.5%.
On the notoriously difficult CHiME5 dataset—featuring dinner party recordings with high noise and overlapping speech—R-vectors achieved a 2.44% relative WER reduction. While smaller than the gains in controlled environments, any improvement on CHiME5 is considered a significant achievement.
Conclusion and Future Directions
The research highlights that environment embeddings are a powerful tool for DSR. A key finding was the importance of Voice Activity Detection (VAD); the extractor performs best when it focuses on speech segments rather than silence.
The primary limitation identified was the simplicity of current "shoebox" room simulators. Future work will focus on more realistic room modeling, iterative refinement using unlabeled real-world data, and the development of multi-task embeddings that simultaneously capture speaker, room, and noise characteristics.
Summary updated