R-Vectors: New Technique for Adaptation to Room Acoustics

Yuri Y. Khokhlov, Alexander Zatvornitskiy, Ivan Medennikov, Ivan Sorokin, Tatiana Prisyach, Aleksei Romanenko, Anton Mitrofanov, Vladimir Bataev, Andrei Andrusenko, Mariya Korenevskaya, Oleg PetrovInterspeech 2019 · DOI

Introduction

Distant Speech Recognition (DSR) remains one of the most challenging frontiers in audio processing. Unlike close-talking microphones, distant sensors—such as those in smart speakers—must contend with significant reverberation and background noise. While the industry has developed robust methods for speaker adaptation (e.g., x-vectors), a standardized approach for adapting models to the specific acoustic environment has been missing. This paper introduces R-vectors, a novel self-supervised embedding technique designed to characterize room acoustics and improve Automatic Speech Recognition (ASR) performance.

The R-Vector Methodology

The core innovation lies in treating a room's acoustic profile as a unique "identity." The researchers developed an R-vector extractor using the following steps:

  1. Self-Supervised Training: Utilizing a room simulator, the authors generated thousands of unique Room Impulse Responses (RIRs). Each RIR was treated as a distinct "class."
  2. Architecture: A Time-Delay Neural Network (TDNN) was trained to identify which RIR was applied to a clean speech sample. This architecture, similar to those used for speaker recognition, outputs a 512-dimensional embedding.
  3. Integration: These R-vectors are extracted from input audio and concatenated with standard features like MFCCs or Filterbanks. For models using Convolutional Neural Networks (CNNs), the R-vector is transformed via an affine layer to maintain spatial feature patterns.

Experimental Results and Insights

The effectiveness of R-vectors was evaluated across three diverse datasets: VOiCES19, the AMI Meeting Corpus, and CHiME5.

VOiCES19: Additive Gains

On the VOiCES19 dataset, the R-vector adapted system achieved a 14% relative Word Error Rate (WER) reduction over the baseline. Crucially, the study demonstrated that R-vectors provide complementary gains. When used alongside Weighted Prediction Error (WPE) dereverberation and x-vector speaker adaptation, R-vectors still provided an additional 6.37% relative improvement, suggesting they capture unique acoustic information.

Table 1: WER results on the VOiCES19 dataset. R-vectors show significant relative gains that are additive when combined with existing dereverberation (WPE) and speaker adaptation (x-vector) techniques.

AMI and CHiME5: Real-World Challenges

In the AMI Meeting Corpus, the researchers observed a performance gap between simulated and real-world data. While simulated distant data saw a 14.6% relative improvement, real distant data (SDM) showed a more modest gain of approximately 1.5%.

Table 2: Performance on the AMI dataset. The results highlight a 14.6% gain on simulated data (IHM_rs) versus a smaller 1.56% gain on real distant microphone (SDM) data.

On the notoriously difficult CHiME5 dataset—featuring dinner party recordings with high noise and overlapping speech—R-vectors achieved a 2.44% relative WER reduction. While smaller than the gains in controlled environments, any improvement on CHiME5 is considered a significant achievement.

Conclusion and Future Directions

The research highlights that environment embeddings are a powerful tool for DSR. A key finding was the importance of Voice Activity Detection (VAD); the extractor performs best when it focuses on speech segments rather than silence.

The primary limitation identified was the simplicity of current "shoebox" room simulators. Future work will focus on more realistic room modeling, iterative refinement using unlabeled real-world data, and the development of multi-task embeddings that simultaneously capture speaker, room, and noise characteristics.

Summary updated

BibTeX
@inproceedings{khokhlov2019r,
  author = {Yuri Y. Khokhlov and Alexander Zatvornitskiy and Ivan Medennikov and Ivan Sorokin and Tatiana Prisyach and Aleksei Romanenko and Anton Mitrofanov and Vladimir Bataev and Andrei Andrusenko and Mariya Korenevskaya and Oleg Petrov},
  title = {{R-Vectors: New Technique for Adaptation to Room Acoustics}},
  booktitle = {Interspeech},
  year = {2019},
  doi = {10.21437/interspeech.2019-2645},
}
Figure