The STC ASR System for the VOiCES from a Distance Challenge 2019

Ivan Medennikov, Yuri Y. Khokhlov, Aleksei Romanenko, Ivan Sorokin, Anton Mitrofanov, Vladimir Bataev, Andrei Andrusenko, Tatiana Prisyach, Mariya Korenevskaya, Oleg Petrov, Alexander ZatvornitskiyInterspeech 2019

TL;DR

This paper presents the STC team's winning system for the VOiCES from a Distance Challenge 2019, tackling the significant challenge of distant, noisy speech recognition. They achieved state-of-the-art results by combining advanced acoustic modeling with novel speaker and room acoustics adaptation techniques, particularly their proposed "R-vectors."

What Problem Does This Paper Address?

Automatic Speech Recognition (ASR) systems typically perform best in controlled environments with clean speech. However, real-world applications often involve speech recorded in noisy, reverberant spaces with multiple speakers, a scenario known as Distant Speech Recognition (DSR). The VOiCES from a Distance Challenge specifically aimed to push the boundaries of DSR by providing a dataset with far-field, noisy speech, creating a significant mismatch with the clean, close-microphone training data available (LibriSpeech). This mismatch is the core problem the STC team sought to overcome.

Methods

The STC system employed a multi-stage approach:

  • Data Augmentation: To bridge the gap between clean training and noisy testing data, they heavily relied on room acoustics simulation. This involved generating simulated reverberant environments and adding various noise sources (from MUSAN, AURORA 2, QUT-NOISE, and TV noise) to the clean training data, effectively creating a much larger and more diverse training set.
  • Dereverberation: The Weighted Prediction Error (WPE) algorithm was used to reduce reverberation effects from both the simulated and original training data, as well as the development/evaluation data.
  • Acoustic Model Architectures: They experimented with several deep learning architectures, including CNN-TDNN-F, TDNN-LSTMP, and BLSTMP-F. The CNN-TDNN-F architecture proved to be the most effective.
  • Adaptation Techniques:
    • Speaker Adaptation: They utilized modern x-vectors, which are DNN-based speaker embeddings, to adapt the acoustic models to individual speakers.
  • Room Acoustics Adaptation (Novelty): A key contribution is their proposed R-vectors. These are embeddings derived from Room Impulse Responses (RIRs) using a Kaldi x-vector framework, designed to adapt models to specific acoustic environments. They found R-vectors provided a more significant improvement than x-vectors alone and that combining them yielded the best results.
  • Lattice-Level Combination: Multiple acoustic models (6 in total), trained with different pronunciation dictionaries and input features, were combined at the lattice level to leverage their complementary strengths.
  • Language Model Rescoring: N-best hypotheses from the combined acoustic models were further refined using Neural Network Language Models (NNLMs). They explored Transformer-XL and FRAGE models trained on both words and sub-word units (Morfessor and BPE), finding sub-word models to be particularly effective.
  • Out-of-Vocabulary (OOV) Handling: To address the issue of words not present in the vocabulary, they augmented the language model training data with artificially generated texts from character-level NNLMs.

Datasets

  • Training Data: An 80-hour subset of the LibriSpeech corpus was used for initial model training. This data was augmented with simulated data, resulting in approximately 1894 hours of training material.
  • Development/Evaluation Data: Portions of the VOICES corpus were used for development and evaluation. This corpus consists of far-field, noisy speech, creating the challenge's core difficulty.

Key Insights and Ideas

  • Room Acoustics Simulation is Crucial: The paper strongly emphasizes that simulating realistic acoustic environments is paramount for tackling DSR, effectively bridging the gap between clean training and noisy testing data.
  • R-vectors Offer Significant Adaptation: The novel R-vector approach for room acoustics adaptation proved to be highly effective, outperforming speaker adaptation alone and being complementary to it. This suggests that explicitly modeling room acoustics can be as, if not more, important than speaker characteristics in DSR.
  • Sub-word NNLMs Excel: Using sub-word units (Morfessor and BPE) in NNLMs for rescoring significantly improved performance, indicating their ability to better handle linguistic variations and unseen word combinations.
  • Lattice Combination Pays Off: Combining multiple acoustic models trained with diverse settings is a robust strategy for achieving better overall performance.

Key Results

The STC system achieved the best Word Error Rate (WER) of 14.7% on the evaluation data for the VOiCES from a Distance Challenge 2019. This was a significant improvement over baseline systems and other participants. For instance, their final combined system with NNLM rescoring achieved a WER of 12.4% on the development set, compared to 15.31% for a baseline n-gram system.

Limitations and Future Work

  • Speech Synthesis Data Augmentation: While explored, speech synthesis-based data augmentation did not yield improvements, likely due to the limited training data for the synthesis models.
  • End-to-End Systems: The authors note that end-to-end systems were not as effective as expected for OOV handling, again attributing this to data limitations.
  • WPE vs. Other Enhancers: While WPE was used, other speech enhancement techniques like GANs and Wavenet were investigated but did not provide consistent benefits when combined with simulated data and adaptation techniques. The paper suggests that joint training of enhancers with acoustic models on simulated data might be a promising avenue.
  • Future Work: The paper implicitly suggests further exploration of advanced data augmentation techniques, potentially more sophisticated end-to-end models, and deeper investigation into the synergy between various speech enhancement and adaptation methods.

Summary updated

BibTeX
@inproceedings{medennikov2019stc,
  author = {Ivan Medennikov and Yuri Y. Khokhlov and Aleksei Romanenko and Ivan Sorokin and Anton Mitrofanov and Vladimir Bataev and Andrei Andrusenko and Tatiana Prisyach and Mariya Korenevskaya and Oleg Petrov and Alexander Zatvornitskiy},
  title = {{The STC ASR System for the VOiCES from a Distance Challenge 2019}},
  booktitle = {Interspeech},
  year = {2019},
}
Figure