Adding noise to improve noise robustness in speech recognition

Nicolás Morales, Liang Gu, Yuqing GaoInterspeech 2007

TL;DR

This paper proposes a novel approach to improve Automatic Speech Recognition (ASR) robustness in noisy environments by adding a "masking noise" to both training and testing data. This technique aims to reduce the mismatch between training and testing conditions, particularly for challenging noise types like cross-talk, leading to significant accuracy gains.

What Problem Does This Paper Address?

Automatic Speech Recognition (ASR) systems are notoriously susceptible to performance degradation when speech is corrupted by noise. This is a major hurdle for real-world applications where clean speech is a rarity. Traditional approaches to combat this include developing noise-robust feature extraction methods, employing speech enhancement techniques to clean up the signal, or adapting acoustic models to the noisy environment. However, many of these methods struggle with non-stationary noises or require prior knowledge of the noise type, which is often unavailable. This paper tackles the problem of improving ASR accuracy and robustness in the face of unknown and potentially non-stationary corrupting noises by introducing a counter-intuitive but effective strategy: adding more noise.

Methods

The core idea is to add a "masking noise" to both the training and testing speech data. This masking noise is chosen to be "well-behaved," meaning it's typically stationary or quasi-stationary. The hypothesis is that this added noise will effectively mask the corrupting noise, thereby reducing the acoustic mismatch between the training and testing conditions.

Mathematically, the authors show that adding masking noise can reduce the gap between the expected distributions of acoustic models and the test data in a logarithmic space. They also explore combining the outputs of multiple ASR systems, each trained with different masking noises and at different Signal-to-Noise Ratios (SNRs), using the ROVER (Recognizer Output Voting Error Reduction) strategy. This allows for a more comprehensive approach to handling varying noise conditions.

Datasets

The experiments were conducted using the TIMIT dataset. Both training and test data were derived from the male speaker files within TIMIT. Artificial noises (cross-talk, babble, white noise, humvee vehicle noise) were added to the clean TIMIT speech to simulate various noisy conditions.

Key Insights and Ideas

  • Masking Noise as a Solution: Adding a well-behaved noise (masking noise) to both training and testing data can surprisingly improve ASR performance in the presence of unknown corrupting noises.
  • Reducing Mismatch: The primary mechanism is reducing the mismatch between training and testing conditions, making the ASR system more resilient to variations.
  • Cross-talk Noise Advantage: The approach is particularly effective for masking cross-talk noise, where background speech interferes with the target speech. Babble noise, in particular, proved adept at masking cross-talk.
  • ROVER for Enhanced Robustness: Combining outputs from multiple ASR systems trained with different masking noises and SNRs using ROVER further boosts accuracy and robustness, especially when corrupting noises vary rapidly.
  • Non-Deterministic Approach: The method is non-deterministic, aligning better with the random nature of noise compared to deterministic enhancement techniques.

Key Results

  • The proposed masking noise approach consistently improved accuracy across various corrupting noises and SNR levels.
  • For cross-talk corrupting noise, the relative accuracy gain with the proposed approach was over 24% compared to a system trained with clean speech.
  • Using babble noise for masking cross-talk corrupting speech yielded a significant relative improvement of 17.2% over the baseline accuracy, compared to only 0.6% when using training noise without masking.
  • Combining outputs from multiple masking-noise-based recognizers using ROVER provided a relative accuracy gain of 8.7% over the best individual output and 24.2% over the system trained with clean speech.

Limitations and Future Work

  • SNR Optimization: The paper acknowledges that finding the optimal SNR level for training and masking noises is crucial. While they used levels that maximized performance, real-world applications would require automatic noise analysis to determine these levels.
  • Constant Noise Levels: The experiments used constant noise levels throughout speech files. Future work aims to apply different levels of masking noise dynamically based on the energy of the corrupting noise in each speech segment.
  • Masking Noise Types: The authors plan to investigate other types of masking noises that might be more suitable for specific corrupting noises, potentially exploring combinations of noises.
  • Comparison with Synthetic Noise: A future comparison with methods that employ synthetic noise, like that proposed by Van Compernolle, is also planned.

Summary updated

BibTeX
@inproceedings{morales2007adding,
  author = {Nicolás Morales and Liang Gu and Yuqing Gao},
  title = {{Adding noise to improve noise robustness in speech recognition}},
  booktitle = {Interspeech},
  year = {2007},
}
Figure