Adaptive stream fusion in multistream recognition of speech

Nima Mesgarani, Samuel Thomas, Hynek HermanskyInterspeech 2011

TL;DR

This paper introduces an adaptive approach to speech recognition that mimics human hearing by processing speech through multiple "streams" representing different acoustic features. By dynamically adjusting how these streams are combined using particle filtering, the system significantly improves robustness to noise and distortions, leading to better phoneme recognition accuracy.

What Problem Does This Paper Address?

Traditional speech recognition systems often struggle with noisy or distorted audio, where the signal quality can vary unpredictably. This paper tackles the challenge of making speech recognition systems more resilient to such non-stationary corruptions. The authors are inspired by the human auditory system, which can effectively filter out noise and focus on relevant acoustic information, even when parts of the signal are degraded. They aim to replicate this adaptive capability in an artificial system by processing speech through multiple parallel "streams," each sensitive to different aspects of the audio signal, and then intelligently fusing the information from these streams.

Methods

The core of the proposed method lies in a multistream recognition paradigm. Instead of processing the raw speech signal directly, the authors first extract various spectrotemporal modulation components from the speech. These components are derived using a bank of 2D Gabor filters tuned to different spectral and temporal frequencies, creating distinct "streams" of information.

For each stream, the system estimates posterior probabilities of phonemes. The crucial innovation is the adaptive fusion of these stream-specific probabilities. This is achieved using a Particle Filter (PF). The PF dynamically adjusts the weights assigned to each stream during fusion. The adaptation process is guided by a statistical model of phoneme posterior estimates. Specifically, the system learns a "clean" model of these statistics from noise-free data and then uses a measure of similarity (Pearson's correlation) between the current noisy stream statistics and the clean model to guide the PF. The PF aims to maximize this similarity, effectively down-weighting streams that are heavily corrupted and up-weighting those that remain informative.

The initial fusion weights are determined by maximizing the correlation between the fused output and the ground truth phoneme labels in the training data.

Datasets

The experiments were conducted using the TIMIT database.

  • Training data: 3000 utterances from 375 speakers.
  • Cross-validation data: 696 utterances from 87 speakers.
  • Test set: 1344 utterances from 168 speakers, sampled at 16 KHz.

The paper also mentions using the Noisex database for additive noises during evaluation.

Key Insights and Ideas

  • Mimicking Human Auditory Processing: The multistream approach is directly inspired by the hypothesized parallel processing in the human auditory system, suggesting that breaking down complex signals into specialized channels can enhance robustness.
  • Spectrotemporal Modulations as Informative Streams: The paper highlights that different spectrotemporal modulation components of speech can be differentially affected by noise, allowing some streams to remain relatively clean even when others are corrupted.
  • Statistical Models for Quality Assessment: Using the autocorrelation matrix of phoneme posterior estimates as a statistical fingerprint of stream quality is an elegant way to assess performance without explicit labels during operation.
  • Particle Filtering for Dynamic Adaptation: The application of Particle Filtering for adaptively adjusting fusion weights is a powerful technique for real-time, non-stationary adaptation, allowing the system to respond to changing noise conditions.
  • Unsupervised Adaptation: The adaptation process is unsupervised in the sense that it doesn't require explicit phoneme labels during the testing phase; it relies on statistical deviations from a learned clean model.

Key Results

  • The proposed adaptive fusion method achieved an average of 13.8% relative error reduction in phoneme recognition accuracy on noisy speech compared to a non-adapted fusion approach.
  • The system demonstrated improved performance across various noise conditions, including additive noises from the Noisex database and specific ripple noises designed to corrupt certain spectrotemporal features more than others.
  • The paper shows that the correlation-based measure for evaluating stream quality stabilizes after approximately 4 seconds of data, making it suitable for practical real-time adaptation.

Limitations and Future Work

  • Limited Number of Streams: The current study focuses on four streams. The authors suggest that increasing the number of streams could potentially lead to further improvements, especially in clean conditions.
  • Constraint on Adaptation: Future work may involve extending the adaptive framework to cases where more constraints on the system's transition dynamics might be necessary for achieving stable adaptation, particularly with a larger number of streams.
  • Generalizability of Noise Models: While the adaptation is unsupervised in terms of labels, it relies on learning a statistical model from clean data. The effectiveness of this approach might depend on how well the learned clean model generalizes to unseen noise types.

Summary updated

BibTeX
@inproceedings{mesgarani2011adaptive,
  author = {Nima Mesgarani and Samuel Thomas and Hynek Hermansky},
  title = {{Adaptive stream fusion in multistream recognition of speech}},
  booktitle = {Interspeech},
  year = {2011},
}
Figure