Towards fusion of feature extraction and acoustic model training: a top down process for robust speech recognition

Yu-Hsiang Bosco Chiu, Bhiksha Raj, Richard M. SternInterspeech 2009 · DOI

Introduction

Automatic Speech Recognition (ASR) systems often struggle when moving from controlled laboratory settings to the unpredictable noise of the real world. Traditional feature extraction methods, such as Mel-Frequency Cepstral Coefficients (MFCCs), are static "bottom-up" processes. They treat every signal the same way, regardless of the acoustic environment or the specific requirements of the downstream classifier.

This research, presented at Interspeech 2009, proposes a shift toward a "top-down" approach inspired by the human auditory system. In biology, efferent nerve fibers allow the brain to tune the response of the cochlea. By fusing feature extraction with acoustic model training, the authors demonstrate a method to discriminatively learn optimal parameters for a physiologically motivated front-end.

Methodology: Learning the Rate-Level Nonlinearity

The core of the proposed method is the optimization of a rate-level nonlinearity, which mimics the sigmoidal firing rate of auditory neurons in response to sound intensity.

The Front-End Architecture

The system utilizes a standard Mel-filter bank, but replaces the simple log-energy step with a learnable sigmoidal function: $$x_i[t] = \frac{\alpha[i]}{1 + \exp(w_1[i] \cdot y_i[t] + w_0[i])}$$ Here, $\alpha$, $w_0$, and $w_1$ represent the gain, offset, and slope of the nonlinearity for each frequency channel $i$.

Discriminative Parameter Optimization

Rather than relying on fixed physiological data, the authors treat these variables as learnable parameters. To optimize them, they employ a Maximum Mutual Information (MMI) criterion.

The discriminative learning loop for optimizing front-end parameters.

As shown in the figure above, the parameters are updated via gradient descent based on the performance of a proxy classifier. This creates a feedback loop where the needs of the phoneme classifier directly inform how the "digital ear" extracts features from the raw signal.

Experimental Setup

The model was trained using the DARPA Resource Management (RM) dataset. To simulate real-world challenges, pink noise was added to the training data at a 10dB Signal-to-Noise Ratio (SNR). The system was then evaluated against unseen noise types—including babble, market, and restaurant noise—at various intensities ranging from 0dB to 20dB.

Key Results and Discussion

The "Learned" rate-level nonlinearity demonstrated superior robustness compared to both standard MFCCs and fixed physiological models.

Accuracy comparison in restaurant noise across different SNR levels.

As illustrated in the results for restaurant noise, the learned model maintains significantly higher accuracy as noise levels increase (lower SNR). While MFCC performance drops sharply at 5dB and 0dB, the discriminatively trained front-end preserves more critical speech information.

Analysis of the learned parameters revealed that the model does not apply a uniform compression. Instead, it learns frequency-dependent profiles, suggesting that the optimal way to process speech in noise involves a complex, non-linear tuning that traditional static features cannot capture.

Conclusion

By integrating feature extraction into the training loop, this research provides a pathway toward more resilient ASR. The success of the learned sigmoidal nonlinearity highlights the value of biologically inspired, data-driven architectures. Future work may explore scaling this approach to larger datasets and deeper neural network architectures, further bridging the gap between human and machine hearing.

Summary updated

BibTeX
@inproceedings{chiu2009towards,
  author = {Yu-Hsiang Bosco Chiu and Bhiksha Raj and Richard M. Stern},
  title = {{Towards fusion of feature extraction and acoustic model training: a top down process for robust speech recognition}},
  booktitle = {Interspeech},
  year = {2009},
  doi = {10.21437/interspeech.2009-6},
}
Figure