IVN-Based Joint Training Of GMM And HMMs Using An Improved VTS-Based Feature Compensation For Noisy Speech Recognition

Jun Du, Qiang Huo

Introduction

Automatic Speech Recognition (ASR) systems often perform exceptionally well in controlled, quiet environments but face significant challenges in real-world applications. The introduction of additive noise, such as car engine sounds, and convolutional distortion, such as specific microphone characteristics, causes acoustic features—typically Mel-Frequency Cepstral Coefficients (MFCCs)—to shift away from the training distribution. This paper addresses these challenges by introducing an enhanced framework that combines improved Vector Taylor Series (VTS) feature compensation with a joint training procedure known as Irrelevant Variability Normalization (IVN).

Proposed Methodology

The core of the proposed approach is a multi-faceted upgrade to the standard VTS and training pipeline, designed to increase resilience to noise and channel distortion while maintaining computational efficiency.

Enhanced VTS Feature Compensation

The authors propose several key improvements to the VTS process:

  1. Boosted Mixture Learning (BML): By employing BML for training the reference "clean" Gaussian Mixture Model (GMM), the system achieves high performance with a compact 32-component model.
  2. Selective High-Order VTS (+HO): To balance accuracy and computational load, high-order VTS approximations are applied selectively to noisy speech mean parameters.
  3. Acoustic Context Expansion (+ACE): Temporal context is leveraged by using a weighted average of neighboring frames to calculate sound probabilities.
  4. Gaussian Convolutional Modeling (+HSG): Instead of treating microphone distortion as a fixed constant, it is modeled as a Gaussian distribution to better handle non-stationary distortions.

IVN-Based Joint Training

A critical innovation is the joint training of the feature compensator and the recognizer (HMMs). Using Single Pass Retraining (SPR), the system iteratively updates both components, ensuring the recognizer is perfectly tuned to the specific artifacts of the compensated data.

Proposed IVN-based joint training and recognition framework.

Experimental Setup

The framework was evaluated using the Aurora3 databases, which contain digit strings recorded in real automobile environments across four languages: German, Danish, Finnish, and Spanish. Two primary conditions were tested: Well-Matched (WM), where training and testing environments are similar, and High-Mismatch (HM), where the system is trained on clean close-talk microphones but tested on noisy hands-free microphones.

Results and Analysis

The improvements were most significant in the challenging High-Mismatch (HM) scenarios. The baseline system struggled particularly with the Danish dataset due to non-stationary channel distortion; the introduction of Gaussian convolutional modeling (+HSG) significantly mitigated this issue, raising accuracy from 54.78% to 81.70%.

Word accuracy (%) for the High-Mismatch (HM) task across four languages, showing incremental gains from VTS improvements.

As shown in the results, each proposed enhancement contributed to a steady increase in word accuracy. In the Finnish HM task, accuracy rose from a baseline of 77.07% to 89.52% with the full suite of VTS improvements. Furthermore, the complete "IVN-Joint" system consistently outperformed VTS-only systems, reaching 94.80% accuracy in the Spanish Well-Matched task compared to a 93.57% baseline.

Conclusion

This research demonstrates that combining physical noise modeling (VTS) with joint training (IVN) provides a robust solution for noisy speech recognition in automotive environments. While the iterative training process adds complexity, the gains in accuracy—especially in high-mismatch environments—are substantial. Future work may involve scaling these techniques to large-vocabulary continuous speech recognition (LVCSR) and exploring their integration with modern deep learning architectures.

Summary updated

Abstract

In our previous work, we proposed a feature compensation approach using high-order vector Taylor series approximation for noisy speech recognition. In this paper, first we improve the feature compensation in both efficiency and accuracy by boosted mixture learning of GMM, applying higher order information of VTS approximation only to the noisy speech mean parameters, acoustic context expansion, and modeling the convolutional distortion as a single Gaussian. Then we design a procedure to perform irrelevant variability normalization based joint training of GMM and HMM using VTS-based feature compensation. The effectiveness of our proposed approach is confirmed by experiments on Aurora3 database.
BibTeX
@inproceedings{du2012ivn,
  author = {Jun Du and Qiang Huo},
  title = {{IVN-Based Joint Training Of GMM And HMMs Using An Improved VTS-Based Feature Compensation For Noisy Speech Recognition}},
  booktitle = {Interspeech},
  year = {2012},
  doi = {10.21437/interspeech.2012-378},
}
Figure