Estimating pronunciation variations from acoustic likelihood score for HMM reconstruction

Yi Liu, Pascale FungInterspeech 2001 · DOI

TL;DR

This paper introduces a novel method to improve speech recognition by better modeling pronunciation variations. Instead of relying on traditional frequency counts, they use acoustic likelihood scores to estimate these variations and then reconstruct Hidden Markov Models (HMMs) by intelligently sharing mixture components, leading to significant accuracy improvements.

What Problem Does This Paper Address?

Speech recognition systems often struggle with spontaneous speech due to the wide range of pronunciation variations that humans naturally employ. Traditional methods for modeling these variations, such as augmenting the lexicon with alternative pronunciations or modifying the decoder, can lead to increased computational complexity and data requirements. Existing approaches to estimate pronunciation probabilities often rely on counting occurrences from alignments, which can be sensitive to the number of states in HMMs and struggle with handling insertions and deletions. This paper aims to address these limitations by proposing a more robust and efficient method for estimating pronunciation variations and integrating them into HMMs.

Methods

The core of this paper lies in two key innovations:

  1. Estimating Pronunciation Variation Probability from Acoustic Likelihood Score: Instead of relying on frequency counts from alignments, the authors propose using the mismatch between acoustic likelihood scores of baseform states and surface form states at the frame level. When a frame-level error occurs (i.e., the recognized phoneme state doesn't match the expected baseform state), they analyze the acoustic likelihoods of the mixture components within the surface form state. The "optimal" mixture component of the surface form state is identified as the one that best explains the input acoustic vector. This frame-level error and optimal component selection are used to calculate a more accurate variation probability. This method is flexible, doesn't require HMMs to have the same number of states, and inherently handles frame-level errors without explicit insertion/deletion modeling.
  2. HMM Reconstruction by Sharing Optimal Mixture Components: Once variation probabilities are estimated, the authors reconstruct the HMMs. Instead of copying entire Gaussian mixture probability density functions (pdfs) from alternate states, they intelligently share only the optimal mixture components identified in the previous step. The mixture weights of the canonical HMM states are then updated to incorporate these shared components, governed by the estimated variation probabilities. This approach avoids dramatically increasing the number of mixture components, thus mitigating computational overhead.

Datasets

  • Training and Evaluation: Hub4NE 1997 Mandarin Broadcast News Database.
  • Acoustic Features: 13 MFCC, 13 AMFCC, and 13 AA MFCC.
  • Data Split: 10 hours of speech for acoustic model training and HMM reconstruction; 724 utterances for testing.
  • Speech Type: Primarily spontaneous speech and conversational speech.

Key Insights and Ideas

  • Acoustic Likelihoods as a Better Indicator of Variation: The paper demonstrates that frame-level acoustic likelihood mismatches are a more accurate indicator of pronunciation variations than simple frequency counts from alignments.
  • Intelligent Component Sharing is Key: Reconstructing HMMs by sharing only the optimal mixture components from alternate states, rather than entire pdfs, is crucial for balancing accuracy gains with computational efficiency.
  • Eliminating Lexicon Expansion: The proposed HMM reconstruction method effectively incorporates pronunciation variations directly into the acoustic models, negating the need to expand the pronunciation lexicon, which simplifies the recognition system.
  • Robustness to HMM Topology: The method is flexible and does not require all HMMs to have the same number of states, making it adaptable to different HMM topologies.

Key Results

  • Syllable Error Rate Reduction: The proposed method achieved an absolute reduction of 2.03% in syllable error rate compared to the baseline system.
  • Improved Accuracy: The accuracy improvement gained from the proposed method was almost double that of previous DP alignment-based methods.
  • Competitive Performance: Reconstructed HMMs with fewer mixtures (11.2 per state) achieved syllable accuracy comparable to or better than baseline systems using more mixtures (16 per state).
  • Lexicon Advantage: Using a one-to-one mapping lexicon with reconstructed HMMs yielded better syllable accuracy than a baseline system with a multi-pronunciation lexicon.

Limitations and Future Work

  • Threshold Setting: The authors suggest that discovering more efficient criteria for setting thresholds (for filtering variation pairs and rare events) is a promising area for future work.
  • Bootstrapping: Utilizing hand-transcribed data for bootstrapping is also mentioned as a future direction.
  • Further Optimization: While the method avoids dramatic increases in mixture components, further research could explore even more optimized ways to manage model complexity.

Summary updated

BibTeX
@inproceedings{liu2001estimating,
  author = {Yi Liu and Pascale Fung},
  title = {{Estimating pronunciation variations from acoustic likelihood score for HMM reconstruction}},
  booktitle = {Interspeech},
  year = {2001},
  doi = {10.21437/eurospeech.2001-20},
}
Figure