A hybrid Maxent/HMM based ASR system

Yasser Hifny, Steve Renals, Neil D. LawrenceInterspeech 2005

TL;DR

This paper introduces a novel hybrid system for Automatic Speech Recognition (ASR) that combines the strengths of Maximum Entropy (MaxEnt) models with traditional Hidden Markov Models (HMMs). By using MaxEnt to directly estimate posterior probabilities of phonetic states, the system aims to overcome limitations of generative acoustic models, leading to improved performance on the TIMIT phone recognition task.

What Problem Does This Paper Address?

Traditional ASR systems often rely on generative acoustic models like Gaussian Mixture Models (GMMs) to model the relationship between acoustic features and phonetic states. While these models estimate likelihoods, they don't directly optimize for posterior probabilities, which are crucial for accurate classification. This can lead to suboptimal performance, especially when dealing with the inherent variability in speech. The paper addresses this by proposing a discriminative approach using Maximum Entropy (MaxEnt) models, which are designed to directly estimate these posterior probabilities, thereby potentially improving the acoustic modeling component of ASR systems.

Methods

The core of this work is the development of a hybrid MaxEnt/HMM system. The MaxEnt models are trained to estimate the posterior probability of phonetic states given acoustic observations. This is achieved by defining a set of "constraints" derived from acoustic matching scores, often generated by Gaussian Mixture Models (GMMs). The MaxEnt principle is then used to find the most unbiased probability distribution that satisfies these constraints.

Key methodological contributions include:

  • Discriminative Acoustic Modeling: Replacing generative GMMs with discriminative MaxEnt models for acoustic scoring.
  • Parametric Constraints: Utilizing finite GMMs to define acoustic constraints, allowing for flexible modeling of acoustic variability.
  • Optimization Procedure: Employing a modified Generalized Improved Iterative Scaling (GIIS) algorithm for efficient parameter estimation of the MaxEnt models.
  • Sparse MaxEnt Models: Introducing a Lasso regularizer to encourage sparsity in the MaxEnt model parameters, aiming to prevent overfitting and improve generalization. This is achieved by adding a penalty term to the conditional maximum likelihood criterion.
  • Hybrid Decoding: Integrating the MaxEnt posterior probabilities into an HMM decoder. The system uses a simple bigram language model and a three-state left-to-right HMM for each phone.

Datasets

  • TIMIT: Used for both training and evaluation of the proposed hybrid MaxEnt/HMM system. The standard 61 phone classes were reduced to 39 labels.

Key Insights and Ideas

  • MaxEnt for Direct Posterior Estimation: The paper highlights the advantage of MaxEnt in directly modeling posterior probabilities, which is a more discriminative approach compared to traditional likelihood-based methods.
  • Parametric Constraints as a Flexible Foundation: Using GMMs to generate constraints provides a flexible way to capture acoustic variability without making strong distributional assumptions.
  • Sparsity for Generalization: The introduction of Lasso regularization is a key insight for improving the generalization performance of MaxEnt models, especially when dealing with a large number of acoustic constraints. This helps in selecting the most relevant constraints and preventing overfitting.
  • Hybrid Approach Synergy: The combination of MaxEnt's discriminative power with HMM's temporal modeling capabilities offers a promising direction for improving ASR accuracy.

Key Results

The proposed hybrid MaxEnt/HMM system achieved a phone accuracy of 66.2% on the TIMIT test set. While this is comparable to some published results, it is noted as being lower than the GMM/HMM HTK system (72.3%) and the Recurrent Neural Network (RNN) system (75%). The training frame accuracy was 69% and the test frame accuracy was 63%.

Limitations and Future Work

  • Acoustic Frame Prior Not Modeled: The paper acknowledges that the acoustic frame prior was not explicitly modeled, which might affect the scaled likelihood approximation during decoding.
  • Performance Gap: The achieved accuracy is lower than some state-of-the-art systems, suggesting room for improvement.
  • Future Work: The authors mention ongoing engineering work to make the system more efficient, particularly in utilizing parameters and selecting discriminative constraints. This is expected to lead to better recognition performance. Further exploration of more advanced decoding techniques and constraint definitions is also implied.

Summary updated

BibTeX
@inproceedings{hifny2005hybrid,
  author = {Yasser Hifny and Steve Renals and Neil D. Lawrence},
  title = {{A hybrid Maxent/HMM based ASR system}},
  booktitle = {Interspeech},
  year = {2005},
}
Figure