Acoustic modeling based on model structure annealing for speech recognition

Sayaka Shiota, Kei Hashimoto, Heiga Zen, Yoshihiko Nankaku, Akinobu Lee, Keiichi TokudaInterspeech 2008 · DOI

This paper, presented at Interspeech 2008, addresses a fundamental "chicken-and-egg" problem in automatic speech recognition (ASR): the interdependence between model parameters and model structure. By applying Deterministic Annealing to the model's structure itself, the authors propose a method to escape the local optima that frequently degrade the performance of traditional Hidden Markov Model (HMM) training.

The Challenge: Context-Dependent Modeling

Modern ASR systems rely on context-dependent models, such as triphones, to account for coarticulation—the way sounds change based on their phonetic neighbors. However, the sheer number of possible triphones leads to data sparsity. To manage this, researchers use context clustering, where phonetic decision trees "tie" similar HMM states together to share training data.

The core difficulty lies in the fact that building an effective decision tree requires accurate HMM state statistics, yet obtaining those statistics requires a well-defined model structure. Standard initialization methods, like "flat-start" or k-means, often break this loop by settling into sub-optimal local maxima, resulting in models that do not fully capture the underlying speech patterns.

Proposed Method: Model Structure Annealing

The researchers introduce Model Structure Annealing, a framework that treats the parameter-tying structure as a hidden variable within the Expectation-Maximization (EM) process. This approach utilizes Deterministic Annealing EM (DAEM) to provide a more global search of the parameter space.

Deterministic Annealing EM (DAEM)

Unlike standard EM, which can be overly "greedy" and prone to local optima, DAEM introduces a "temperature" parameter ($1/\beta$). At high temperatures (low $eta$), the probability distribution over hidden variables is nearly uniform, allowing the model to explore a wide range of state alignments and tying structures. As the system "cools" (as $eta$ increases), the model gradually settles into a stable, optimized configuration.

Conceptual diagram of the annealing process for state sequences (left) and decision trees (right). As the temperature cools, the probability distributions sharpen toward the optimal configuration.

Joint Optimization and Variational Approximation

The authors define a likelihood function that includes both state sequences ($q$) and multiple possible decision trees ($m$) as hidden variables. To make the simultaneous optimization of states and structures computationally tractable, they employ a variational approach to decouple these variables. This allows the system to iteratively update state sequences and tying structures within the annealing schedule. In the early "hot" stages, the model considers a weighted combination of different structures, eventually converging to a single, refined architecture for final decoding.

Experimental Evaluation

The method was evaluated using the ATR Japanese speech database (b-set), focusing on a single male speaker (MHT). The training set consisted of 450 phonetically balanced sentences, while the test set included 53 sentences. The models were three-state left-to-right HMMs for 43 Japanese phonemes, using 19 mel-cepstral coefficients (including delta and delta-delta features).

Key Results and Insights

The "DAEM-joint" method, which anneals both states and trees, was compared against several baselines, including standard k-means initialization and a version that only annealed HMM states.

  • Phoneme Accuracy: DAEM-joint achieved a peak accuracy of 94.16%, significantly outperforming the k-means baseline (93.38%) and the state-only DAEM approach (93.79%). This represents an 11.7% relative error reduction over the conventional k-means method.
  • Log-Likelihood: The proposed method consistently reached higher log-likelihood values on training data, indicating a superior mathematical fit for the acoustic data compared to traditional initialization techniques.
  • Temperature Scheduling: The experiments revealed that the "temperature schedule" parameter ($\alpha$) is critical. As shown in the performance plots, there is a clear "sweet spot" for cooling; cooling too rapidly leads back to local maxima, while cooling too slowly increases computational cost without further gains.
Phoneme accuracy as a function of the temperature schedule parameter alpha. The DAEM-joint approach (purple line) consistently outperforms baselines at the optimal schedule.

Conclusion and Future Work

Model Structure Annealing successfully demonstrates that treating model architecture as a hidden variable can lead to more robust ASR systems. By escaping local optima during the early stages of training, the DAEM-joint approach yields more accurate phonetic models without requiring manual phoneme boundary labels.

While the current study was limited to a single-speaker dataset and a small number of decision trees, it provides a strong proof-of-concept. Future research will focus on scaling this method to larger, multi-speaker datasets and expanding the variety of decision trees used during the annealing process to further enrich the search space.

Summary updated

BibTeX
@inproceedings{shiota2008acoustic,
  author = {Sayaka Shiota and Kei Hashimoto and Heiga Zen and Yoshihiko Nankaku and Akinobu Lee and Keiichi Tokuda},
  title = {{Acoustic modeling based on model structure annealing for speech recognition}},
  booktitle = {Interspeech},
  year = {2008},
  doi = {10.21437/interspeech.2008-111},
}
Figure