Toward an optimum feature set and HMM model parameters for automatic phonetic alignment of spontaneous speech

Montri Karnjanadecha, Stephen A. ZahorianInterspeech 2012

TL;DR

This paper tackles the challenging problem of automatically aligning phonetic boundaries in spontaneous, noisy speech, a crucial step for developing robust speech technologies. They propose an HMM-based forced alignment system that leverages passage-by-passage adaptation and evaluate various acoustic features, finding that DCTC_78 features combined with a 4-state HMM and isolated-word training yield the best results for both recognition and alignment accuracy.

What Problem Does This Paper Address?

Developing automatic speech recognition (ASR) and other speech technologies often relies on accurately labeled speech data, particularly time-aligned phonetic transcriptions. However, creating these labels manually is incredibly time-consuming and expensive. While existing methods for phonetic alignment work well for clean, read speech, they struggle with the variability, noise, and spontaneous nature of real-world speech found in databases like the Open Source Multi-Language Audio (OSMLA) database. This paper aims to bridge this gap by developing an automatic phonetic alignment system that is robust to these challenges and can be applied to unlabeled spontaneous speech.

Methods

The core of the proposed method is an HMM-based ASR system running in "forced alignment" mode. This means the system uses a given speech signal and its corresponding word transcription to find the most likely sequence of phonetic units and their boundaries. The key innovation lies in how the HMM models are trained and adapted:

  • Multi-Pass Training and Adaptation: Instead of relying on pre-trained models, the system adapts existing HMMs (initially trained on the TIMIT database) to the specific characteristics of the target speech data. This adaptation is performed iteratively over multiple passes, using the refined phonetic transcriptions from previous passes to improve subsequent ones. This "passage-by-passage" adaptation is crucial for handling the variability within spontaneous speech.
  • Hybrid Architecture: The system combines dictionary lookup (mapping words to phonetic sequences) with HMMs for alignment. A special "SP" (short pause) phone is added to the dictionary entries to allow the system to automatically insert short pauses between words, improving alignment accuracy.
  • Feature Evaluation: The paper systematically evaluates several acoustic feature sets, including MFCC, LPCC, PLP, and a novel set called DCTC (Discrete Cosine Transform Coefficients). DCTC features, particularly DCTC_78, are explored for their potential to capture spectral and temporal modulations effectively.

Datasets

  • TIMIT: This standard acoustic-phonetic corpus was used for initial HMM training and for evaluating different feature sets and HMM parameters. TIMIT is known for its clean, read speech and detailed phonetic transcriptions.
  • OSMLA (Open Source Multi-Language Audio) Database: This database, comprising spontaneous speech from English, Mandarin, and Russian, was the target for the proposed automatic phonetic alignment system. It is characterized by noise, various accents, and different recording conditions. While the OSMLA database has sentence/phrase level transcriptions, it lacks full phonetic alignment, making it ideal for testing the proposed method. A subset of 212 English passages was used for evaluation.

Key Insights and Ideas

  • Feature Choice Matters Significantly: The study highlights that the choice of acoustic features has a substantial impact on both speech recognition and phonetic alignment accuracy. DCTC features, especially DCTC_78, demonstrated superior performance compared to traditional features like MFCC, LPCC, and PLP.
  • Isolated-Word Training for Alignment: Interestingly, the "isolated-word training" method for HMMs yielded better alignment accuracy than the "full training" method, even though full training resulted in higher recognition accuracy. This suggests that optimizing for alignment might require a different training strategy than optimizing for pure recognition.
  • Passage-by-Passage Adaptation is Key: Adapting HMM models to individual speech passages (segments of a speaker's recording) rather than the entire database or a fixed set of models proved to be the most effective strategy for handling the variability in spontaneous speech.
  • DCTC_78 and 4-State HMMs are a Winning Combination: The research identified DCTC_78 features combined with 4-state HMMs as the optimal configuration for achieving the best balance of recognition and alignment accuracy on the TIMIT dataset.

Key Results

  • DCTC_78 Dominance: The DCTC_78 feature set consistently outperformed other tested features, achieving the highest alignment accuracy (90.4%) with 4-state HMMs and isolated-word training on TIMIT.
  • Improved Alignment on OSMLA: When applied to the OSMLA database, DCTC_78 features showed significantly lower insertion and substitution errors compared to MFCC_39. Table 6 indicates that DCTC_78 achieved higher alignment accuracy across various tolerance levels (except for 100 ms) compared to MFCC_39. For instance, with a 20 ms tolerance, DCTC_78 achieved 82.3% alignment accuracy, while MFCC_39 achieved 78.6%.
  • Robustness to Noise and Spontaneity: The passage-by-passage adaptation strategy, combined with the chosen features and HMM configuration, demonstrated promising results in aligning spontaneous and potentially noisy speech, a significant improvement over methods that rely on clean, read speech.

Limitations and Future Work

  • Limitations: The paper acknowledges that while the proposed system is promising, there's still room for improvement, particularly in handling extremely noisy speech. The current adaptation method might not be fully robust to all types of noise and acoustic conditions present in the OSMLA database.
  • Future Work: The authors plan to focus on:
  • Including noisy speech in the adaptation data to further enhance robustness.
  • Investigating more advanced boundary refinement methods to improve alignment accuracy beyond what HMM alignment alone can achieve.
  • Exploring the inclusion of other types of speech data and further optimizing the adaptation process.

Summary updated

BibTeX
@inproceedings{karnjanadecha2012toward,
  author = {Montri Karnjanadecha and Stephen A. Zahorian},
  title = {{Toward an optimum feature set and HMM model parameters for automatic phonetic alignment of spontaneous speech}},
  booktitle = {Interspeech},
  year = {2012},
}
Figure