Enhanced Polyphone Decision Tree Adaptation for Accented Speech Recognition

Udhyakumar Nallasamy, Florian Metze, Tanja Schultz

Introduction

Automatic Speech Recognition (ASR) systems are typically trained on vast datasets of standard speech. However, their performance often degrades significantly when encountering regional accents or dialects not well-represented in the training data. The primary technical hurdle lies in the Polyphone Decision Tree (PDT), which clusters acoustic states based on phonetic context. Because accents alter these contextual rules, the tree must be adapted. Unfortunately, capturing accent-specific nuances often requires more data than is typically available, leading to overfitting when using traditional adaptation methods.

Methodology: Semi-continuous PDT Specialization (SPDTS)

To address the challenge of data sparsity, this research introduces Semi-continuous PDT Specialization (SPDTS). This approach moves away from fully continuous models—where every state requires a unique set of complex Gaussian distributions—to a more efficient semi-continuous framework.

The core innovation is a two-level decision tree structure:

  1. Level 1 (Shared Codebooks): The system utilizes a set of shared "codebooks" (clusters of Gaussians) that represent fundamental, accent-independent acoustic building blocks.
  2. Level 2 (Accent-Specific Weights): The tree is further refined into a massive number of "leaf" states. Rather than learning unique Gaussians, these states share the parent node's codebook and only learn a unique set of mixture weights.

By decoupling accent-independent parameters (codebooks) from accent-dependent parameters (mixture weights), the system can grow from 3,000 to 15,000 states using only a few hours of audio, as mixture weights are significantly easier to estimate than full Gaussian parameters.

Experimental Setup

The effectiveness of SPDTS was evaluated on two distinct language pairs with varying amounts of adaptation data, as detailed in the table below.

Dataset characteristics for Arabic and English accent adaptation.
  • Arabic: The source data consisted of 202 hours of Mixed/Modern Standard Arabic (GALE corpus), with 29.7 hours of Levantine accent used for adaptation.
  • English: The source data consisted of 66.3 hours of US English (WSJ corpus), with only 3.0 hours of UK English (WSJCAM0 corpus) available for adaptation.

Key Results and Insights

The SPDTS method consistently outperformed standard adaptation techniques, such as Maximum A Posteriori (MAP) adaptation, across both languages:

  • Word Error Rate (WER) Improvements: The method achieved a 7% relative improvement in WER for the Arabic accent and a 13.7% relative improvement for the English accent compared to the MAP-adapted baseline.
  • Model Fit: Analysis of cross-entropy—a measure of how well the model predicts the adaptation data—demonstrated that SPDTS provided a significantly better fit than other methods.
Cross-entropy comparison across different adaptation methods, showing SPDTS achieving the lowest entropy (best fit).
  • Scalability: The approach proved effective even for large-scale systems. On an 1,100-hour Arabic system, SPDTS maintained a 5.1% relative improvement, indicating its viability for production-grade ASR.

Conclusion and Future Directions

SPDTS demonstrates that in low-data scenarios, increasing the number of states with simplified parameters (mixture weights) is more beneficial than using fewer states with complex parameters. While this work focuses on acoustic variations, future research aims to integrate "Accent Adaptive Training." This would allow a single ASR system to handle multiple accents seamlessly by swapping accent-specific tree layers while maintaining a common set of canonical codebooks.

Summary updated

BibTeX
@inproceedings{nallasamy2012enhanced,
  author = {Udhyakumar Nallasamy and Florian Metze and Tanja Schultz},
  title = {{Enhanced Polyphone Decision Tree Adaptation for Accented Speech Recognition}},
  booktitle = {Interspeech},
  year = {2012},
  doi = {10.21437/interspeech.2012-516},
}
Figure