Accent- and speaker-specific polyphone decision trees for non-native speech recognition

Dominic Telaar, Mark C. FuhsInterspeech 2013

TL;DR

This paper explores how to improve automatic speech recognition (ASR) for non-native English speakers, specifically those with South-Asian (SoA) accents. They introduce a novel method to quantify differences between accent-specific decision trees and find that while accent-specific models offer benefits, adapting to individual speakers is ultimately more impactful for accuracy, especially with larger models.

What Problem Does This Paper Address?

English is a global language, leading to a vast number of non-native speakers. ASR systems, typically trained on native speakers, often struggle with the pronunciation and speaking style variations inherent in accented speech. Adapting these systems to specific accents or individual speakers is crucial for improving performance. However, the availability of accented speech data is often limited, making it challenging to build robust accent-specific models. This paper investigates how to best leverage limited accented data, comparing the effectiveness of accent-specific models versus speaker-specific adaptation for South-Asian accented English speakers.

Methods

The core of this work lies in analyzing and comparing different approaches to building ASR systems for accented speech. The authors utilize polyphone decision trees, a standard technique in modern ASR for modeling phonetic co-articulations. They introduce a similarity measure based on conditional entropy to quantify the differences between these decision trees, allowing them to analyze how accent and speaker variations manifest in the tree structures.

Their experimental approach involves:

  • Baseline System: A standard ASR system trained on a large dataset of US-accented English.
  • Accent-Specific Systems: Systems trained on data specifically from South-Asian accented speakers.
  • Speaker-Specific Adaptation: Adapting the baseline system to individual speakers using techniques like Maximum A Posteriori (MAP) estimation or Maximum Likelihood (ML) retraining.
  • Decision Tree Analysis: Using their novel similarity measure to compare the structures of decision trees trained on different datasets (US-accented, SoA-accented, and speaker-specific).

Datasets

The study uses several datasets for training and evaluation:

  • Baseline Database: Primarily US-accented English speech, with a maximum of one hour per speaker.
  • US Acc. Database: All training data spoken by native US English speakers.
  • Non-SoA Acc. Database: Speech from speakers not categorized as SoA-accented, including US and other accents.
  • SoA Acc. Database: Speech specifically from South-Asian accented speakers. This dataset is smaller than the others.
  • Test Set: Consists of 94,000 running words from 11 speakers, with substantial adaptation data (65-175 hours per speaker).

Key Insights and Ideas

  • Accent vs. Speaker Specificity: While accent-specific models can improve performance over a general US-accented baseline, adapting to individual speakers yields even greater gains, especially when significant speaker-specific data is available.
  • Decision Tree Structure Matters: The paper demonstrates that accent variations lead to discernible differences in the structure of polyphone decision trees. Their novel similarity measure effectively quantifies these differences.
  • Data Sparsity Challenges: The limited size of the SoA-accented dataset poses a challenge. The authors found that for larger systems (more GMMs), the benefit of accent-specific trees diminishes, suggesting that speaker adaptation becomes more dominant.
  • Co-articulation Differences: Non-native speakers exhibit different co-articulation patterns compared to native speakers, and these differences are reflected in the decision trees. However, individual speaker variations can be more significant than accent-group variations.

Key Results

  • A purely SoA-accented speaker-independent system showed a 27.6% relative reduction in Word Error Rate (WER) compared to the baseline US-accented system.
  • After speaker adaptation, the performance of all systems became nearly the same, indicating that speaker adaptation is highly effective and can overcome accent-specific differences.
  • For smaller systems (1k GMMs), speaker-dependent (SD) systems outperformed adapted baseline and adapted SoA-accented systems. However, this advantage diminished with increasing model size.
  • The similarity analysis revealed that SoA-accented trees differed from US-accented trees, and both were dissimilar to speaker-dependent trees, confirming distinct structural patterns.
  • The impact of the polyphone decision tree on WER decreased quickly as the number of models grew. GMM adaptation/retraining provided significant gains, especially for larger systems.

Limitations and Future Work

  • Data Sparsity: The limited size of the SoA-accented dataset is a significant limitation. The authors suggest that future work could explore more advanced techniques for leveraging small accented datasets.
  • Individual Speaker Variation: The paper highlights that individual speaker variations are substantial and can limit the benefit of accent-specific trees. Future research could focus on better modeling these individual nuances.
  • Generalizability: While the study focuses on SoA-accented speakers, further research could investigate the effectiveness of these methods for other accent groups.
  • Model Size Impact: The diminishing returns of accent-specific trees with larger model sizes warrant further investigation into optimal model configurations for accented speech.

Summary updated

BibTeX
@inproceedings{telaar2013accent,
  author = {Dominic Telaar and Mark C. Fuhs},
  title = {{Accent- and speaker-specific polyphone decision trees for non-native speech recognition}},
  booktitle = {Interspeech},
  year = {2013},
}
Figure