Investigating Phoneme Similarity with Artificially Accented Speech

Margot Masson, Julie Carson-berndsenACL 2023

TL;DR

This paper explores how to generate artificial speech with non-native accents to better understand and model phoneme confusions that challenge automatic speech recognition (ASR) systems. By using text-to-speech (TTS) synthesis to create accented speech and then analyzing the resulting ASR errors, the authors demonstrate that artificial accent data can indeed capture real-world phoneme similarity patterns, offering a promising avenue for improving ASR robustness.

What Problem Does This Paper Address?

Despite significant advancements in automatic speech recognition (ASR) thanks to deep learning, systems still struggle with accented speech, particularly non-native accents. This is largely due to the scarcity of diverse, well-labeled accented speech data for training and testing. Existing approaches often lack explainability regarding why accents cause recognition errors. This paper aims to bridge this gap by investigating whether artificially generated accented speech can effectively mimic the phoneme confusions that occur in real non-native speakers, thereby providing a more data-driven and interpretable way to improve ASR systems' robustness to accents.

Methods

The core of the paper revolves around a "variation method" for generating accented speech. This method involves:

  1. Text-to-Phoneme Transcription: Converting input text into a sequence of phonemes.
  2. Phoneme Variation: Applying accent-specific modifications to these phonemes. This is achieved by:
  3. Phonetic Compatibility Matrix: Identifying phonemes present in the target language (e.g., English) but absent in the speaker's native language (e.g., French).
  4. Similarity Matrix: Determining which native language phonemes are perceived as similar to the absent target language phonemes. The paper explores two paradigms for this:
  5. Knowledge-Based (KB1, KB2): Using phonetic features (like place and manner of articulation) and calculating similarity via Jaccard distance (KB1) or Euclidean distance in a 3D phonetic space (KB2).
  6. Data-Driven (DD1, DD2): * DD1: Directly using a TTS system configured with a French voice to synthesize English text, implicitly capturing accent variations.
  7. DD2: This is the paper's novel contribution. It involves generating artificial French-accented speech (using DD1), then feeding this into an ASR system (Wav2Vec 2.0) to derive a confusion matrix. This confusion matrix then serves as the similarity matrix for introducing variations.
  8. Speech Synthesis: Using a TTS engine to generate audio from the varied phoneme sequences.

The generated accented speech is then evaluated using an ASR system (Wav2Vec 2.0) to measure Word Error Rate (WER) and Phoneme Error Rate (PER), and to analyze phoneme similarity clusters via dendrograms.

Datasets

  • TIMIT Corpus: Used as the source for English text sentences for generating accented speech and as a baseline for natural US-English speech.
  • Microsoft Azure TTS: Utilized as the text-to-speech engine for generating both artificial US-English and French-accented English speech.
  • Wav2Vec 2.0: Employed as the Automatic Speech Recognition (ASR) system for evaluating the generated speech and for deriving confusion matrices.

No new benchmark datasets were introduced; instead, the paper focuses on generating synthetic data and analyzing its impact on existing ASR models.

Key Insights and Ideas

  • Artificial Accents Capture Real Errors: The paper strongly suggests that artificially generated accented speech, particularly when derived from ASR confusions (DD2 method), can effectively mimic the phoneme substitution patterns of non-native speakers.
  • Data-Driven Similarity is Key: The data-driven approach (DD2) yielded more informative phoneme similarity clusters than knowledge-based methods, indicating that real-world ASR errors provide a more nuanced understanding of accent-related confusions.
  • Phoneme Substitutions Drive ASR Errors: The significant drop in ASR accuracy when using accented speech (both artificial and natural) suggests that phoneme substitutions are a primary source of errors for ASR systems, even when language models are present.
  • Explaining Accent Robustness: The methodology offers a path towards understanding why ASR systems struggle with accents, moving beyond just measuring performance degradation.

Key Results

  • Increased Error Rates: As expected, artificial accented speech (all methods) resulted in significantly higher Word Error Rates (WER) and Phoneme Error Rates (PER) compared to natural US-English speech (NV1). For instance, WER increased from ~0.077 for NV1 to ~0.658 for DD1.
  • DD2 Outperforms KB Methods: The data-driven method DD2, which uses ASR-derived confusions, generally showed better performance (lower WER/PER) or more insightful similarity clustering compared to knowledge-based methods (KB1, KB2).
  • Hierarchical Clustering Reveals Patterns: Dendrograms of phoneme confusions showed that the DD2 method captured specific French-English confusions (e.g., 'th' sounds clustering with 's', 'z', 'd', 't'), aligning with known L1-French pronunciation patterns.

Limitations and Future Work

  • ASR Model Influence: The Wav2Vec 2.0 model's fine-tuning and its character-to-phoneme conversion step could influence the observed errors. The authors plan to use a Wav2Vec 2.0 phoneme model in future work.
  • Limited Language Scope: The study focused solely on French-accented English. The authors believe the method is applicable to other languages and plan to explore them.
  • Dataset Specificity: Experiments were conducted only on the TIMIT dataset. Future work will involve testing on other datasets to gain broader insights.
  • Future Directions: The authors intend to enhance their knowledge-based methods with phonotactic constraints and explore how non-native variants are captured in deep learning models. They also aim to develop a methodology for challenging ASR systems to determine how far a variant can deviate from an expected phoneme while still being recognized.

Summary updated

BibTeX
@inproceedings{masson2023investigating,
  author = {Margot Masson and Julie Carson-berndsen},
  title = {{Investigating Phoneme Similarity with Artificially Accented Speech}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2023},
}
Figure