Phonetic Error Analysis of Raw Waveform Acoustic Models with Parametric and Non-Parametric CNNs

Erfan Loweimi, Andrea Carmantini, Peter Bell, Steve Renals, Zoran CvetkovicarXiv 2024 · OpenReview

Abstract

In this paper, we analyse the error patterns of raw waveform acoustic models in TIMIT's phone recognition task. Our analysis goes beyond the conventional phone error rate (PER) metric. We categorise phones into broad phonetic classes (BPCs) and compute the PER for each category. We also construct confusion matrices to compare patterns with Filterbank and Wav2vec 2.0 systems. Our models, consisting of parametric (Sinc2Net) or non-parametric CNNs and Bidirectional LSTMs, achieve down to 13.7%/15.2% PERs on TIMIT Dev/Test sets, outperforming reported raw waveform models in the literature. Furthermore, transfer learning from the Wall Street Journal (WSJ) corpus reduces the PER to 11.8%/13.7% on the Dev/Test sets.

Introduction

In speech recognition, the standard metric for success is the Phone Error Rate (PER). While PER quantifies how often a model fails, it provides little insight into the nature of these failures—such as which phonetic classes are most problematic or which sounds are frequently confused.

Traditional systems rely on hand-engineered features like Mel Filterbanks (FBank), which often discard phase information. This research investigates whether "raw waveform" models, which learn features directly from audio, can overcome these limitations. We provide a granular look at error patterns across different broad phonetic classes (BPCs) to understand the specific strengths and weaknesses of raw waveform architectures.

Methods

The proposed acoustic model architecture consists of three primary stages:

  1. Feature Extraction (CNN): We compare standard non-parametric CNNs with Sinc2Net, a parametric CNN where filters are constrained to sinc-squared functions. This approach enhances interpretability and biological plausibility.
  2. Sequential Modeling (BLSTM): A Bidirectional Long Short-Term Memory network captures temporal context.
  3. Classification: Fully connected layers lead to a dual-head output for context-dependent triphones and context-independent monophones, the latter serving as a regularizer.
Model architecture featuring CNN feature extraction, BLSTM sequential modeling, and dual-head classification.

To analyze errors, individual phones were mapped into categorization schemes based on manner of articulation (e.g., fricative, nasal) and voicing.

Key Insights and Results

Performance Benchmarks

The raw waveform models achieved state-of-the-art performance for this class of models, reaching a PER of 13.7% (Dev) / 15.2% (Test). Incorporating transfer learning from the WSJ corpus further improved results to 11.8% (Dev) / 13.7% (Test).

Comparison of PER across architectures and detailed phonetic error breakdown (PER, Substitution, Deletion, Insertion).

Phonetic Analysis

  • The Vowel Challenge: Vowels remain the most difficult class across all models. This is likely due to high speaker-specific variability (identity and emotion) acting as noise for phonetic recognition.
  • The Phase Advantage: Raw waveform models significantly outperformed Filterbank models in recognizing Nasals and Silence. This suggests that preserving phase information in raw waveforms is crucial for these subtle acoustic events.
  • Transfer Learning Synergy: Raw waveform models showed a higher capacity for improvement via transfer learning (17.6% relative gain) compared to Filterbank models (8.4%), indicating they can learn more complex representations when provided with larger datasets.
  • Universal Confusions: Certain errors appear inherent to phonetic similarity regardless of architecture; for example, Plosives are consistently confused with Fricatives.

Conclusion

While raw waveform models excel at capturing specific phonetic nuances like nasals and silence, vowels remain a significant hurdle due to speaker variability. Future work will explore alternative modeling techniques to improve speaker-invariant representations and extend this phonetic analysis to other languages and modern architectures like Transformers.

Summary updated

Abstract

In this paper, we analyse the error patterns of the raw waveform acoustic models in TIMIT's phone recognition task. Our analysis goes beyond the conventional phone error rate (PER) metric. We categorise the phones into three groups: {affricate, diphthong, fricative, nasal, plosive, semi-vowel, vowel, silence}, {consonant, vowel+, silence}, and {voiced, unvoiced, silence} and, compute the PER for each broad phonetic class in each category. We also construct a confusion matrix for each category using the substitution errors and compare the confusion patterns with those of the Filterbank and Wav2vec 2.0 systems. Our raw waveform acoustic models consists of parametric (Sinc2Net) or non-parametric CNNs and Bidirectional LSTMs, achieving down to 13.7%/15.2% PERs on TIMIT Dev/Test sets, outperforming reported PERs for raw waveform models in the literature. We also investigate the impact of transfer learning from WSJ on the phonetic error patterns and confusion matrices. It reduces the PER to 11.8%/13.7% on the Dev/Test sets.
BibTeX
@misc{loweimi2024phonetic,
  author = {Erfan Loweimi and Andrea Carmantini and Peter Bell and Steve Renals and Zoran Cvetkovic},
  title = {{Phonetic Error Analysis of Raw Waveform Acoustic Models with Parametric and Non-Parametric CNNs}},
  year = {2024},
  eprint = {2406.00898},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2406.00898},
}
Figure