Abstract
In this paper, we analyse the error patterns of raw waveform acoustic models in TIMIT's phone recognition task. Our analysis goes beyond the conventional phone error rate (PER) metric. We categorise phones into broad phonetic classes (BPCs) and compute the PER for each category. We also construct confusion matrices to compare patterns with Filterbank and Wav2vec 2.0 systems. Our models, consisting of parametric (Sinc2Net) or non-parametric CNNs and Bidirectional LSTMs, achieve down to 13.7%/15.2% PERs on TIMIT Dev/Test sets, outperforming reported raw waveform models in the literature. Furthermore, transfer learning from the Wall Street Journal (WSJ) corpus reduces the PER to 11.8%/13.7% on the Dev/Test sets.
Introduction
In speech recognition, the standard metric for success is the Phone Error Rate (PER). While PER quantifies how often a model fails, it provides little insight into the nature of these failures—such as which phonetic classes are most problematic or which sounds are frequently confused.
Traditional systems rely on hand-engineered features like Mel Filterbanks (FBank), which often discard phase information. This research investigates whether "raw waveform" models, which learn features directly from audio, can overcome these limitations. We provide a granular look at error patterns across different broad phonetic classes (BPCs) to understand the specific strengths and weaknesses of raw waveform architectures.
Methods
The proposed acoustic model architecture consists of three primary stages:
- Feature Extraction (CNN): We compare standard non-parametric CNNs with Sinc2Net, a parametric CNN where filters are constrained to sinc-squared functions. This approach enhances interpretability and biological plausibility.
- Sequential Modeling (BLSTM): A Bidirectional Long Short-Term Memory network captures temporal context.
- Classification: Fully connected layers lead to a dual-head output for context-dependent triphones and context-independent monophones, the latter serving as a regularizer.
To analyze errors, individual phones were mapped into categorization schemes based on manner of articulation (e.g., fricative, nasal) and voicing.
Key Insights and Results
Performance Benchmarks
The raw waveform models achieved state-of-the-art performance for this class of models, reaching a PER of 13.7% (Dev) / 15.2% (Test). Incorporating transfer learning from the WSJ corpus further improved results to 11.8% (Dev) / 13.7% (Test).
Phonetic Analysis
- The Vowel Challenge: Vowels remain the most difficult class across all models. This is likely due to high speaker-specific variability (identity and emotion) acting as noise for phonetic recognition.
- The Phase Advantage: Raw waveform models significantly outperformed Filterbank models in recognizing Nasals and Silence. This suggests that preserving phase information in raw waveforms is crucial for these subtle acoustic events.
- Transfer Learning Synergy: Raw waveform models showed a higher capacity for improvement via transfer learning (17.6% relative gain) compared to Filterbank models (8.4%), indicating they can learn more complex representations when provided with larger datasets.
- Universal Confusions: Certain errors appear inherent to phonetic similarity regardless of architecture; for example, Plosives are consistently confused with Fricatives.
Conclusion
While raw waveform models excel at capturing specific phonetic nuances like nasals and silence, vowels remain a significant hurdle due to speaker variability. Future work will explore alternative modeling techniques to improve speaker-invariant representations and extend this phonetic analysis to other languages and modern architectures like Transformers.
Summary updated