Phonetic Error Analysis of Raw Waveform Acoustic Models

Erfan Loweimi, Zhengjun Yue, Andrea Carmantini, Zoran Cvetkovic, Steve Renals, Peter BellarXiv 2026 · arXiv

TL;DR

This paper moves beyond the standard aggregate Phone Error Rate (PER) to conduct a granular phonetic error analysis of raw waveform acoustic models on the TIMIT dataset. By combining parametric CNNs with Bidirectional LSTMs, the authors achieve state-of-the-art raw waveform performance and reveal that while these models can outperform traditional hand-engineered filterbanks, their underlying error patterns and phonetic confusions are still fundamentally dictated by inherent acoustic-phonetic similarities.

What Problem Does This Paper Address?

In automatic speech recognition (ASR), systems are typically evaluated using an aggregate metric like Phone Error Rate (PER). While useful, PER is a blunt instrument: it doesn't tell you which types of speech sounds (e.g., vowels, nasals, plosives) the model struggles with, or how it confuses them.

While detailed phonetic error analyses have been performed for traditional acoustic models that rely on hand-crafted features (like Mel Filterbanks), they haven't been deeply explored for raw waveform models. Raw waveform models jointly learn speech parameterization and acoustic modeling using convolutional layers. This paper addresses a key open question: Does this learnable front-end introduce qualitatively different error behaviors, or does it suffer from the same systematic confusions as traditional feature-engineering pipelines?

Methods

The authors propose an acoustic model architecture that pairs a convolutional front-end with sequential modeling layers to process raw audio waveforms:

  1. Convolutional Layer: They experiment with both standard (non-parametric) CNNs and parametric CNNs like SincNet (which uses rectangular filters in the frequency domain) and Sinc2Net (which uses triangular filters to closely mimic Mel Filterbanks).
  2. Sequential Layer: A Bidirectional LSTM (BLSTM) layer captures the bidirectional temporal context of the extracted features.
  3. Multi-Head Output: The model uses a fully connected layer followed by a softmax classifier with two heads: a primary context-dependent (CD) triphone head and a context-independent (CI) monophone head for regularization.
Figure 1: Proposed raw waveform acoustic model architecture. The convolutional layer (parametric or non-parametric) is fol- lowed by BLSTM and fully-connected (FC) layers, with context- dependent (CD) and context-independent (CI) output heads.

To understand the errors, the authors decompose the overall PER across three distinct Broad Phonetic Class (BPC) categorizations (e.g., an 8-class set, a consonant/vowel+/silence set, and a voiced/unvoiced/silence set) and construct detailed confusion matrices from substitution errors.

Datasets

  • TIMIT: Used for phone recognition training, development, and evaluation.
  • Wall Street Journal (WSJ SI-284): Used to evaluate the impact of cross-corpus transfer learning via pre-training.

Key Insights and Ideas

  • The Temporal Dynamics Advantage: Adding BLSTM layers yields massive error reductions for specific BPCs—namely Diphthongs (~28% relative gain), Fricatives (~19%), and Semi-vowels (~18%). These classes rely heavily on temporal trajectories (like gliding formants or sustained spectral context). Conversely, stationary Vowels see much lower gains (~10%), proving that sequential modeling specifically resolves ambiguities in transition-dependent sounds.
  • The Consonant-Vowel Asymmetry in Transfer Learning: Pre-training on the larger WSJ corpus improves consonant recognition roughly three times more than vowel recognition. Why? Vowel identity is highly speaker-dependent and tied to vocal tract geometry. Because WSJ contains more data but fewer unique speakers than TIMIT (284 vs 630), it fails to provide the speaker diversity required to drastically improve vowel discrimination, whereas consonants benefit immensely from the broader variety of phonetic contexts.
  • Nature Over Architecture: The confusion matrices reveal persistent error clusters: Plosives $\leftrightarrow$ Fricatives and Vowels $\leftrightarrow$ Diphthongs $\leftrightarrow$ Semi-vowels. Because these exact patterns mirror those found in traditional Filterbank systems, it proves that dominant model confusions are driven by inherent human acoustic-phonetic similarities rather than artifacts of the chosen feature representation.
BPC confusion matrices across different model configurations

Key Results

  • State-of-the-Art on TIMIT: When trained from scratch, the proposed Sinc2Net+BLSTM model achieves a 15.3% Test PER, outperforming all prior raw waveform models on this benchmark.
  • Surpassing Filterbanks: With WSJ transfer learning, the proposed CNN+BLSTM model reaches 11.3% Dev / 12.3% Test PER, successfully outperforming the strong Filterbank baseline (11.5% / 13.1%).
  • Data Requirements: As shown in the per-BPC breakdown, without transfer learning, Filterbanks outperform raw waveform models due to stronger inductive biases. However, once sufficient data is provided via WSJ pre-training, the jointly learned raw waveform representation surpasses the fixed pipeline across most phonetic classes.
Per-BPC PER comparison between Filterbank and raw waveform models

Limitations and Future Work

The authors note that certain rare phonetic classes, such as Affricates (which only contain two phones in this setup: /ch/ and /jh/), suffer from small sample sizes, leading to high statistical variance in the error analysis.

For future work, the authors suggest leveraging these per-BPC insights to design targeted improvements, such as class-specific data augmentation or specialized loss weighting. They also note that extending this diagnostic framework to modern end-to-end and self-supervised speech models is a natural next step.

Summary updated

Abstract

We analyse error patterns of raw waveform acoustic models on TIMIT phone recognition beyond the overall phone error rate (PER). PER is decomposed across three broad phonetic class (BPC) categorisations, and confusion matrices are constructed from substitution errors. Our models combine parametric (SincNet, Sinc2Net) or non-parametric CNNs with Bidirectional LSTMs, achieving 13.9%/15.3% PER on Dev/Test, the best reported results for raw waveform models on TIMIT. Transfer learning from WSJ reduces PER to 11.3%/12.3%, surpassing the Filterbank baseline. Per-BPC analysis reveals that BLSTM layers benefit transition-dependent classes most, while WSJ transfer learning improves consonants roughly three times more than vowels. Confusion patterns are consistent across raw waveform and Filterbank systems, indicating that the dominant confusions reflect inherent phonetic similarities.
BibTeX
@misc{loweimi2026phonetic,
  author = {Erfan Loweimi and Zhengjun Yue and Andrea Carmantini and Zoran Cvetkovic and Steve Renals and Peter Bell},
  title = {{Phonetic Error Analysis of Raw Waveform Acoustic Models}},
  year = {2026},
  eprint = {2606.07030},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2606.07030},
}
Figure