End-to-end Phoneme Sequence Recognition using Convolutional Neural Networks

Dimitri Palaz, Ronan Collobert, Mathew Magimai-DossarXiv 2013 · OpenReview

Abstract

Most phoneme recognition state-of-the-art systems rely on classical neural network classifiers fed with highly tuned features, such as Mel Frequency Cepstrum Coefficients (MFCC). Recent advances in deep learning have questioned these systems, yet many state-of-the-art models still rely on hand-crafted features. This paper investigates a convolutional neural network (CNN) approach for raw speech signals. We demonstrate that it is possible to learn an end-to-end phoneme sequence classifier directly from raw signals, achieving performance on the TIMIT and WSJ datasets comparable to existing systems based on MFCCs, questioning the need for complex hand-crafted features on large datasets.

Introduction

For decades, Automatic Speech Recognition (ASR) has followed a "divide and conquer" pipeline. Raw audio is typically converted into hand-crafted features like MFCCs or Perceptual Linear Prediction (PLP) features, based on human knowledge of acoustics and psychoacoustics. These features are then processed by a classifier (e.g., an ANN) and a decoder (e.g., an HMM). This reliance on manual feature engineering represents a potential bottleneck in the development of truly autonomous systems.

While deep learning has revolutionized computer vision by learning representations directly from pixels, the speech community has historically relied on pre-processed features. This research asks a fundamental question: Can we build a truly end-to-end system that learns its own features directly from the raw speech signal?

Proposed Architecture

The proposed system is a unified, end-to-end architecture composed of a Convolutional Neural Network (CNN) and a Conditional Random Field (CRF) decoder.

High-level end-to-end architecture combining a CNN for feature learning and modeling with a CRF for decoding.

The system processes a sequence of raw speech signal frames. The CNN handles both feature learning and modeling, while the CRF performs the final decoding into a phoneme sequence. Crucially, the entire pipeline is trained using backpropagation to maximize the log-likelihood of the correct label sequence.

Convolutional Neural Network Design

The CNN is divided into two distinct stages to replace the traditional feature extraction and classification pipeline:

  1. Filter Stage (Feature Learning): This stage consists of several convolutional layers that apply filters over the temporal signal. These are followed by max-pooling layers, which provide robustness against slight temporal distortions (temporal invariance), and tanh non-linearities.
  2. Classification Stage (Modeling): The learned features are fed into linear layers that produce scores for each phoneme class.
Detailed view of the CNN architecture, showing the filter stage for feature learning and the classification stage for modeling.

Experimental Evaluation

The approach was evaluated on two standard benchmarks:

  • TIMIT Corpus: A classic acoustic-phonetic dataset. The researchers tested various granularities, including a standard 39-phoneme set and a more complex 183-state set (3 states per phoneme).
  • Wall Street Journal (WSJ): A larger corpus (SI-284 set, ~80 hours) used to test the scalability of the raw-signal approach to larger datasets.

Key Results

The raw-signal CNN approach proved highly competitive with traditional MFCC-based baselines:

  • TIMIT: The CNN+CRF system using 183 states achieved a test accuracy of 70.08%, significantly outperforming the MFCC-based baseline of 66.65%.
  • WSJ: On the larger dataset, the raw-signal system achieved 72.88% accuracy, compared to the baseline's 72.39%. This confirmed that the approach scales effectively to large-scale ASR tasks.

Discussion and Insights

The success of this model highlights several key points. First, bypassing human intuition in feature design is not only possible but can lead to superior performance. The CNN learns relevant acoustic features from scratch that are as effective as those developed over decades of research.

Second, the authors highlight that max-pooling is essential for handling raw audio, as it allows the network to ignore minor temporal shifts in acoustic events. Finally, increasing the granularity of phoneme states (from 39 to 183) significantly improved accuracy on TIMIT, suggesting the CNN is capable of capturing very fine-grained temporal patterns in the raw signal.

Conclusion

This work demonstrates that end-to-end phoneme recognition from raw speech signals is a viable and competitive alternative to traditional feature-based methods. While the current model focuses on context-independent phonemes, future work will focus on extending the model to handle context-dependent phonemes (triphones) and exploring deeper CNN architectures to further improve feature representation.

Summary updated

Abstract

Most phoneme recognition state-of-the-art systems rely on a classical neural network classifiers, fed with highly tuned features, such as MFCC or PLP features. Recent advances in ``deep learning'' approaches questioned such systems, but while some attempts were made with simpler features such as spectrograms, state-of-the-art systems still rely on MFCCs. This might be viewed as a kind of failure from deep learning approaches, which are often claimed to have the ability to train with raw signals, alleviating the need of hand-crafted features. In this paper, we investigate a convolutional neural network approach for raw speech signals. While convolutional architectures got tremendous success in computer vision or text processing, they seem to have been let down in the past recent years in the speech processing field. We show that it is possible to learn an end-to-end phoneme sequence classifier system directly from raw signal, with similar performance on the TIMIT and WSJ datasets than existing systems based on MFCC, questioning the need of complex hand-crafted features on large datasets.
BibTeX
@misc{palaz2013end,
  author = {Dimitri Palaz and Ronan Collobert and Mathew Magimai-Doss},
  title = {{End-to-end Phoneme Sequence Recognition using Convolutional Neural Networks}},
  year = {2013},
  eprint = {1312.2137},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1312.2137},
}
Figure