Abstract
Most phoneme recognition state-of-the-art systems rely on classical neural network classifiers fed with highly tuned features, such as Mel Frequency Cepstrum Coefficients (MFCC). Recent advances in deep learning have questioned these systems, yet many state-of-the-art models still rely on hand-crafted features. This paper investigates a convolutional neural network (CNN) approach for raw speech signals. We demonstrate that it is possible to learn an end-to-end phoneme sequence classifier directly from raw signals, achieving performance on the TIMIT and WSJ datasets comparable to existing systems based on MFCCs, questioning the need for complex hand-crafted features on large datasets.
Introduction
For decades, Automatic Speech Recognition (ASR) has followed a "divide and conquer" pipeline. Raw audio is typically converted into hand-crafted features like MFCCs or Perceptual Linear Prediction (PLP) features, based on human knowledge of acoustics and psychoacoustics. These features are then processed by a classifier (e.g., an ANN) and a decoder (e.g., an HMM). This reliance on manual feature engineering represents a potential bottleneck in the development of truly autonomous systems.
While deep learning has revolutionized computer vision by learning representations directly from pixels, the speech community has historically relied on pre-processed features. This research asks a fundamental question: Can we build a truly end-to-end system that learns its own features directly from the raw speech signal?
Proposed Architecture
The proposed system is a unified, end-to-end architecture composed of a Convolutional Neural Network (CNN) and a Conditional Random Field (CRF) decoder.
The system processes a sequence of raw speech signal frames. The CNN handles both feature learning and modeling, while the CRF performs the final decoding into a phoneme sequence. Crucially, the entire pipeline is trained using backpropagation to maximize the log-likelihood of the correct label sequence.
Convolutional Neural Network Design
The CNN is divided into two distinct stages to replace the traditional feature extraction and classification pipeline:
- Filter Stage (Feature Learning): This stage consists of several convolutional layers that apply filters over the temporal signal. These are followed by max-pooling layers, which provide robustness against slight temporal distortions (temporal invariance), and
tanhnon-linearities. - Classification Stage (Modeling): The learned features are fed into linear layers that produce scores for each phoneme class.
Experimental Evaluation
The approach was evaluated on two standard benchmarks:
- TIMIT Corpus: A classic acoustic-phonetic dataset. The researchers tested various granularities, including a standard 39-phoneme set and a more complex 183-state set (3 states per phoneme).
- Wall Street Journal (WSJ): A larger corpus (SI-284 set, ~80 hours) used to test the scalability of the raw-signal approach to larger datasets.
Key Results
The raw-signal CNN approach proved highly competitive with traditional MFCC-based baselines:
- TIMIT: The CNN+CRF system using 183 states achieved a test accuracy of 70.08%, significantly outperforming the MFCC-based baseline of 66.65%.
- WSJ: On the larger dataset, the raw-signal system achieved 72.88% accuracy, compared to the baseline's 72.39%. This confirmed that the approach scales effectively to large-scale ASR tasks.
Discussion and Insights
The success of this model highlights several key points. First, bypassing human intuition in feature design is not only possible but can lead to superior performance. The CNN learns relevant acoustic features from scratch that are as effective as those developed over decades of research.
Second, the authors highlight that max-pooling is essential for handling raw audio, as it allows the network to ignore minor temporal shifts in acoustic events. Finally, increasing the granularity of phoneme states (from 39 to 183) significantly improved accuracy on TIMIT, suggesting the CNN is capable of capturing very fine-grained temporal patterns in the raw signal.
Conclusion
This work demonstrates that end-to-end phoneme recognition from raw speech signals is a viable and competitive alternative to traditional feature-based methods. While the current model focuses on context-independent phonemes, future work will focus on extending the model to handle context-dependent phonemes (triphones) and exploring deeper CNN architectures to further improve feature representation.
Summary updated