Abstract
In speech processing, Automatic Speech Recognition (ASR) typically combines phonetic extraction with linguistic modeling. This paper isolates the phonetic component to compare how CNN, RNN, Transformer, and Conformer architectures perform on phoneme recognition. Using the LibriSpeech-960 dataset, the study demonstrates that the Conformer achieves the highest accuracy (90.3%) by leveraging both local and global context. However, CNN-based architectures like ContextNet maintain a significant advantage in computational efficiency, exhibiting linear scaling with sequence length compared to the quadratic complexity of attention-based models.
Introduction
While full ASR systems are the standard for turning audio into text, they often obscure the specific strengths and weaknesses of the underlying neural architectures in "hearing" sounds. Phoneme recognition—identifying the smallest units of sound like 'th' or 'sh'—provides a cleaner testbed. By stripping away the language model, researchers can evaluate how different architectures extract phonetic features from raw audio.
Experimental Setup
The study compares four major architectures under controlled constraints, typically limiting parameters to approximately 5 million and layers to 4:
- CNN (ContextNet): Utilizes depthwise separable convolutions and Squeeze-and-Excite (SE) modules for local and global pattern capture.
- RNN (Bi-LSTM): A traditional sequential processor that compresses history into a hidden state vector.
- Transformer: Employs self-attention, allowing every audio frame to interact with every other frame.
- Conformer: A hybrid architecture combining CNN-style local feature extraction with Transformer-style global attention.
Models were trained on LibriSpeech-960, with ground-truth phoneme labels generated via the Montreal Forced Aligner, resulting in 37 distinct phoneme classes.
Architecture Performance and Receptive Fields
A key finding of the study is the concept of "phonetic localization." Transformers and Conformers use self-attention to cluster similar-sounding phonemes across a sequence, regardless of their distance. This allows them to continue improving as the "receptive field" (the amount of context the model sees) increases.
In contrast, CNNs like ContextNet tend to hit a performance plateau. While they improve as their viewing window expands, their accuracy saturates after approximately 5 seconds of context. Despite this, the Conformer remains the top performer by integrating convolution modules directly into the Transformer block to capture local nuances.
As shown in the results, the Conformer achieved a peak accuracy of 90.3% on the test-clean set. The pure Transformer followed at 88.4%, while the traditional LSTM lagged significantly behind at 87.4%, suggesting that modern attention and convolution-based models have largely superseded recurrent architectures for this task.
Computational Efficiency
While the Conformer leads in accuracy, the CNN-based ContextNet is the champion of efficiency. At smaller scales (e.g., 1 million parameters), ContextNet often outperforms more complex Transformers. More importantly, its computational cost scales linearly with the length of the audio sequence.
The quadratic complexity ($O(T^2)$) of the Transformer's attention mechanism causes a sharp increase in inference time as sequences grow longer. For audio files exceeding 10 seconds, the CNN's linear scaling makes it significantly faster and more suitable for deployment on resource-constrained edge devices.
Conclusion and Future Work
The Conformer currently represents the state-of-the-art for phoneme recognition accuracy due to its balanced approach to local and global context. However, the "quadratic wall" of attention mechanisms remains a hurdle for scaling to very long audio files. Future research may focus on making Conformers more parameter-efficient or exploring "streaming" scenarios where models must make decisions in real-time before an entire utterance is complete.
Summary updated