A Comparison of Transformer, Convolutional, and Recurrent Neural Networks on Phoneme Recognition

Kyuhong Shim, Wonyong SungarXiv 2022 · DOI · arXiv

Abstract

In speech processing, Automatic Speech Recognition (ASR) typically combines phonetic extraction with linguistic modeling. This paper isolates the phonetic component to compare how CNN, RNN, Transformer, and Conformer architectures perform on phoneme recognition. Using the LibriSpeech-960 dataset, the study demonstrates that the Conformer achieves the highest accuracy (90.3%) by leveraging both local and global context. However, CNN-based architectures like ContextNet maintain a significant advantage in computational efficiency, exhibiting linear scaling with sequence length compared to the quadratic complexity of attention-based models.

Introduction

While full ASR systems are the standard for turning audio into text, they often obscure the specific strengths and weaknesses of the underlying neural architectures in "hearing" sounds. Phoneme recognition—identifying the smallest units of sound like 'th' or 'sh'—provides a cleaner testbed. By stripping away the language model, researchers can evaluate how different architectures extract phonetic features from raw audio.

Experimental Setup

The study compares four major architectures under controlled constraints, typically limiting parameters to approximately 5 million and layers to 4:

  1. CNN (ContextNet): Utilizes depthwise separable convolutions and Squeeze-and-Excite (SE) modules for local and global pattern capture.
  2. RNN (Bi-LSTM): A traditional sequential processor that compresses history into a hidden state vector.
  3. Transformer: Employs self-attention, allowing every audio frame to interact with every other frame.
  4. Conformer: A hybrid architecture combining CNN-style local feature extraction with Transformer-style global attention.

Models were trained on LibriSpeech-960, with ground-truth phoneme labels generated via the Montreal Forced Aligner, resulting in 37 distinct phoneme classes.

Architecture Performance and Receptive Fields

A key finding of the study is the concept of "phonetic localization." Transformers and Conformers use self-attention to cluster similar-sounding phonemes across a sequence, regardless of their distance. This allows them to continue improving as the "receptive field" (the amount of context the model sees) increases.

In contrast, CNNs like ContextNet tend to hit a performance plateau. While they improve as their viewing window expands, their accuracy saturates after approximately 5 seconds of context. Despite this, the Conformer remains the top performer by integrating convolution modules directly into the Transformer block to capture local nuances.

Performance comparison across architectures with varying receptive fields.

As shown in the results, the Conformer achieved a peak accuracy of 90.3% on the test-clean set. The pure Transformer followed at 88.4%, while the traditional LSTM lagged significantly behind at 87.4%, suggesting that modern attention and convolution-based models have largely superseded recurrent architectures for this task.

Computational Efficiency

While the Conformer leads in accuracy, the CNN-based ContextNet is the champion of efficiency. At smaller scales (e.g., 1 million parameters), ContextNet often outperforms more complex Transformers. More importantly, its computational cost scales linearly with the length of the audio sequence.

Inference time vs. sequence length for different architectures.

The quadratic complexity ($O(T^2)$) of the Transformer's attention mechanism causes a sharp increase in inference time as sequences grow longer. For audio files exceeding 10 seconds, the CNN's linear scaling makes it significantly faster and more suitable for deployment on resource-constrained edge devices.

Conclusion and Future Work

The Conformer currently represents the state-of-the-art for phoneme recognition accuracy due to its balanced approach to local and global context. However, the "quadratic wall" of attention mechanisms remains a hurdle for scaling to very long audio files. Future research may focus on making Conformers more parameter-efficient or exploring "streaming" scenarios where models must make decisions in real-time before an entire utterance is complete.

Summary updated

Abstract

Phoneme recognition is a very important part of speech recognition that requires the ability to extract phonetic features from multiple frames. In this paper, we compare and analyze CNN, RNN, Transformer, and Conformer models using phoneme recognition. For CNN, the ContextNet model is used for the experiments. First, we compare the accuracy of various architectures under different constraints, such as the receptive field length, parameter size, and layer depth. Second, we interpret the performance difference of these models, especially when the observable sequence length varies. Our analyses show that Transformer and Conformer models benefit from the long-range accessibility of self-attention through input frames.
BibTeX
@misc{shim2022comparison,
  author = {Kyuhong Shim and Wonyong Sung},
  title = {{A Comparison of Transformer, Convolutional, and Recurrent Neural Networks on Phoneme Recognition}},
  year = {2022},
  eprint = {2210.00367},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2210.00367},
}
Figure