A Study of All-Convolutional Encoders for Connectionist Temporal Classification

Kalpesh Krishna, Liang Lu, Kevin Gimpel, Karen Livescu

Abstract

Connectionist temporal classification (CTC) is a popular sequence prediction approach for automatic speech recognition that is typically used with models based on recurrent neural networks (RNNs). We explore whether deep convolutional neural networks (CNNs) can be used effectively instead of RNNs as the "encoder" in CTC. CNNs lack an explicit representation of the entire sequence, but have the advantage that they are much faster to train. We present an exploration of CNNs as encoders for CTC models, in the context of character-based (lexicon-free) automatic speech recognition. In particular, we explore a range of one-dimensional convolutional layers, which are particularly efficient. We compare the performance of our CNN-based models against typical RNN-based models in terms of training time, decoding time, model size and word error rate (WER) on the Switchboard Eval2000 corpus. We find that our CNN-based models are close in performance to LSTMs, while not matching them, and are much faster to train and decode.

Introduction

Automatic Speech Recognition (ASR) has long been dominated by Recurrent Neural Networks (RNNs), specifically LSTMs, because of their innate ability to model long-range temporal dependencies in audio sequences. However, RNNs are inherently sequential, which limits parallelization on modern hardware like GPUs.

Connectionist Temporal Classification (CTC) is a popular framework that allows training these models without requiring pre-segmented audio-to-text alignments. This paper investigates whether the recurrence can be removed entirely by using an "all-convolutional" encoder. By leveraging the massive parallelization of 1D CNNs, the authors aim to significantly speed up the ASR pipeline without sacrificing excessive accuracy.

Methodology

The proposed architecture utilizes a 1D CNN that convolves strictly across the time dimension. The input consists of 40 log mel frequency filterbank features plus their first-order derivatives, resulting in 80 features per frame.

Architecture of the proposed 1D CNN encoder.

As illustrated in Figure 1, the model begins with a 1D convolution followed by a 2x1 max-pooling layer to reduce temporal resolution. This is followed by a series of "ResBlocks" (Residual Blocks) containing batch normalization, ReLU nonlinearities, and residual connections. The network concludes with two 512-unit fully connected layers and a final projection layer to the character set. Decoding is evaluated using both simple greedy decoding and beam-search decoding integrated with character-level language models (LMs).

Experimental Results

The models were trained on the Switchboard corpus (~300 hours) and evaluated on the HUB5 Eval2000 dataset.

Accuracy vs. Efficiency

A key finding was that smaller filters in deeper networks (e.g., a filter size of 5 with 28 residual blocks) performed better and overfitted less than shallower networks with larger filters. The best CNN model achieved a 21.3% WER on the Switchboard subset with a 9-gram LM, which is competitive with the 19.7% achieved by the LSTM baseline.

Validation Word Error Rate (WER) vs. Wall-Clock (WC) training hours.

While LSTMs remain the accuracy leaders, CNNs offer a significant advantage in training speed. As shown in Figure 3, the CNN-based models (solid blue line) converge much faster in wall-clock time compared to LSTM variants. On a per-epoch basis, CNNs were roughly 3x faster to train. The most dramatic result was in decoding speed: for real-time applications (batch size 1), the CNN was nearly 15 times faster than the LSTM, decoding the entire Eval2000 set in 115 seconds compared to the LSTM's 1813 seconds.

Conclusion

All-convolutional encoders represent a highly attractive option for large-scale or resource-constrained ASR applications. Although they still lag slightly behind LSTMs in raw Word Error Rate, their superior training and decoding efficiency allows for faster iteration and deployment. Future work will investigate word-level CTC models and improved regularization techniques to further close the performance gap with recurrent architectures.

Summary updated

Abstract

Connectionist temporal classification (CTC) is a popular sequence prediction approach for automatic speech recognition that is typically used with models based on recurrent neural networks (RNNs). We explore whether deep convolutional neural networks (CNNs) can be used effectively instead of RNNs as the "encoder" in CTC. CNNs lack an explicit representation of the entire sequence, but have the advantage that they are much faster to train. We present an exploration of CNNs as encoders for CTC models, in the context of character-based (lexicon-free) automatic speech recognition. In particular, we explore a range of one-dimensional convolutional layers, which are particularly efficient. We compare the performance of our CNN-based models against typical RNNbased models in terms of training time, decoding time, model size and word error rate (WER) on the Switchboard Eval2000 corpus. We find that our CNN-based models are close in performance to LSTMs, while not matching them, and are much faster to train and decode.
BibTeX
@inproceedings{krishna2018study,
  author = {Kalpesh Krishna and Liang Lu and Kevin Gimpel and Karen Livescu},
  title = {{A Study of All-Convolutional Encoders for Connectionist Temporal Classification}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2018},
  url = {https://openreview.net/forum?id=qzyjn_ogWi},
}
Figure