BERT Meets CTC: New Formulation of End-to-End Speech Recognition with Pre-trained Masked Language Model

Yosuke Higuchi, Brian Yan, Siddhant Arora, Tetsuji Ogawa, Tetsunori Kobayashi, Shinji Watanabe

Abstract

Traditional Connectionist Temporal Classification (CTC) models for Automatic Speech Recognition (ASR) are efficient but limited by a conditional independence assumption that hinders their ability to capture linguistic context. We present BERT-CTC, a novel framework that integrates a pre-trained, frozen BERT model into the CTC pipeline. By leveraging self-attention to fuse acoustic features with BERT’s contextual embeddings, BERT-CTC relaxes the independence constraint. Our approach achieves significant accuracy improvements over standard CTC and RNN-Transducer (RNN-T) models across multiple benchmarks while maintaining competitive inference speeds through an iterative mask-predict algorithm.

Introduction

Automatic Speech Recognition (ASR) involves mapping high-frequency acoustic signals to discrete linguistic symbols. A major challenge in this domain is the "input-output gap," where long-range linguistic dependencies must be captured from continuous audio.

Connectionist Temporal Classification (CTC) models are widely used due to their simplicity and non-autoregressive nature, which allows for fast inference. However, they rely on a "conditional independence assumption," assuming each output token is independent of others given the audio. This limitation prevents CTC models from effectively learning complex language patterns, such as homophones or grammar. While autoregressive models like RNN-Transducers (RNN-T) address this by predicting tokens sequentially, they are computationally expensive and slower during inference.

BERT-CTC aims to bridge this gap by imbuing the CTC framework with the linguistic capabilities of a pre-trained language model like BERT, without sacrificing the efficiency inherent to non-autoregressive decoding.

Methodology

The BERT-CTC architecture consists of three primary components: a Conformer-based audio encoder, a frozen pre-trained BERT model, and a self-attention fusion module.

Architectural comparison of CTC, RNN-T, and the proposed BERT-CTC model.

Model Components

  1. Audio Encoder: Extracts acoustic features from the speech signal $O$.
  2. Frozen BERT: Acts as a feature extractor for hypothesized text sequences $\bar{W}$. Keeping BERT frozen preserves its sophisticated linguistic knowledge while simplifying training.
  3. Self-Attention Fusion: This module attends to both audio features and BERT embeddings, learning the inter-dependencies between sound and text to produce the final probability $p(a_t | \text{BERT}(\bar{W}), O)$.

Training and Inference

During training, a masking strategy similar to BERT’s original pre-training is employed. Ground-truth transcripts are partially masked and fed into BERT. The fusion layer then combines these linguistic hints with audio features to predict the full transcript via CTC loss. To bridge the gap between acoustic features and BERT’s large vocabulary, a hierarchical CTC loss is used, training an intermediate layer of the audio encoder to predict a simplified subword vocabulary.

Inference follows an iterative mask-predict algorithm. An initial hypothesis is generated using only the audio encoder. Then, for $K$ iterations (typically 20), low-confidence tokens are replaced with [MASK], and the sequence is refined by the fusion layer using BERT context and audio features.

Experimental Evaluation

BERT-CTC was evaluated on standard benchmarks including LibriSpeech (100h), TED-LIUM2, and AISHELL-1.

Performance comparison of BERT-CTC against CTC and RNN-T baselines across different datasets.

As shown in the results, BERT-CTC consistently outperforms both CTC and RNN-T baselines. On the LibriSpeech-100h "other" test set, BERT-CTC achieved a 16.6% Word Error Rate (WER), a substantial improvement over the 22.0% WER of the CTC baseline and the 22.2% of the RNN-T. In Mandarin (AISHELL-1), it reached a 3.9% Character Error Rate (CER), matching state-of-the-art performance with fewer trainable parameters.

Furthermore, BERT-CTC demonstrates superior inference speed compared to RNN-T. On a GPU, it achieved a Real-Time Factor (RTF) of 0.097, significantly faster than the 0.481 RTF of RNN-T, despite the iterative decoding process.

Discussion and Conclusion

BERT-CTC successfully relaxes the conditional independence assumption of CTC by explicitly conditioning predictions on linguistic context. This allows the model to correct phonetically plausible but contextually incorrect errors (e.g., changing "a gave" to "again"). While the model is currently constrained by BERT's vocabulary and is not yet optimized for real-time streaming, it represents a significant step forward in combining the strengths of pre-trained language models with efficient non-autoregressive ASR frameworks.

Summary updated

BibTeX
@inproceedings{higuchi2022bert,
  author = {Yosuke Higuchi and Brian Yan and Siddhant Arora and Tetsuji Ogawa and Tetsunori Kobayashi and Shinji Watanabe},
  title = {{BERT Meets CTC: New Formulation of End-to-End Speech Recognition with Pre-trained Masked Language Model}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2022},
  doi = {10.18653/v1/2022.findings-emnlp.402},
}
Figure