BERT Meets CTC: New Formulation of End-to-End Speech Recognition with Pre-trained Masked Language Model

Yosuke Higuchi, Brian Yan, Siddhant Arora, Tetsuji Ogawa, Tetsunori Kobayashi, Shinji Watanabe

TL;DR

BERT-CTC is a novel speech recognition framework that injects the deep linguistic knowledge of pre-trained BERT models into the efficient Connectionist Temporal Classification (CTC) pipeline. By using an iterative "mask-predict" strategy, it breaks CTC’s traditional limitation—the assumption that output tokens are independent—resulting in significant accuracy gains across multiple languages and tasks while remaining faster than standard autoregressive models.

What Problem Does This Paper Address?

Automatic Speech Recognition (ASR) systems generally fall into two camps: Connectionist Temporal Classification (CTC) and autoregressive models (like RNN-Transducers).

CTC is highly efficient because it predicts all output tokens simultaneously. However, it suffers from a "conditional independence assumption," meaning it assumes the probability of one word doesn't depend on the words around it. This makes it struggle with context-heavy elements like homophones. Autoregressive models solve this by predicting tokens one by one based on previous ones, but they are slower and more complex to train.

The authors aim to bridge this gap. They want the efficiency and simplicity of CTC but with the rich, bidirectional linguistic context provided by a Masked Language Model (MLM) like BERT.

Methods

The authors introduce BERT-CTC, which uses a pre-trained, frozen BERT model as a sophisticated feature extractor for text context.

  1. Architecture: The system consists of a Conformer-based audio encoder and a frozen BERT model. A self-attention module then fuses the acoustic features from the audio with the contextual embeddings from BERT.
  2. Training with Masking: During training, the model is fed the audio and a partially masked version of the ground-truth transcript. It learns to "fill in the blanks" using both the sound and the surrounding text context.
  3. Iterative Inference: At test time, the model doesn't know the transcript. It uses a mask-predict algorithm:
  4. Iteration 1: It predicts a rough transcript based only on audio (treating all text as "masked").
  5. Subsequent Iterations: It identifies tokens it is least confident about, masks them, and re-predicts them using the audio plus the context of the "confident" tokens from the previous step.

This architecture is compared against standard CTC and RNN-T in Figure 1: Comparisons between different model architectures for end-to-end ASR.

Figure 1: Comparisons between different model architectures for end-to-end ASR.

Datasets

The model was evaluated on several standard benchmarks:

  • LibriSpeech (100h): Read English audiobooks.
  • TED-LIUM2: Spontaneous English (TED talks).
  • AISHELL-1: Read Mandarin Chinese.
  • SLURP: A Spoken Language Understanding (SLU) dataset used to test intent classification (e.g., "What is the weather?").

Key Insights and Ideas

  • Relaxing Independence: By conditioning the CTC output on BERT embeddings, the model explicitly learns inter-dependencies between tokens. It essentially "looks ahead" and "looks back" at the hypothesized transcript to refine its guesses.
  • Frozen BERT is Enough: Surprisingly, the authors found that they didn't need to fine-tune BERT. Keeping BERT frozen and only training the fusion layers was sufficient to capture linguistic nuances, which significantly reduces the number of trainable parameters.
  • Hierarchical CTC is the Secret Sauce: BERT uses a massive vocabulary (30k+ tokens), which is often too sparse for an audio encoder to learn directly. The authors used a "hierarchical loss," where the audio encoder also tries to predict a much smaller subword vocabulary (300-500 tokens). This auxiliary task helps the audio encoder converge.
  • The Power of Iteration: As shown in Figure 2, the model's performance improves dramatically over the first 10-15 iterations of refinement before plateauing. BERT-CTC results on development sets across iterations.
Figure 2: BERT-CTC results on development sets, us- ing different number of decoding iterations.

Key Results

BERT-CTC consistently outperformed both CTC and RNN-T baselines.

  • Accuracy: On LibriSpeech-100h (test-other), BERT-CTC achieved a 16.6% Word Error Rate (WER), a massive improvement over CTC (22.0%) and RNN-T (22.2%).
  • Multilingual Success: It achieved a 3.9% Character Error Rate (CER) on AISHELL-1, matching state-of-the-art performance with significantly fewer trainable parameters.
  • Downstream Tasks: In intent classification (SLURP), BERT-CTC reached 87.8% accuracy, surpassing previous state-of-the-art results.

Detailed comparisons can be found in the results table: ASR Results Table

Table 1: WER [%] on LibriSpeech-100h and TED-LIUM2, and CER [%] on AISHELL-1. † indicates that the models are slightly different from the original CTC or RNN-T in that they are trained with hierarchical CTC loss.

Limitations and Future Work

  • Vocabulary Constraint: BERT-CTC is tied to the specific vocabulary of the pre-trained BERT model. If the ASR task involves many "out-of-vocabulary" words (like technical jargon), performance might suffer.
  • Computational Overhead: While faster than autoregressive models on GPUs due to parallelization, running 20 iterations of a large BERT model is still computationally expensive compared to a single-pass CTC model.
  • Non-Streaming: The current version requires the "full context," meaning it can't easily be used for live, real-time captioning without modifications like causal masking or a two-pass approach.

Summary updated

Abstract

This paper presents BERT-CTC, a novel formulation of end-to-end speech recognition that adapts BERT for connectionist temporal classification (CTC). Our formulation relaxes the conditional independence assumptions used in conventional CTC and incorporates linguistic knowledge through the explicit output dependency obtained by BERT contextual embedding. BERT-CTC attends to the full contexts of the input and hypothesized output sequences via the self-attention mechanism. This mechanism encourages a model to learn inner/inter-dependencies between the audio and token representations while maintaining CTC's training efficiency. During inference, BERT-CTC combines a mask-predict algorithm with CTC decoding, which iteratively refines an output sequence. The experimental results reveal that BERT-CTC improves over conventional approaches across variations in speaking styles and languages. Finally, we show that the semantic representations in BERT-CTC are beneficial towards downstream spoken language understanding tasks.
BibTeX
@inproceedings{higuchi2022bert,
  author = {Yosuke Higuchi and Brian Yan and Siddhant Arora and Tetsuji Ogawa and Tetsunori Kobayashi and Shinji Watanabe},
  title = {{BERT Meets CTC: New Formulation of End-to-End Speech Recognition with Pre-trained Masked Language Model}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2022},
  doi = {10.18653/v1/2022.findings-emnlp.402},
}
Figure