Improving Hybrid CTC/Attention End-to-end Speech Recognition with Pretrained Acoustic and Language Model

Keqi Deng, Songjun Cao, Yike Zhang, Long Ma

TL;DR

The "Preformer" is a novel speech recognition architecture that successfully integrates both pretrained acoustic models (wav2vec 2.0) and language models (DistilGPT2) into a single system. By redesigning the decoder to use only a single cross-attention layer, the authors achieved a 4.6% Character Error Rate (CER) on the AISHELL-1 benchmark—a 27% relative improvement over standard Transformer baselines.

What Problem Does This Paper Address?

End-to-end (E2E) Automatic Speech Recognition (ASR) has revolutionized the field by replacing complex pipelines with single neural networks. However, these models are data-hungry. While self-supervised pretraining (like wav2vec 2.0 for audio or GPT for text) has helped, it’s surprisingly difficult to use both at once in dominant sequence-to-sequence (S2S) models.

The bottleneck is the decoder. In a standard Transformer ASR model, every layer of the decoder looks at the acoustic features from the encoder (via cross-attention). This creates a "dependency trap": the decoder is so tightly coupled with acoustic representations that you can't easily initialize it with a text-only pretrained language model (LM) like GPT2. There is a fundamental architecture mismatch between a text generator and a speech-conditioned decoder.

Methods

The authors introduce the Preformer, a hybrid CTC/Attention architecture designed to play nice with pretrained models.

  1. The Encoder (w2v-encoder): The encoder is initialized with a pretrained wav2vec 2.0 model. To ensure it converges well, the authors use a Connectionist Temporal Classification (CTC) loss as an auxiliary task. This helps the model learn the alignment between speech and text more effectively during fine-tuning.
  2. The One-Cross Decoder (OCD): This is the paper's most significant innovation. In a vanilla Transformer, the decoder has $M$ layers, each containing self-attention and cross-attention. The OCD changes this to $N$ "self-layers" (only self-attention) followed by exactly one "cross-layer" (cross-attention).
  3. Initialization: Because the first $N$ layers of the OCD only deal with text, they can be initialized directly with a pretrained DistilGPT2 model. The final cross-attention layer is the only part that bridges the gap between text and audio.
  4. Joint Decoding: During inference, the model combines scores from the CTC branch and the OCD to produce the final transcript, leveraging the strengths of both alignment-based and sequence-based recognition.

Datasets

  • AISHELL-1: A popular Mandarin speech corpus (178 hours) used for training and evaluation.
  • AISHELL-2: Used (without transcriptions) to pretrain a Mandarin-specific wav2vec 2.0 base model.
  • External Text Corpora: Used for the pretraining of the DistilGPT2 model (specifically the uer/gpt2-distil-chinese-cluecorpussmall version).

Key Insights and Ideas

  • "One Cross is Enough": The authors argue that if your text representation is high-quality (thanks to a pretrained LM), you don't need to peek at the acoustic features in every single decoder layer. One well-placed cross-attention layer at the end is sufficient to disambiguate the text based on the audio.
  • Overcoming Mismatch: By isolating the text-processing layers from the audio-conditioning layers, the OCD eliminates the architectural mismatch that previously prevented GPT-style models from being used as ASR decoders.
  • Freezing for Success: The researchers found that fixing (freezing) the pretrained LM parameters during fine-tuning yielded the best results. They suggest that fine-tuning these on a relatively small labeled ASR dataset can actually "damage" the rich linguistic knowledge the LM acquired during its massive pretraining phase.
  • Cross-Lingual Benefits: Interestingly, the paper shows that even using an English pretrained wav2vec 2.0 model to initialize a Mandarin ASR system provides a significant boost over training from scratch, though a native Mandarin model is still superior.

Key Results

The Preformer sets a high bar on the AISHELL-1 dataset:

  • 4.6% CER on the test set, outperforming the vanilla Transformer baseline (6.3% CER).
  • 27% Relative Reduction in error rate compared to the baseline.
  • Efficiency: The OCD actually uses fewer parameters (about 2.2M less) than a standard Transformer decoder because it removes multiple cross-attention mechanisms, yet it performs significantly better.
  • Competitive Performance: The Preformer even outperformed the popular Conformer architecture when the latter didn't use advanced data augmentation like SpecAugment.

Limitations and Future Work

  • Language Specificity: While the results on Mandarin are impressive, the "one-cross" assumption needs to be tested across more diverse languages and much larger datasets (e.g., 10,000+ hours) to see if the architecture holds up.
  • Parameter Freezing: The authors kept the DistilGPT2 weights fixed. Future work could explore "adapter" modules or specific learning rate schedules to allow the LM to adapt slightly to the "speech-like" text of ASR transcripts without losing its general knowledge.
  • Comparison to Fusion: The paper compares OCD to "shallow fusion" (simply adding an LM score at the end), but more comparisons against "cold fusion" or other deep integration methods would provide more context on OCD's relative strengths.

Summary updated

Abstract

Recently, self-supervised pretraining has achieved impressive results in end-to-end (E2E) automatic speech recognition (ASR). However, the dominant sequence-to-sequence (S2S) E2E model is still hard to fully utilize the self-supervised pre-training methods because its decoder is conditioned on acoustic representation thus cannot be pretrained separately. In this paper, we propose a pretrained Transformer (Preformer) S2S ASR architecture based on hybrid CTC/attention E2E models to fully utilize the pretrained acoustic models (AMs) and language models (LMs). In our framework, the encoder is initialized with a pretrained AM (wav2vec2.0). The Preformer leverages CTC as an auxiliary task during training and inference. Furthermore, we design a one-cross decoder (OCD), which relaxes the dependence on acoustic representations so that it can be initialized with pretrained LM (DistilGPT2). Experiments are conducted on the AISHELL-1 corpus and achieve a $4.6\%$ character error rate (CER) on the test set. Compared with our vanilla hybrid CTC/attention Transformer baseline, our proposed CTC/attention-based Preformer yields $27\%$ relative CER reduction. To the best of our knowledge, this is the first work to utilize both pretrained AM and LM in a S2S ASR system.
BibTeX
@inproceedings{deng2021improving,
  author = {Keqi Deng and Songjun Cao and Yike Zhang and Long Ma},
  title = {{Improving Hybrid CTC/Attention End-to-end Speech Recognition with Pretrained Acoustic and Language Model}},
  booktitle = {ASRU},
  year = {2021},
  url = {https://openreview.net/forum?id=FEHpq2gcAw},
}
Figure