Integrating Pre-Trained Speech and Language Models for End-to-End Speech Recognition

Yukiya Hono, Koh Mitsuda, Tianyu Zhao, Kentaro Mitsui, Toshiaki Wakatsuki, Kei SawadaACL 2024 · OpenReview

Abstract

Advances in machine learning have made it possible to perform various text and speech processing tasks, such as automatic speech recognition (ASR), in an end-to-end (E2E) manner. E2E approaches utilizing pre-trained models are gaining attention for conserving training data and resources. However, most of their applications in ASR involve only one of either a pre-trained speech or a language model. This paper proposes integrating a pre-trained speech representation model and a large language model (LLM) for E2E ASR. The proposed model enables the optimization of the entire ASR process, including acoustic feature extraction and acoustic and language modeling, by combining pre-trained models with a bridge network and also enables the application of remarkable developments in LLM utilization, such as parameter-efficient domain adaptation and inference optimization. Experimental results demonstrate that the proposed model achieves a performance comparable to that of modern E2E ASR models by utilizing powerful pre-training models with the proposed integrated approach.

Introduction

Traditional Automatic Speech Recognition (ASR) systems have historically relied on a pipeline of separate components: an acoustic model to interpret audio signals and a language model to ensure linguistic coherence in the output. While modern End-to-End (E2E) models have simplified this process, they often still require "shallow fusion"—integrating an external language model during decoding—which increases computational complexity.

Furthermore, most existing research focuses on pre-training either the speech component (e.g., HuBERT) or the text component (e.g., GPT), but rarely both within a unified architecture. The primary challenge is the "modality gap": speech signals are high-frequency and continuous, whereas text is discrete and sparse. Bridging these two modalities without sacrificing the linguistic intelligence of the LLM is the core problem addressed in this research.

Proposed Architecture: Nue-ASR

The researchers developed Nue-ASR, an end-to-end model that integrates a pre-trained HuBERT speech encoder with a GPT-NeoX language model via a specialized "bridge network." By treating speech features as a continuous prompt for the LLM, the system achieves state-of-the-art performance and eliminates the need for complex external language model decoding.

Figure 1: Overall architecture of the proposed Nue-ASR model, integrating HuBERT and GPT-NeoX via a bridge network.

The architecture consists of three main parts:

  1. Speech Encoder (HuBERT): This component extracts latent acoustic representations directly from raw waveforms.
  2. Bridge Network: This critical component compresses speech features (generated every 20ms) and maps them into the LLM’s embedding space.
  3. Large Language Model (GPT-NeoX): A 3.6-billion parameter decoder-only Transformer. It receives compressed speech features as a "prefix prompt" and generates the transcription token-by-token.

Bridge Network Strategies

The bridge network is essential for aligning the high-frequency speech representations with the token-based input expected by the LLM. The authors explored three distinct compression strategies to bridge the modality gap:

Figure 5: Comparison of the three bridge network strategies: (a) Downsampling, (b) CTC remove, and (c) CTC average.
  • Downsampling: Utilizes simple convolutional layers to reduce the sequence length.
  • CTC Remove: Employs a Connectionist Temporal Classification (CTC) branch to identify and strip out "blank" frames, focusing only on informative segments.
  • CTC Average: Averages adjacent frames that the CTC branch predicts as the same character, providing a more stable representation.

Experimental Setup and Datasets

The study focused on the Japanese language, utilizing several key corpora to evaluate performance across different domains:

  • ReazonSpeech (v1.0): A massive 19,000-hour corpus from Japanese TV broadcasts.
  • JSUT basic5000: High-quality single-speaker recordings.
  • Common Voice 8.0 (Japanese): Crowdsourced, multi-speaker data.
  • Corpus of Spontaneous Japanese (CSJ): Academic lecture recordings used to test robustness to real-world speech disfluencies.

Key Insights and Results

Experimental results demonstrate that Nue-ASR is highly effective:

  • Competitive Accuracy: Nue-ASR achieved a Character Error Rate (CER) of 7.8% on the ReazonSpeech test set, outperforming Whisper-large-v3 (11.9% with greedy decoding).
  • The LLM as an Acoustic Decoder: Even with a "frozen" GPT model, the bridge network learned to map speech features effectively. However, fine-tuning the LLM provided significant performance gains.
  • CTC-Based Robustness: While downsampling works for short clips, CTC-based bridge networks were far more robust for longer utterances, effectively syncing speech timing to text rhythm.
  • Domain Adaptation: Using Parameter-Efficient Fine-Tuning (PEFT) via LoRA, the model was adapted to spontaneous speech (CSJ) by updating only 22 million parameters, significantly reducing error rates on lecture data.
  • Inference Efficiency: Utilizing DeepSpeed-Inference resulted in a 1.4x speedup, achieving a Real-Time Factor (RTF) of 0.15 on a standard T4 GPU.

Limitations and Future Work

Despite its success, the model has limitations. It requires significant paired speech-text data (over 15,000 hours) for effective modality alignment. Additionally, the decoder-only architecture can struggle with alignment in very long segments, sometimes losing the beginning or end of a paragraph. Future work will explore source-target attention mechanisms and scaling to larger, multilingual LLMs to create a more universal speech-to-text engine.

Summary updated

Abstract

Advances in machine learning have made it possible to perform various text and speech processing tasks, such as automatic speech recognition (ASR), in an end-to-end (E2E) manner. E2E approaches utilizing pre-trained models are gaining attention for conserving training data and resources. However, most of their applications in ASR involve only one of either a pre-trained speech or a language model. This paper proposes integrating a pre-trained speech representation model and a large language model (LLM) for E2E ASR. The proposed model enables the optimization of the entire ASR process, including acoustic feature extraction and acoustic and language modeling, by combining pre-trained models with a bridge network and also enables the application of remarkable developments in LLM utilization, such as parameter-efficient domain adaptation and inference optimization. Experimental results demonstrate that the proposed model achieves a performance comparable to that of modern E2E ASR models by utilizing powerful pre-training models with the proposed integrated approach.
BibTeX
@inproceedings{hono2024integrating,
  author = {Yukiya Hono and Koh Mitsuda and Tianyu Zhao and Kentaro Mitsui and Toshiaki Wakatsuki and Kei Sawada},
  title = {{Integrating Pre-Trained Speech and Language Models for End-to-End Speech Recognition}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2024},
  url = {https://openreview.net/forum?id=zCqSi9S1lw},
}
Figure