Abstract
Advances in machine learning have made it possible to perform various text and speech processing tasks, such as automatic speech recognition (ASR), in an end-to-end (E2E) manner. E2E approaches utilizing pre-trained models are gaining attention for conserving training data and resources. However, most of their applications in ASR involve only one of either a pre-trained speech or a language model. This paper proposes integrating a pre-trained speech representation model and a large language model (LLM) for E2E ASR. The proposed model enables the optimization of the entire ASR process, including acoustic feature extraction and acoustic and language modeling, by combining pre-trained models with a bridge network and also enables the application of remarkable developments in LLM utilization, such as parameter-efficient domain adaptation and inference optimization. Experimental results demonstrate that the proposed model achieves a performance comparable to that of modern E2E ASR models by utilizing powerful pre-training models with the proposed integrated approach.
Introduction
Traditional Automatic Speech Recognition (ASR) systems have historically relied on a pipeline of separate components: an acoustic model to interpret audio signals and a language model to ensure linguistic coherence in the output. While modern End-to-End (E2E) models have simplified this process, they often still require "shallow fusion"—integrating an external language model during decoding—which increases computational complexity.
Furthermore, most existing research focuses on pre-training either the speech component (e.g., HuBERT) or the text component (e.g., GPT), but rarely both within a unified architecture. The primary challenge is the "modality gap": speech signals are high-frequency and continuous, whereas text is discrete and sparse. Bridging these two modalities without sacrificing the linguistic intelligence of the LLM is the core problem addressed in this research.
Proposed Architecture: Nue-ASR
The researchers developed Nue-ASR, an end-to-end model that integrates a pre-trained HuBERT speech encoder with a GPT-NeoX language model via a specialized "bridge network." By treating speech features as a continuous prompt for the LLM, the system achieves state-of-the-art performance and eliminates the need for complex external language model decoding.
The architecture consists of three main parts:
- Speech Encoder (HuBERT): This component extracts latent acoustic representations directly from raw waveforms.
- Bridge Network: This critical component compresses speech features (generated every 20ms) and maps them into the LLM’s embedding space.
- Large Language Model (GPT-NeoX): A 3.6-billion parameter decoder-only Transformer. It receives compressed speech features as a "prefix prompt" and generates the transcription token-by-token.
Bridge Network Strategies
The bridge network is essential for aligning the high-frequency speech representations with the token-based input expected by the LLM. The authors explored three distinct compression strategies to bridge the modality gap:
- Downsampling: Utilizes simple convolutional layers to reduce the sequence length.
- CTC Remove: Employs a Connectionist Temporal Classification (CTC) branch to identify and strip out "blank" frames, focusing only on informative segments.
- CTC Average: Averages adjacent frames that the CTC branch predicts as the same character, providing a more stable representation.
Experimental Setup and Datasets
The study focused on the Japanese language, utilizing several key corpora to evaluate performance across different domains:
- ReazonSpeech (v1.0): A massive 19,000-hour corpus from Japanese TV broadcasts.
- JSUT basic5000: High-quality single-speaker recordings.
- Common Voice 8.0 (Japanese): Crowdsourced, multi-speaker data.
- Corpus of Spontaneous Japanese (CSJ): Academic lecture recordings used to test robustness to real-world speech disfluencies.
Key Insights and Results
Experimental results demonstrate that Nue-ASR is highly effective:
- Competitive Accuracy: Nue-ASR achieved a Character Error Rate (CER) of 7.8% on the ReazonSpeech test set, outperforming Whisper-large-v3 (11.9% with greedy decoding).
- The LLM as an Acoustic Decoder: Even with a "frozen" GPT model, the bridge network learned to map speech features effectively. However, fine-tuning the LLM provided significant performance gains.
- CTC-Based Robustness: While downsampling works for short clips, CTC-based bridge networks were far more robust for longer utterances, effectively syncing speech timing to text rhythm.
- Domain Adaptation: Using Parameter-Efficient Fine-Tuning (PEFT) via LoRA, the model was adapted to spontaneous speech (CSJ) by updating only 22 million parameters, significantly reducing error rates on lecture data.
- Inference Efficiency: Utilizing DeepSpeed-Inference resulted in a 1.4x speedup, achieving a Real-Time Factor (RTF) of 0.15 on a standard T4 GPU.
Limitations and Future Work
Despite its success, the model has limitations. It requires significant paired speech-text data (over 15,000 hours) for effective modality alignment. Additionally, the decoder-only architecture can struggle with alignment in very long segments, sometimes losing the beginning or end of a paragraph. Future work will explore source-target attention mechanisms and scaling to larger, multilingual LLMs to create a more universal speech-to-text engine.
Summary updated