Introduction
Large Language Models (LLMs) have significantly advanced Automatic Speech Recognition (ASR) by integrating contextual reasoning and linguistic "common sense." However, the standard approach—integrating a speech encoder with a massive LLM—is computationally expensive. This paper introduces EFIN (Encoder First INtegration), a training strategy designed to optimize ASR performance under a fixed computational budget. EFIN achieves a 21.1% relative reduction in error rates while using 50% less computational power compared to traditional joint-training methods.
The EFIN Framework
The core challenge in LLM-based ASR is the high cost of optimizing billions of parameters. The researchers propose that the "intelligence" of the speech encoder is often undervalued. They introduce a three-stage strategy to prioritize encoder optimization before full system integration.
Three-Stage Training Strategy
- Independent Encoder Fine-tuning: The speech encoder is fine-tuned independently using its original objective (e.g., Whisper's training task). This is computationally efficient as the encoder is much smaller than the LLM.
- Preliminary Alignment: The encoder and LLM are frozen while a small projection layer is trained. The authors found that "preliminary convergence" (stopping early) is sufficient, saving significant compute.
- LLM Adaptation: The LLM and projection layer undergo joint training using LoRA (Low-Rank Adaptation) to minimize parameter updates and costs.
Scaling Laws and Performance
To quantify efficiency, the authors derived a Neural Scaling Law for LLM-ASR, modeling the relationship between Character Error Rate (CER) and total floating-point operations (FLOPs) as a power-law equation: $L = \beta C^{-\alpha}$.
As shown in the scaling curves, EFIN (Strategy-5) is the most efficient path. It outperformed the strongest baseline (Strategy-3) with half the FLOPs. Interestingly, jointly training all components from scratch (Strategy-6) led to degraded performance, highlighting the difficulty of simultaneous multi-modal optimization.
Key Insights
- Encoder-First Advantage: Investing compute in a high-quality, independently trained encoder provides better features for the LLM, leading to faster and better overall convergence. It is more cost-effective to spend compute on a small encoder than a giant LLM.
- Predictable Scaling: ASR performance follows a power law, allowing for precise estimation of the compute required to reach specific performance targets.
- Impact of Backbone: Upgrading the encoder (e.g., from Whisper-medium to Whisper-large-v2) shifts the entire scaling curve downward, confirming that a better encoder is the most efficient way to scale.
Conclusion
EFIN provides a blueprint for efficient multi-modal LLM training. By prioritizing the optimization of smaller modal-specific encoders and utilizing early-exit alignment strategies, researchers can achieve state-of-the-art ASR performance with significantly reduced computational overhead. Future work will explore these scaling laws across different languages and varying LLM sizes.
Summary updated