Efficient Scaling for LLM-based ASR

Bingshen Mu, Yiwen Shao, Kun Wei, Dong Yu, Lei XiearXiv 2025 · DOI · arXiv

Introduction

Large Language Models (LLMs) have significantly advanced Automatic Speech Recognition (ASR) by integrating contextual reasoning and linguistic "common sense." However, the standard approach—integrating a speech encoder with a massive LLM—is computationally expensive. This paper introduces EFIN (Encoder First INtegration), a training strategy designed to optimize ASR performance under a fixed computational budget. EFIN achieves a 21.1% relative reduction in error rates while using 50% less computational power compared to traditional joint-training methods.

The EFIN Framework

The core challenge in LLM-based ASR is the high cost of optimizing billions of parameters. The researchers propose that the "intelligence" of the speech encoder is often undervalued. They introduce a three-stage strategy to prioritize encoder optimization before full system integration.

Figure 1: Architecture of the LLM-based ASR system featuring a speech encoder, projection layer, and LLM with LoRA adaptation.

Three-Stage Training Strategy

  1. Independent Encoder Fine-tuning: The speech encoder is fine-tuned independently using its original objective (e.g., Whisper's training task). This is computationally efficient as the encoder is much smaller than the LLM.
  2. Preliminary Alignment: The encoder and LLM are frozen while a small projection layer is trained. The authors found that "preliminary convergence" (stopping early) is sufficient, saving significant compute.
  3. LLM Adaptation: The LLM and projection layer undergo joint training using LoRA (Low-Rank Adaptation) to minimize parameter updates and costs.

Scaling Laws and Performance

To quantify efficiency, the authors derived a Neural Scaling Law for LLM-ASR, modeling the relationship between Character Error Rate (CER) and total floating-point operations (FLOPs) as a power-law equation: $L = \beta C^{-\alpha}$.

Figure 5: Scaling curves for different training strategies, showing that Strategy 5 (EFIN) achieves the lowest CER for a given compute budget.

As shown in the scaling curves, EFIN (Strategy-5) is the most efficient path. It outperformed the strongest baseline (Strategy-3) with half the FLOPs. Interestingly, jointly training all components from scratch (Strategy-6) led to degraded performance, highlighting the difficulty of simultaneous multi-modal optimization.

Key Insights

  • Encoder-First Advantage: Investing compute in a high-quality, independently trained encoder provides better features for the LLM, leading to faster and better overall convergence. It is more cost-effective to spend compute on a small encoder than a giant LLM.
  • Predictable Scaling: ASR performance follows a power law, allowing for precise estimation of the compute required to reach specific performance targets.
  • Impact of Backbone: Upgrading the encoder (e.g., from Whisper-medium to Whisper-large-v2) shifts the entire scaling curve downward, confirming that a better encoder is the most efficient way to scale.

Conclusion

EFIN provides a blueprint for efficient multi-modal LLM training. By prioritizing the optimization of smaller modal-specific encoders and utilizing early-exit alignment strategies, researchers can achieve state-of-the-art ASR performance with significantly reduced computational overhead. Future work will explore these scaling laws across different languages and varying LLM sizes.

Summary updated

BibTeX
@misc{mu2025efficient,
  author = {Bingshen Mu and Yiwen Shao and Kun Wei and Dong Yu and Lei Xie},
  title = {{Efficient Scaling for LLM-based ASR}},
  year = {2025},
  eprint = {2508.04096},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2508.04096},
}
Figure