Marco-ASR: A Principled and Metric-Driven Framework for Fine-Tuning Large-Scale ASR Models for Domain Adaptation

Xuanfan Ni, Fei Yang, Fengping Tian, Qingjuan Li, Chenyang Lyu, Yichao Du, Longyue Wang, Weihua Luo, Kaifu ZhangarXiv 2025 · DOI · arXiv

Introduction

While general-purpose Automatic Speech Recognition (ASR) models like Whisper and Qwen2-Audio have achieved remarkable zero-shot performance across broad datasets, they frequently encounter significant performance degradation in specialized "Out-of-Domain" (OOD) scenarios. In domains such as medical transcription, legal proceedings, or noisy financial calls, the distribution shift in terminology, acoustic conditions, and speaker characteristics often renders standard models insufficient.

Marco-ASR is introduced as a principled framework designed to bridge this gap. By moving beyond traditional fine-tuning heuristics, it provides a metric-driven approach to adapt massive ASR models to niche domains, ensuring training stability and significant accuracy gains.

The Challenge of Large-Scale ASR Adaptation

Fine-tuning modern, Large Language Model (LLM)-based ASR systems is notoriously difficult. These models possess massive parameter spaces and are highly sensitive to training configurations. A primary hurdle identified by researchers is the "Loss Mirage": the observation that training loss often decreases steadily while the actual Word Error Rate (WER) fluctuates or even degrades. This decoupling makes standard early-stopping criteria unreliable.

Furthermore, different architectures exhibit varying levels of sensitivity. For instance, the deep decoder of Whisper-Large-V3 (32 layers) is prone to "catastrophic forgetting" and hallucinations if the learning rate is too aggressive, whereas lighter models like Whisper-Turbo (4 layers) show more robustness. Marco-ASR addresses these complexities through a two-stage strategy focusing on learning rate calibration and targeted data augmentation.

The Marco-ASR Framework

1. Performance-Informed Learning Strategy

Marco-ASR replaces static learning rates with a dynamic, performance-informed strategy tailored to the model architecture:

  • Encoder-Decoder Models (e.g., Whisper): The framework monitors the standard deviation of WER during training cycles. If the WER exhibits high volatility—indicating training instability—the framework automatically lowers the learning rate. Conversely, if the model remains stable, the rate is maintained or increased to accelerate convergence.
  • LLM-Based Models (e.g., Qwen2-Audio): These models leverage their internal linguistic knowledge to adapt rapidly. Marco-ASR uses the initial "domain gap" (the baseline WER on the target domain) to calibrate the starting learning rate. A larger initial gap prompts a more aggressive learning rate to bridge the domain divide quickly.

2. Target-Profile-Driven Data Augmentation

Rather than applying generic noise, Marco-ASR utilizes a "technical profile" derived from a small sample of the target domain. This profile measures specific acoustic characteristics, including Signal-to-Noise Ratio (SNR), sample rate, bit depth, and loudness (LUFS).

The framework then applies a transformation chain to clean, public training data to mimic these conditions. This chain includes resampling, reverberation, noise injection, and spectral filtering, effectively teaching the model to handle the target environment's specific acoustic signature before it encounters the actual domain data.

Experimental Setup and Benchmarking

The framework was evaluated across a diverse array of datasets to test its versatility:

  • Multilingual Performance: Evaluated using Fleurs, Common Voice 21, and GigaSpeech-2.
  • Multi-Domain Robustness: Tested on AMI (meetings), Earnings-22 (finance), WeNet Speech (YouTube), and MDT-AC001 (in-car Chinese speech).
  • Covo-DA Benchmark: The authors introduced a specialized version of the Common Voice corpus, augmented to simulate the challenging conditions of telephony channels.

Key Insights and Results

The implementation of Marco-ASR yielded several critical insights into the behavior of large-scale ASR models:

  • The Importance of Augmentation: Fine-tuning on profile-matched augmented data consistently outperformed both the baseline and models fine-tuned on original data alone.
  • LLM Efficiency: LLM-based ASR models demonstrated remarkable efficiency, often reaching competitive performance in as few as 200 training steps due to their pre-existing linguistic capabilities.
  • Quantifiable Gains:
    • Telephony: On the Covo-DA (English) set, Whisper-Large-V3 improved from a baseline of 28.47 WER to 20.55.
  • LLM Adaptation: Qwen2-Audio-Base saw a dramatic improvement on the telephony set, dropping from 75.39 WER to 19.10.
  • Acoustic Noise: On the MDT in-car dataset, Qwen2-Audio-Base improved from 10.87 WER to 7.49, demonstrating the framework's ability to handle both specialized terminology and environmental noise.

Conclusion

Marco-ASR provides a robust, metric-driven protocol for the domain adaptation of large-scale ASR models. By identifying the pitfalls of loss-based training and introducing profile-driven augmentation, the framework offers a reliable path for developers to deploy high-accuracy speech recognition in specialized environments. Future work will focus on refining early-stopping criteria and extending these automated pipelines to low-resource languages and more extreme acoustic environments.

Summary updated

Abstract

Automatic Speech Recognition (ASR) models have achieved remarkable accuracy in general settings, yet their performance often degrades in domain-specific applications due to data mismatch and linguistic variability. This challenge is amplified for modern Large Language Model (LLM)-based ASR systems, whose massive scale and complex training dynamics make effective fine-tuning non-trivial. To address this gap, this paper proposes a principled and metric-driven fine-tuning framework for adapting both traditional and LLM-based ASR models to specialized domains. The framework emphasizes learning rate optimization based on performance metrics, combined with domain-specific data transformation and augmentation. We empirically evaluate our framework on state-of-the-art models, including Whisper, Whisper-Turbo, and Qwen2-Audio, across multi-domain, multilingual, and multi-length datasets. Our results not only validate the proposed framework but also establish practical protocols for improving domain-specific ASR performance while preventing overfitting.
BibTeX
@misc{ni2025marco,
  author = {Xuanfan Ni and Fei Yang and Fengping Tian and Qingjuan Li and Chenyang Lyu and Yichao Du and Longyue Wang and Weihua Luo and Kaifu Zhang},
  title = {{Marco-ASR: A Principled and Metric-Driven Framework for Fine-Tuning Large-Scale ASR Models for Domain Adaptation}},
  year = {2025},
  eprint = {2512.22165},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2512.22165},
}
Figure