Agentic Data Sampling Framework for Multilingual ASR via Linguistic-Aware Error-Driven Optimization

Zhiheng Qian, Junhao Du, Jing Peng, Hanqi Li, Kunyang Peng, Feng Xue, Lu Chen, Shuai Wang, Kai YuarXiv 2026 · arXiv

This paper introduces a novel approach to training multilingual Automatic Speech Recognition (ASR) models by treating data selection not as a static preprocessing step, but as an adaptive, "agentic" feedback loop. By employing a Large Language Model (LLM) agent to analyze linguistic error patterns, the framework intelligently decides which data the model needs to see next to improve most efficiently.

TL;DR

The authors propose an agentic framework that uses an LLM to adaptively sample training data for multilingual ASR based on phonological, morphological, and syntactic error analysis. This method outperforms full-data training and random sampling while significantly accelerating convergence and reducing the total data required.

What Problem Does This Paper Address?

Training a single ASR model to handle multiple languages is challenging because languages differ vastly in their data availability, phonetic structures, and grammatical rules. Traditionally, researchers use simple heuristics—like balancing language ratios or selecting data based on loss—to decide what to train on.

However, standard metrics like Word Error Rate (WER) are "black boxes"; they tell you a model is failing but not why. An error could be a simple mishearing of a sound (phonology), a missed plural ending (morphology), or a dropped preposition (syntax). Without knowing the linguistic source of the error, data sampling remains a game of trial and error. This paper bridges the gap between error diagnosis and data optimization.

Methods

The researchers introduce a closed-loop system that operates in iterative training rounds. The workflow consists of several key components:

  1. Linguistic Rubric: Instead of just looking at WER, the system generates a detailed report. It uses tools like eSpeak NG to convert text to phonemes (IPA) to track sound-level errors and rule-based detectors to identify morphological (e.g., gender/tense markers) and syntactic (e.g., word order) mistakes.
  2. The LLM Agent: An LLM acts as the "brain" of the sampling process. It receives the linguistic rubric, training statistics, and budget constraints.
  3. Hierarchical Algorithm Pool: The agent doesn't just pick data; it picks strategies. It chooses from a pool of seven algorithms (see Table 1 below) across three levels of granularity:
  4. Domain Level: Adjusting the weights of entire languages (e.g., using DoReMi or DOGE).
  5. Sample Level: Selecting specific utterances that match difficult phoneme distributions.
  6. Token Level: Applying operations like gradient weighting to specific words within a sentence (e.g., RHO-1 or TOSS).
Table 1. Multi-level data sampling algorithm pool.

The entire process is visualized in the framework overview below:

Fig. 1. Overview of the proposed agentic data sampling framework. The blue nodes show the common ASR model training pipeline, while the purple nodes belong to our agentic data sampling workflow.

Datasets

The study utilized a 12-language subset of the Common Voice corpus. The languages included Belarusian, Chinese, Dutch, French, German, Italian, Japanese, Polish, Portuguese, Russian, Spanish, and Turkish.

  • Training: 600 hours total (50 hours per language).
  • Evaluation: 12 hours for development and 12 hours for testing (1 hour per language).
  • Backbone: The models were fine-tuned from Whisper-medium using Low-Rank Adaptation (LoRA).

Key Insights and Ideas

  • Evaluation as Actionable Feedback: The core "aha moment" is moving evaluation from a final endpoint to a training signal. By explaining errors through a linguistic lens, the agent can make targeted interventions—for example, selecting more samples containing specific difficult phonemes.
  • Dynamic Strategy Shifting: The agent exhibits "adaptive behavior." In early rounds, it tends to choose conservative, loss-based strategies. As it receives specific linguistic feedback, it shifts to more aggressive, targeted algorithms like Phone Error Distribution sampling.
  • Linguistic Synergy: The framework was particularly effective for Romance languages (Spanish, Portuguese, French). The authors suggest the agent may be leveraging shared linguistic features across these related languages to optimize sampling.

Key Results

The agentic approach demonstrated clear superiority over traditional methods:

  • Accuracy: The "Agentic-300h" method achieved an average error rate of 19.4, significantly beating the no-sampling baseline (23.9) and random sampling (22.5).
  • Language Coverage: It achieved the best results on 9 out of 12 evaluated languages.
  • Efficiency: The method achieved better-than-full-data performance while using less data and compute, showing much faster convergence in the training process.
  • Error Reduction: Beyond WER, the model showed a 52.1% reduction in morphological errors (like inflectional markers) and a 23.4% reduction in preposition/function-word errors compared to the base Whisper model.

Limitations and Future Work

While the results are promising, the authors note a few areas for growth:

  • Syntactic Analysis: The number of word-order deviations in the test set was too small to draw statistically robust conclusions about global syntactic modeling.
  • Scalability: Future work will involve extending the framework to a broader range of languages and conducting more granular ablations to see which parts of the linguistic rubric (phonology vs. morphology vs. syntax) contribute most to the gains.
  • Agent Diversity: Exploring how different LLM backbones affect the quality of the sampling decisions is another planned direction.

Summary updated

Abstract

Multilingual automatic speech recognition (ASR) has achieved rapid progress with large-scale pretrained models and multilingual speech corpora. Previous studies have proved the pivotal role of data sampling in multilingual ASR performance. However, existing sampling strategies largely rely on time-consuming manual, heuristic-based designs. To address this, we propose a linguistic-aware agentic data sampling framework that introduces a data sampling LLM agent into the multilingual ASR training loop. The agent is driven by a linguistically grounded rubric that analyzes ASR errors from phonological, morphological, and syntactic perspectives, and uses the resulting feedback together with training statistics, budget constraints, and algorithm profiles to select domain-level, sample-level, and token-level sampling strategies. This design turns evaluation results into adaptive data-selection decisions rather than treating WER/CER as final endpoints only. We conduct experiments on twelve Common Voice languages using Whisper-medium as the ASR backbone. The results show that our method achieves performance better than full-data training while using less training data and lower computational resources, and substantially accelerates training convergence compared to conventional sampling strategies.
BibTeX
@misc{qian2026agentic,
  author = {Zhiheng Qian and Junhao Du and Jing Peng and Hanqi Li and Kunyang Peng and Feng Xue and Lu Chen and Shuai Wang and Kai Yu},
  title = {{Agentic Data Sampling Framework for Multilingual ASR via Linguistic-Aware Error-Driven Optimization}},
  year = {2026},
  eprint = {2610.36538},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2610.36538},
}
Figure