TL;DR
Adapting speech recognition (ASR) models to new domains usually involves training a student on noisy "pseudo-labels" generated by a larger teacher model. This paper shows that treating these noisy labels as rewards in a Reinforcement Learning (RL) framework—specifically Group Relative Policy Optimization (GRPO)—is far more effective than treating them as ground-truth targets via Supervised Fine-Tuning (SFT), leading to relative error reductions of up to 56%.
What Problem Does This Paper Address?
Automatic Speech Recognition (ASR) models often struggle when moved from general datasets (like audiobooks) to specific domains (like banking or healthcare). Since manual transcription is expensive and slow, the standard solution is self-training: a strong "teacher" model (like Whisper) transcribes the target audio, and a "student" model is fine-tuned on these pseudo-labels using standard cross-entropy loss (SFT).
The structural flaw in SFT is that it treats every token in the pseudo-label as a target, forcing the student to memorize the teacher's mistakes. This often leads to the student inheriting or even amplifying the teacher's error profile. While previous work focused on filtering or improving the labels themselves, this paper asks: Can we change the training objective to be inherently robust to label noise?
Methods
The authors propose using Group Relative Policy Optimization (GRPO) instead of SFT.
- Standard SFT: Minimizes token-level cross-entropy. It is "unconditional"—it forces the model toward the target regardless of what the model itself would have predicted.
- GRPO: For each audio snippet, the student samples a group of $G$ different hypotheses (rollouts). Each hypothesis is scored against the noisy pseudo-label using a reward function (1 - Word Error Rate). These rewards are then normalized within the group to calculate "advantages."
- The "Filtering" Mechanism: The key innovation is the realization that GRPO is on-policy. The gradient only acts on sequences the student actually generates. If the teacher makes a mistake that the student doesn't naturally produce, that error is never reinforced. Because two different models (teacher and student) rarely make the exact same mistake, the student effectively "filters" the teacher's noise simply by sampling from its own distribution.
Datasets
The study evaluates the approach across several challenging conversational domains:
- Banking (DefinedAI): The primary benchmark, featuring English customer-agent telephone calls (~54h training).
- Healthcare & Insurance (DefinedAI): Used to test domain transfer (~11h and ~27h training, respectively).
- Common Voice 26: Used for multilingual testing in low-resource languages (Hindi, Hebrew, Vietnamese, and Serbian).
- Teachers: A "quality ladder" of Whisper models (Tiny to Large-V3) and other architectures like Qwen3 ASR, Phi-4 Multimodal, and Audio Flamingo 3.
Key Insights and Ideas
- Labels as Rewards, Not Targets: By treating pseudo-labels as a reference for a reward rather than a ground-truth target, the model avoids verbatim memorization of teacher errors.
- SFT is Poisonous for RL: In typical LLM training, SFT is a prerequisite for RL. Here, the authors find that running SFT before GRPO is actually harmful when labels are noisy. SFT forces teacher errors into the student's "support" (the set of sequences it is likely to generate), which collapses the reward variance that GRPO needs to distinguish good hypotheses from bad ones.
- The "Tiny" Beats "Large" Effect: Remarkably, a GRPO student trained on the weakest labels (Whisper Tiny) outperformed an SFT student trained on the strongest labels (Whisper Large-V3). This suggests that the training objective matters more than the teacher's quality.
- Self-Limiting Noise: As shown in Figure 1, SFT copies nearly 100% of teacher errors. In contrast, direct GRPO copies fewer errors as label quality worsens, effectively "ignoring" the most corrupted utterances.
Key Results
- Banking Domain: GRPO achieved a 12.17% WER using Whisper Large-V3 labels, compared to 17.21% for SFT.
- Robustness: When using Whisper Tiny labels (which are very noisy), GRPO reached 14.06% WER, while SFT lagged far behind at 32.11%.
- Error Analysis: Rollout statistics (Figure 2/4) explain the success: erroneous teacher words appear in only 10-30% of GRPO rollouts. Because they are rare, their negative impact cancels out during group-relative normalization.
- Domain Transfer: The advantage held steady in Healthcare (18.72 → 14.64 WER) and Insurance (19.59 → 15.70 WER) domains.
Limitations and Future Work
- The Support Requirement: Direct GRPO only works if the student model's initial distribution already contains the "correct" tokens. For non-Latin scripts (like Hindi or Hebrew) where the English-pre-trained student couldn't even produce the target script, a short initial SFT stage was necessary to establish the basic output format.
- Conservative Bias: The WER-based reward tends to penalize "hallucinations" (insertions) heavily, leading the model to be slightly more prone to deletions.
- Future Directions: The authors suggest combining GRPO with iterative pseudo-labeling (where the student becomes the teacher for the next round) and using confidence-based filtering to decide when to use SFT vs. GRPO.
Summary updated