Pseudo-Labels as Rewards, Not Targets: Group-Relative Policy Optimization for Semi-Supervised ASR

Shashi Kumar, Yanis Labrak, Sergio Burdisso, Esaú Villatoro-Tello, Petr Motlicek, Kadri Hacioglu, Andreas StolckearXiv 2026 · arXiv

TL;DR

Adapting speech recognition (ASR) models to new domains usually involves training a student on noisy "pseudo-labels" generated by a larger teacher model. This paper shows that treating these noisy labels as rewards in a Reinforcement Learning (RL) framework—specifically Group Relative Policy Optimization (GRPO)—is far more effective than treating them as ground-truth targets via Supervised Fine-Tuning (SFT), leading to relative error reductions of up to 56%.

What Problem Does This Paper Address?

Automatic Speech Recognition (ASR) models often struggle when moved from general datasets (like audiobooks) to specific domains (like banking or healthcare). Since manual transcription is expensive and slow, the standard solution is self-training: a strong "teacher" model (like Whisper) transcribes the target audio, and a "student" model is fine-tuned on these pseudo-labels using standard cross-entropy loss (SFT).

The structural flaw in SFT is that it treats every token in the pseudo-label as a target, forcing the student to memorize the teacher's mistakes. This often leads to the student inheriting or even amplifying the teacher's error profile. While previous work focused on filtering or improving the labels themselves, this paper asks: Can we change the training objective to be inherently robust to label noise?

Methods

The authors propose using Group Relative Policy Optimization (GRPO) instead of SFT.

  1. Standard SFT: Minimizes token-level cross-entropy. It is "unconditional"—it forces the model toward the target regardless of what the model itself would have predicted.
  2. GRPO: For each audio snippet, the student samples a group of $G$ different hypotheses (rollouts). Each hypothesis is scored against the noisy pseudo-label using a reward function (1 - Word Error Rate). These rewards are then normalized within the group to calculate "advantages."
  3. The "Filtering" Mechanism: The key innovation is the realization that GRPO is on-policy. The gradient only acts on sequences the student actually generates. If the teacher makes a mistake that the student doesn't naturally produce, that error is never reinforced. Because two different models (teacher and student) rarely make the exact same mistake, the student effectively "filters" the teacher's noise simply by sampling from its own distribution.

Datasets

The study evaluates the approach across several challenging conversational domains:

  • Banking (DefinedAI): The primary benchmark, featuring English customer-agent telephone calls (~54h training).
  • Healthcare & Insurance (DefinedAI): Used to test domain transfer (~11h and ~27h training, respectively).
  • Common Voice 26: Used for multilingual testing in low-resource languages (Hindi, Hebrew, Vietnamese, and Serbian).
  • Teachers: A "quality ladder" of Whisper models (Tiny to Large-V3) and other architectures like Qwen3 ASR, Phi-4 Multimodal, and Audio Flamingo 3.

Key Insights and Ideas

  • Labels as Rewards, Not Targets: By treating pseudo-labels as a reference for a reward rather than a ground-truth target, the model avoids verbatim memorization of teacher errors.
  • SFT is Poisonous for RL: In typical LLM training, SFT is a prerequisite for RL. Here, the authors find that running SFT before GRPO is actually harmful when labels are noisy. SFT forces teacher errors into the student's "support" (the set of sequences it is likely to generate), which collapses the reward variance that GRPO needs to distinguish good hypotheses from bad ones.
  • The "Tiny" Beats "Large" Effect: Remarkably, a GRPO student trained on the weakest labels (Whisper Tiny) outperformed an SFT student trained on the strongest labels (Whisper Large-V3). This suggests that the training objective matters more than the teacher's quality.
  • Self-Limiting Noise: As shown in Figure 1, SFT copies nearly 100% of teacher errors. In contrast, direct GRPO copies fewer errors as label quality worsens, effectively "ignoring" the most corrupted utterances.
Figure 1. Fraction of erroneous pseudo-label (PL) words that the student reproduces verbatim in its final output, as a function of the number of word errors in the utterance’s pseudo-label.

Key Results

  • Banking Domain: GRPO achieved a 12.17% WER using Whisper Large-V3 labels, compared to 17.21% for SFT.
  • Robustness: When using Whisper Tiny labels (which are very noisy), GRPO reached 14.06% WER, while SFT lagged far behind at 32.11%.
  • Error Analysis: Rollout statistics (Figure 2/4) explain the success: erroneous teacher words appear in only 10-30% of GRPO rollouts. Because they are rare, their negative impact cancels out during group-relative normalization.
Figure 2. Rollout statistics as a function of the number of PL word errors per utterance. Left: fraction of the G=4 rollouts per utterance that contain the erroneous PL word. Right: within-group standard deviation of the reward across the G=4 rollouts.
  • Domain Transfer: The advantage held steady in Healthcare (18.72 → 14.64 WER) and Insurance (19.59 → 15.70 WER) domains.

Limitations and Future Work

  • The Support Requirement: Direct GRPO only works if the student model's initial distribution already contains the "correct" tokens. For non-Latin scripts (like Hindi or Hebrew) where the English-pre-trained student couldn't even produce the target script, a short initial SFT stage was necessary to establish the basic output format.
  • Conservative Bias: The WER-based reward tends to penalize "hallucinations" (insertions) heavily, leading the model to be slightly more prone to deletions.
  • Future Directions: The authors suggest combining GRPO with iterative pseudo-labeling (where the student becomes the teacher for the next round) and using confidence-based filtering to decide when to use SFT vs. GRPO.

Summary updated

Abstract

Adapting ASR to a domain with only untranscribed audio typically relies on pseudo-labeling, a strong teacher transcribes the audio and a student is fine-tuned on it. Because supervised fine-tuning (SFT) treats every pseudo-label token as a target, the student inherits the teacher's errors. We show that using the same pseudo-labels as a reward reference for Group Relative Policy Optimization (GRPO) is more robust to label noise. In the banking domain, GRPO yields 29.3% to 56.2% relative WER improvements over SFT on Whisper-generated labels, and the advantage holds across four Whisper sizes, four teacher families, three domains and four languages. While SFT reproduces almost 100% of teacher errors verbatim, GRPO passes through only 27%. Rollout statistics explain why, erroneous pseudo-label words rarely appear in on-policy rollouts, so group-relative advantages stay aligned with the clean transcript, whereas SFT forces them into the policy's support and collapses the reward variance GRPO relies on.
BibTeX
@misc{kumar2026pseudo,
  author = {Shashi Kumar and Yanis Labrak and Sergio Burdisso and Esaú Villatoro-Tello and Petr Motlicek and Kadri Hacioglu and Andreas Stolcke},
  title = {{Pseudo-Labels as Rewards, Not Targets: Group-Relative Policy Optimization for Semi-Supervised ASR}},
  year = {2026},
  eprint = {2610.32572},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2610.32572},
}
Figure