JudgeMix: Judge-Guided Mixed-Rollout Data Synthesis for LLM Agents

Cheng-Yu Hsieh, Ting-Yao Hu, Sanjoy Chowdhury, Oncel Tuzel, Raviteja Vemulapalli

TL;DR

JudgeMix is a novel data synthesis framework that trains LLM agents by dynamically interleaving target model actions with expert interventions based on real-time "judge" evaluations. By only intervening when the model struggles and reweighting supervision to favor error correction over redundant mimicry, JudgeMix significantly outperforms standard imitation learning, boosting performance by up to 10 points on complex tool-use benchmarks.

What Problem Does This Paper Address?

Training autonomous LLM agents typically relies on Supervised Fine-Tuning (SFT) using "golden" trajectories generated by an expert (like GPT-4). However, this leads to a classic distribution mismatch (or exposure bias): the model only learns from perfect paths. Once it makes a small mistake during inference, it finds itself in a state it never saw during training and quickly spirals into failure.

Existing "target-aware" methods try to fix this by mixing the target model's own rollouts with expert guidance. But these methods are often "blind"—they either intervene at random steps (like DAgger) or overwrite the model’s actions even when they are already correct. This disrupts the model's native behavior and wastes the "data budget" on redundant supervision.

Methods

JudgeMix introduces a "Judge-in-the-loop" approach to data synthesis. The framework consists of three main components:

  1. Judge-Guided Intervention: During data generation, the target model proposes an action. A "judge" (usually a stronger LLM) evaluates if the action is semantically correct. If it is, the rollout continues with the target model's action. If not, the expert intervenes, providing a correct action to keep the trajectory on track.
  2. Error-Recovery State Collection: When the judge rejects an action, JudgeMix doesn't just throw it away. It records the "erroneous state" resulting from that mistake and pairs it with an expert's recovery action. This explicitly teaches the model how to bounce back from its own failures.
  3. Judge-Guided Supervision Reweighting: During the actual fine-tuning, JudgeMix applies a weighted loss. Steps where the target model was already correct are "downweighted" ($w < 1$), while steps where the model failed receive full supervision ($w = 1$). This prevents the model from being forced to blindly mimic the expert's exact phrasing when its own reasoning was already valid.
Overview of the JudgeMix framework showing the intervention and error-recovery process.

Datasets

The authors evaluated JudgeMix on two rigorous agentic tool-use benchmarks:

  • AppWorld: A platform for personal assistant tasks (e.g., managing emails, calendars, and Spotify) across multiple stateful APIs.
  • AutomationBench: A benchmark focused on professional workplace workflows involving enterprise software like CRM systems, spreadsheets, and helpdesks.

The experiments used GLM-5 as the expert/judge and Qwen3.5 (9B and 2B versions) as the target models.

Key Insights and Ideas

  • Precision Matters: Randomly inserting expert actions (as in DAgger) is inefficient. JudgeMix's semantic guidance improves the trajectory success rate during data generation by 27 points compared to random intervention.
  • Don't Fix What Isn't Broken: The "aha moment" here is the supervision reweighting. For a capable model (9B), the best performance actually occurred when supervision on correct steps was set to zero ($w=0$). This suggests that once a model is "mostly right," forcing it to mimic an expert's specific style can actually be counterproductive.
  • The "Sweet Spot" for Small Models: Interestingly, smaller models (2B) benefit from a moderate weight ($w=0.5$) on correct steps. The authors hypothesize that even when a weak model gets the right answer, its underlying reasoning might be shaky, so some expert mimicry still helps.
Ablation study showing how varying the supervision weight (w) impacts performance across different model sizes.

Key Results

JudgeMix consistently outperformed all baselines, including Expert Demonstrations, Rejection Sampling (RFT), and On-policy Expert Correction (OEC).

  • AutomationBench: JudgeMix achieved a 28.5 pass@1 score, a massive jump over the 17.8 achieved by standard expert demonstrations and the 21.0 from DAgger.
  • AppWorld: On the "Test-Normal" set, JudgeMix (40.9) beat the runner-up DAgger (35.6) by over 5 points.
  • Robustness: Even without a "task verifier" (a programmatic way to check if a task is done), JudgeMix maintained its lead, proving it is robust for open-ended tasks where final rewards are hard to define.

Limitations and Future Work

  • Model Coupling: Because JudgeMix is "target-aware," the dataset you generate for a 2B model might not be optimal for a 70B model. This limits data reusability.
  • Computational Cost: Querying a judge at every single step of data generation adds overhead. However, the authors argue this is offset by the fact that trajectories stay "on track" more often, leading to shorter, more successful training examples.
  • Judge Reliability: The system is only as good as its judge. In highly ambiguous domains, an unreliable judge could provide misleading signals. Future work could explore using multiple judges or self-correction mechanisms.

Summary updated

Abstract

Supervised fine-tuning (SFT) on expert trajectories is a standard paradigm for training LLM agents. However, it suffers from a well-known training-inference distribution mismatch, calling for target-aware data synthesis strategies that ground the training data in the target model's native state distribution. To bridge this gap, emerging approaches incorporate the expert either to guide mixed rollouts alongside the target model, or to provide dense step-wise supervision. Nonetheless, existing methods apply expert assistance blind to the target model's inherent behavior: they either disrupt the rollout using sub-optimal random intervention heuristics, or they enforce indiscriminate uniform supervision that unnecessarily overwrites the model's already-correct actions. We introduce JudgeMix, a judge-guided data synthesis framework that dynamically interleaves the target model and expert based on step-wise semantic evaluations. JudgeMix queries expert guidance precisely when the target model struggles, ensuring high trajectory success rates while keeping the state distribution aligned with the target model. It additionally uses the judge evaluation to reweight expert supervision on different steps, preserving model's inherent valid behavior while correcting its errors. On two agentic tool-use benchmarks, AppWorld and AutomationBench, JudgeMix demonstrates consistent superior performance to existing methods, with improvements up to 10 pass@1 points.
BibTeX
@misc{hsieh2026judgemix,
  author = {Cheng-Yu Hsieh and Ting-Yao Hu and Sanjoy Chowdhury and Oncel Tuzel and Raviteja Vemulapalli},
  title = {{JudgeMix: Judge-Guided Mixed-Rollout Data Synthesis for LLM Agents}},
  year = {2026},
  eprint = {2611.16174},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.16174},
}
Figure