TL;DR
JudgeMix is a novel data synthesis framework that trains LLM agents by dynamically interleaving target model actions with expert interventions based on real-time "judge" evaluations. By only intervening when the model struggles and reweighting supervision to favor error correction over redundant mimicry, JudgeMix significantly outperforms standard imitation learning, boosting performance by up to 10 points on complex tool-use benchmarks.
What Problem Does This Paper Address?
Training autonomous LLM agents typically relies on Supervised Fine-Tuning (SFT) using "golden" trajectories generated by an expert (like GPT-4). However, this leads to a classic distribution mismatch (or exposure bias): the model only learns from perfect paths. Once it makes a small mistake during inference, it finds itself in a state it never saw during training and quickly spirals into failure.
Existing "target-aware" methods try to fix this by mixing the target model's own rollouts with expert guidance. But these methods are often "blind"—they either intervene at random steps (like DAgger) or overwrite the model’s actions even when they are already correct. This disrupts the model's native behavior and wastes the "data budget" on redundant supervision.
Methods
JudgeMix introduces a "Judge-in-the-loop" approach to data synthesis. The framework consists of three main components:
- Judge-Guided Intervention: During data generation, the target model proposes an action. A "judge" (usually a stronger LLM) evaluates if the action is semantically correct. If it is, the rollout continues with the target model's action. If not, the expert intervenes, providing a correct action to keep the trajectory on track.
- Error-Recovery State Collection: When the judge rejects an action, JudgeMix doesn't just throw it away. It records the "erroneous state" resulting from that mistake and pairs it with an expert's recovery action. This explicitly teaches the model how to bounce back from its own failures.
- Judge-Guided Supervision Reweighting: During the actual fine-tuning, JudgeMix applies a weighted loss. Steps where the target model was already correct are "downweighted" ($w < 1$), while steps where the model failed receive full supervision ($w = 1$). This prevents the model from being forced to blindly mimic the expert's exact phrasing when its own reasoning was already valid.
Datasets
The authors evaluated JudgeMix on two rigorous agentic tool-use benchmarks:
- AppWorld: A platform for personal assistant tasks (e.g., managing emails, calendars, and Spotify) across multiple stateful APIs.
- AutomationBench: A benchmark focused on professional workplace workflows involving enterprise software like CRM systems, spreadsheets, and helpdesks.
The experiments used GLM-5 as the expert/judge and Qwen3.5 (9B and 2B versions) as the target models.
Key Insights and Ideas
- Precision Matters: Randomly inserting expert actions (as in DAgger) is inefficient. JudgeMix's semantic guidance improves the trajectory success rate during data generation by 27 points compared to random intervention.
- Don't Fix What Isn't Broken: The "aha moment" here is the supervision reweighting. For a capable model (9B), the best performance actually occurred when supervision on correct steps was set to zero ($w=0$). This suggests that once a model is "mostly right," forcing it to mimic an expert's specific style can actually be counterproductive.
- The "Sweet Spot" for Small Models: Interestingly, smaller models (2B) benefit from a moderate weight ($w=0.5$) on correct steps. The authors hypothesize that even when a weak model gets the right answer, its underlying reasoning might be shaky, so some expert mimicry still helps.
Key Results
JudgeMix consistently outperformed all baselines, including Expert Demonstrations, Rejection Sampling (RFT), and On-policy Expert Correction (OEC).
- AutomationBench: JudgeMix achieved a 28.5 pass@1 score, a massive jump over the 17.8 achieved by standard expert demonstrations and the 21.0 from DAgger.
- AppWorld: On the "Test-Normal" set, JudgeMix (40.9) beat the runner-up DAgger (35.6) by over 5 points.
- Robustness: Even without a "task verifier" (a programmatic way to check if a task is done), JudgeMix maintained its lead, proving it is robust for open-ended tasks where final rewards are hard to define.
Limitations and Future Work
- Model Coupling: Because JudgeMix is "target-aware," the dataset you generate for a 2B model might not be optimal for a 70B model. This limits data reusability.
- Computational Cost: Querying a judge at every single step of data generation adds overhead. However, the authors argue this is offset by the fact that trajectories stay "on track" more often, leading to shorter, more successful training examples.
- Judge Reliability: The system is only as good as its judge. In highly ambiguous domains, an unreliable judge could provide misleading signals. Future work could explore using multiple judges or self-correction mechanisms.
Summary updated