TL;DR
This paper demonstrates that "decision models"—specialized LLMs that output probabilities over discrete choices—can be mediocre at performing tasks themselves while remaining excellent at teaching other models how to solve them. By evaluating actions with the benefit of hindsight (seeing the full trajectory) and asking the right questions, these models provide dense, high-quality reward signals that significantly boost the performance of smaller agents in complex, long-horizon environments.
What Problem Does This Paper Address?
Building agents that can handle "long-horizon" tasks—those requiring many steps with strict dependencies—is notoriously difficult. A classic failure mode is the "greedy subgoal" trap: an agent might try to place a mug on a table before heating it, simply because "place mug" is a visible part of the final goal.
Reinforcement Learning (RL) typically relies on outcome rewards (did the agent succeed or fail at the end?), which are sparse and don't tell the agent which specific actions were helpful. While step-level rewards (process supervision) can solve this, they usually require expensive human labeling or constant, costly queries to a massive "teacher" LLM. The authors investigate whether cheaper, specialized "decision models" can fill this gap as automated judges and teachers.
Methods
The researchers evaluate six models (three dedicated decision models like JEV and Decider v1.2, and three general LLMs like Qwen and GPT-6 Luna) across three distinct roles:
- Actor: The model chooses the next action based on the current state.
- Judge: The model evaluates a previously recorded action. The authors test two contexts: Prefix (seeing only what happened before the action) and Hindsight (seeing the entire trajectory). They also test two questions: Value ("Will the task succeed?") and Progress ("Did this action help?").
- Teacher: The model's "Judge" scores are used as a dense reward signal to train a smaller policy (e.g., Qwen2.5-1.5B) using GRPO (Group Relative Policy Optimization).
Datasets
The study uses two primary benchmarks for long-horizon agentic tasks:
- ALFWorld: A text-based household environment where agents must navigate rooms and manipulate objects (e.g., "put a heated mug on the table").
- WebShop: A simulated e-commerce site where agents must search, filter, and purchase items meeting specific criteria.
Key Insights and Ideas
- Acting is harder than Judging: Models often fail as Actors because they can't resist the "easier" final subgoal. For instance, JEV succeeds on 70.8% of tasks with independent subgoals but drops to 14.7% when subgoals must be ordered.
- Hindsight is a Superpower: When acting, a model must anticipate consequences. When judging with hindsight, it can observe them. Seeing the full trajectory allows a Judge to recognize that a seemingly "useless" action (like opening a microwave) was actually a vital prerequisite for a later success.
- The Question Matters: Asking a Judge "Will the task succeed?" (Value) results in flat, uninformative scores because every action in a successful trajectory gets a high score. Asking "Did this action help?" (Progress) creates a much sharper signal, clearly separating milestone actions from "no-ops" or detours.
- Weak-to-Strong Teaching: A model that is a poor actor can still be a great teacher. JEV only completes ~43% of ALFWorld tasks as an actor, but the 1.5B policy it teaches reaches nearly 83% success.
Key Results
- Performance Boost: Using JEV as a Teacher (Hindsight + Progress) raised a Qwen2.5-1.5B policy's success rate from 72.8% to 82.9% on ALFWorld and from 56.8% to 71.4% on WebShop compared to standard outcome-only RL.
- Efficiency: The 1.5B model trained with a decision-model teacher outperformed a much larger 7B model trained with standard outcome rewards.
- Cost-Effective Supervision: Decision models provide more accurate judgments at a 28% to 43% lower cost per score compared to prompted general-purpose LLMs.
Limitations and Future Work
- Trajectory Length: The experiments were limited to episodes of roughly 30 actions. Scaling these "hindsight" judgments to much longer software engineering or tool-use tasks remains an open question.
- Closed Models: Several high-performing models (JEV, GPT Decisions) are closed-source APIs with undisclosed training procedures, making it hard to pinpoint exactly why they judge better than their base LLMs.
- Online vs. Offline: The current "Judge" requires the full trajectory (offline). Developing methods that can provide "hindsight-like" credit in an online setting (as the agent acts) is a promising future direction.
Summary updated