Actor, Judge, or Teacher? Understanding Decision Models in Long-Horizon Agents

Haochen Liu, Qianhao Zhu, Zixuan Wang, Dingming Li, Tongbo Chen, Peng Qiu, Yingqi Qu, Weiming Lu, Jun Xiao, Yongliang ShenarXiv 2026 · arXiv

TL;DR

This paper demonstrates that "decision models"—specialized LLMs that output probabilities over discrete choices—can be mediocre at performing tasks themselves while remaining excellent at teaching other models how to solve them. By evaluating actions with the benefit of hindsight (seeing the full trajectory) and asking the right questions, these models provide dense, high-quality reward signals that significantly boost the performance of smaller agents in complex, long-horizon environments.

What Problem Does This Paper Address?

Building agents that can handle "long-horizon" tasks—those requiring many steps with strict dependencies—is notoriously difficult. A classic failure mode is the "greedy subgoal" trap: an agent might try to place a mug on a table before heating it, simply because "place mug" is a visible part of the final goal.

Reinforcement Learning (RL) typically relies on outcome rewards (did the agent succeed or fail at the end?), which are sparse and don't tell the agent which specific actions were helpful. While step-level rewards (process supervision) can solve this, they usually require expensive human labeling or constant, costly queries to a massive "teacher" LLM. The authors investigate whether cheaper, specialized "decision models" can fill this gap as automated judges and teachers.

Methods

The researchers evaluate six models (three dedicated decision models like JEV and Decider v1.2, and three general LLMs like Qwen and GPT-6 Luna) across three distinct roles:

  1. Actor: The model chooses the next action based on the current state.
  2. Judge: The model evaluates a previously recorded action. The authors test two contexts: Prefix (seeing only what happened before the action) and Hindsight (seeing the entire trajectory). They also test two questions: Value ("Will the task succeed?") and Progress ("Did this action help?").
  3. Teacher: The model's "Judge" scores are used as a dense reward signal to train a smaller policy (e.g., Qwen2.5-1.5B) using GRPO (Group Relative Policy Optimization).
Figure 1: One decision model queried in three ways, on the task of putting a heated mug on a table. Actor: choosing the next action. Judge: scoring an action of a recorded trajectory under two contexts (Prefix, Hindsight) and two questions (Value, Progress). Teacher: scoring the actions of policy rollouts, with the centered scores added to the GRPO outcome advantage. The middle row lists the six models we evaluate.

Datasets

The study uses two primary benchmarks for long-horizon agentic tasks:

  • ALFWorld: A text-based household environment where agents must navigate rooms and manipulate objects (e.g., "put a heated mug on the table").
  • WebShop: A simulated e-commerce site where agents must search, filter, and purchase items meeting specific criteria.

Key Insights and Ideas

  • Acting is harder than Judging: Models often fail as Actors because they can't resist the "easier" final subgoal. For instance, JEV succeeds on 70.8% of tasks with independent subgoals but drops to 14.7% when subgoals must be ordered.
  • Hindsight is a Superpower: When acting, a model must anticipate consequences. When judging with hindsight, it can observe them. Seeing the full trajectory allows a Judge to recognize that a seemingly "useless" action (like opening a microwave) was actually a vital prerequisite for a later success.
  • The Question Matters: Asking a Judge "Will the task succeed?" (Value) results in flat, uninformative scores because every action in a successful trajectory gets a high score. Asking "Did this action help?" (Progress) creates a much sharper signal, clearly separating milestone actions from "no-ops" or detours.
  • Weak-to-Strong Teaching: A model that is a poor actor can still be a great teacher. JEV only completes ~43% of ALFWorld tasks as an actor, but the 1.5B policy it teaches reaches nearly 83% success.
Figure 2: Actor success rates (%) on ALFWorld.

Key Results

  • Performance Boost: Using JEV as a Teacher (Hindsight + Progress) raised a Qwen2.5-1.5B policy's success rate from 72.8% to 82.9% on ALFWorld and from 56.8% to 71.4% on WebShop compared to standard outcome-only RL.
  • Efficiency: The 1.5B model trained with a decision-model teacher outperformed a much larger 7B model trained with standard outcome rewards.
  • Cost-Effective Supervision: Decision models provide more accurate judgments at a 28% to 43% lower cost per score compared to prompted general-purpose LLMs.
Figure 4: Prefix (left) and Hindsight (right) scores on the same states in Section 3.1.1.

Limitations and Future Work

  • Trajectory Length: The experiments were limited to episodes of roughly 30 actions. Scaling these "hindsight" judgments to much longer software engineering or tool-use tasks remains an open question.
  • Closed Models: Several high-performing models (JEV, GPT Decisions) are closed-source APIs with undisclosed training procedures, making it hard to pinpoint exactly why they judge better than their base LLMs.
  • Online vs. Offline: The current "Judge" requires the full trajectory (offline). Developing methods that can provide "hindsight-like" credit in an online setting (as the agent acts) is a promising future direction.

Summary updated

Abstract

Decision models answer a question with a typed output, such as a choice among options with a probability for each, so one model can choose an agent's actions, evaluate actions already taken, or score every action to teach a policy with reinforcement learning. Prior work has studied them mainly as actors, judging their usefulness by how well they act. We ask whether a decision model that acts poorly can still teach well. We compare three decision models and three general LLMs as actors and judges on ALFWorld, and use JEV as a teacher on ALFWorld and WebShop. As actors, all six models are pulled away from a subgoal that must come first by an easier one. As judges that see the whole trajectory, the same models credit the steps toward that subgoal, and asking whether each action helped, rather than whether the task will succeed, separates useful from useless actions. As a teacher, JEV raises a Qwen2.5-1.5B policy over GRPO from 72.8% to 82.9% success on ALFWorld and from 56.8% to 71.4% on WebShop. These results suggest that a poor actor can still teach well, because judging an action after the episode ends reveals the consequences that acting must anticipate. Code is available at https://github.com/ZJU-REAL/decision-agents.
BibTeX
@misc{liu2026actor,
  author = {Haochen Liu and Qianhao Zhu and Zixuan Wang and Dingming Li and Tongbo Chen and Peng Qiu and Yingqi Qu and Weiming Lu and Jun Xiao and Yongliang Shen},
  title = {{Actor, Judge, or Teacher? Understanding Decision Models in Long-Horizon Agents}},
  year = {2026},
  eprint = {2611.17167},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.17167},
}
Figure