Process Oriented Evaluation for Long Horizon Agentic Tasks

This direction is no longer present in the current build. Its last version stays readable and citable here.

Version 1 of 126 papers30 conceptsfrozen 2026-10-04

In the early days of evaluating language models, success was often binary: did the model provide the correct answer or not? However, as we move toward autonomous agents capable of navigating operating systems, conducting scientific research, or manipulating robotic arms, this "pass-fail" paradigm breaks down. If an agent fails to complete a 50-step workflow, a binary metric treats a model that failed at step 1 identically to one that failed at step 49.

To bridge this gap, a new consensus has emerged across the field: we must evaluate the process, not just the outcome. By decomposing complex tasks into intermediate milestones, researchers are gaining a high-resolution view of agent capability, allowing them to distinguish between "near-misses" and total failures.

The Binary Trap and the Need for Granularity

The fundamental problem with binary success metrics is that they provide no "gradient" for improvement. In long-horizon tasks, the probability of an agent successfully completing every single step is statistically low, often leading to a "floor effect" where all models appear equally incompetent.

To combat this, Chang Ma and colleagues introduced AgentBoard: An Analytical Evaluation Board of Multi-turn LLM Agents, which champions a Progress Rate metric. Instead of a 0 or 1, the Progress Rate quantifies how much of the task was completed, either by measuring the distance to the goal state or by tracking the completion of hierarchical subgoals. This granularity is essential for identifying models that are "almost there" but perhaps tripped up by a minor tool-use error at the final stage.

Similarly, in the realm of cybersecurity, Andy K. Zhang et al. developed Cybench: A Framework for Evaluating Cybersecurity Capabilities and Risk of Language Models. By breaking down professional-level Capture the Flag (CTF) tasks into intermediate subtasks, they can award partial credit, revealing a "capability cliff" where models might excel at initial reconnaissance but fail at complex cryptographic exploitation.

Mechanisms of Progress Tracking

How do we actually measure "partial success" in a way that is objective and automated? The research points to three primary mechanisms:

  1. Graph-Based Traversal: In CRAB: Cross-environment Agent Benchmark for Multimodal Language Model Agents, Tianqi Xu and Linyao Chen use a graph-based evaluation system. Each node in the graph represents a checkpoint or state change (e.g., "file downloaded," "terminal opened"). By monitoring these checkpoints, the system calculates a completion ratio based on the agent's path through the task graph.
  2. Predicate-Based Logic: For embodied agents in household environments, Matthew Chang and Gunjan Chhablani introduced PARTNR: A Benchmark for Planning and Reasoning in Embodied Multi-agent Tasks. This framework uses predicate logic to verify environmental states—such as whether an object is "in" a specific container—allowing for a precise percentage of task fulfillment.
  3. Sequential Milestone Mapping: In complex CRM workflows, the SCUBA benchmark by Ran Xu and Jing Gu tracks intermediate progress markers. This approach identifies exactly where an agent fails in a multi-step business process, providing "process rewards" that can be used to fine-tune models more effectively than final outcome rewards alone.

Diagnosing the "Knowing-Doing Gap"

Process-oriented evaluation does more than just provide a better score; it serves as a diagnostic tool. Davide Paglieri and the team behind BALROG: Benchmarking Agentic LLM and VLM Reasoning On Games use a progression-based scoring system to map agent milestones to human win probabilities. This revealed a significant "knowing-doing gap": models often understand the rules of a game (the "knowing") but fail to execute the sequence of actions required to win (the "doing").

This diagnostic power is also evident in Executing Instructions in Situated Collaborative Interactions, where Alane Suhr and Claudia Yan measure performance at every individual instruction step. This allows researchers to quantify an agent's resilience to "cascading errors"—where a single mistake in a long sequence makes all subsequent steps impossible.

From Engineering to Science: Evaluating the Trajectory

As agents move into scientific domains, the evaluation must shift from "did the code run?" to "was the research path sound?" Hein-Mun Lam and Tingting Chen address this in FML-bench by analyzing the Exploration Diversity and research trajectory of agents. By tracking completion rates across the transition from hypothesis to implementation, they can identify whether an agent is merely refining a single failed idea or exploring a broad breadth of scientific possibilities.

This sentiment is echoed by Xuefeng Bai in their survey on feedback mechanisms, which argues that evaluating intermediate reasoning steps is the only way to determine if a success was a result of robust planning or merely an "accidental" correct outcome.

By moving toward these granular, milestone-based metrics, the field is developing a more nuanced understanding of agent intelligence—one that values the journey as much as the destination.

Papers in this version

History & citationVersion 1 · 2026-10-04

No longer present in the current build · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_process_oriented_evaluation_for_long_hor_aba821e4,
  title = {Process Oriented Evaluation for Long Horizon Agentic Tasks},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/process-oriented-evaluation-for-long-horizon-agentic-tasks/rdr_aba821e4c03d43aa9be277ab9dded632},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-22},
}

This exact version (version 1, frozen text):

@misc{lacuna_process_oriented_evaluation_for_long_hor_aba821e4_v1,
  title = {Process Oriented Evaluation for Long Horizon Agentic Tasks},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/process-oriented-evaluation-for-long-horizon-agentic-tasks/rdr_aba821e4c03d43aa9be277ab9dded632/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-22},
}