Frontier-OPD: Discovering the Edge of Agent Capability and Teaching Agents to Go Beyond

Zheng Jiang, Yiming Chen, Houde Qian, Ling Li, Chaoyang Li, Lifeng SunarXiv 2026 · arXiv

TL;DR

Frontier-OPD is a novel framework for training multi-turn agents by identifying their "learning frontier"—the specific decision points where a student agent struggles but can successfully continue if given a brief nudge by a teacher. By focusing distillation on these high-impact moments and validating them through autonomous continuation, the method significantly outperforms standard on-policy distillation baselines across complex interactive benchmarks.

What Problem Does This Paper Address?

Building effective multi-turn agents (like LLMs that can navigate websites or perform household tasks) is difficult because a single mistake early on can derail the entire interaction. On-Policy Distillation (OPD) tries to fix this by having a "teacher" model provide feedback on the "student" model's own generated paths.

However, the authors identify a major gap: not all teacher feedback is equally useful.

  1. The "Too Easy" Problem: If a student already performs well, teacher guidance adds little value.
  2. The "Too Hard" Problem: If an intervention leads the student into a state far beyond its current capabilities, it will likely fail immediately after the teacher stops helping.

The challenge is finding the "Goldilocks zone"—the learning frontier—where the student is currently failing but is capable of learning how to recover and finish the task.

Methods

Frontier-OPD operationalizes the learning frontier through a four-step cycle, illustrated in the overview below:

Overview of the Frontier-OPD framework: 1) Explore trajectories, 2) Locate frontier nodes, 3) Validate teacher bridges, and 4) Optimize.
  1. Explore: The student performs rollouts in the environment.
  2. Locate: The system builds an Empirical Trajectory Graph. By hashing observations and actions into "state keys," it merges repeated visits across different runs. It then uses trajectory-balanced aggregation to rank these nodes based on student-teacher disagreement, ensuring that a single looping trajectory doesn't skew the results.
  3. Validate: For the top-ranked "frontier candidates," the system inserts a short teacher bridge (1-K turns of expert action). Crucially, it then lets the student resume control for several turns.
  4. Filter: Guidance is only accepted for training if the "bridge + student continuation" shows better agreement with the teacher than the original student-only path. This ensures the student can actually build on the teacher's help.
  5. Distill: The accepted teacher responses are added to the training pool for a standard OPD update.

Datasets

The authors evaluated Frontier-OPD on three diverse, interactive benchmarks:

  • ALFWorld: A text-based environment for embodied household tasks (e.g., "put a clean sponge in the cabinet").
  • WebShop: A grounded web interaction task requiring agents to search and navigate product pages to fulfill specific shopping requests.
  • ScienceWorld: A complex simulator for scientific reasoning and procedural tasks.

The experiments primarily used Qwen3-32B as the teacher and Qwen3-1.7B or Qwen3-4B as the students.

Key Insights and Ideas

  • Continuation as Validation: The most important insight is that a teacher's action is only "usable" if the student can successfully pick up the ball and keep running afterward. This behavioral proxy filters out guidance that is too advanced for the student's current state.
  • State-Level vs. Trajectory-Level: By organizing experience into a graph of shared states, the model identifies persistent weaknesses across different attempts rather than just reacting to one-off failures.
  • Training Efficiency: While Frontier-OPD has a higher per-step computational cost (due to the extra validation rollouts), it actually reaches target performance levels faster in terms of total wall-clock time because the data it collects is much higher quality.
Performance comparison across ALFWorld, WebShop, and ScienceWorld showing Frontier-OPD (green) outperforming various baselines.

Key Results

Frontier-OPD consistently achieved the highest autonomous performance across all tested scales and environments:

  • ALFWorld: The 1.7B student reached a 45.5% success rate, more than doubling the 22.1% achieved by standard OPD.
  • ScienceWorld: Performance jumped from 6.2% (OPD) to 14.5% (Frontier-OPD).
  • Efficiency: In ALFWorld, Frontier-OPD reached a 55% success rate in 46 minutes, compared to 74 minutes for standard OPD, despite the more complex collection process.

Limitations and Future Work

  • Environment Requirements: The method requires a replayable environment to test teacher bridges, which may not be feasible in all real-world settings.
  • State Abstraction: The use of hashed state keys is a proxy; it might occasionally merge histories that have different underlying learning needs.
  • Skill Tracing: While the method improves overall task success, the authors note it doesn't yet directly trace whether specific skills are mastered at specific nodes. Future work could involve more granular measurements of how the "frontier" expands over time.

Summary updated

Abstract

On-policy distillation aligns teacher feedback with a student's current behavior, yet acquiring useful supervision for multi-turn agents requires identifying both where guidance is needed and whether the student can sustain the interaction afterward. We introduce Frontier-OPD, which addresses this problem through the student's learning frontier: decision situations that remain challenging but admit useful autonomous continuation after brief teacher guidance. To operationalize this frontier, Frontier-OPD organizes on-policy experience into a shared-state trajectory graph, where trajectory-balanced aggregation of teacher-student disagreement identifies distinct intervention candidates while preserving the histories needed for replay. Short teacher bridges are then assessed by comparing agreement over the bridge and the ensuing student continuation with an equally long segment of the original trajectory, providing a behavioral proxy for guidance that the current student can build on. Accepted teacher responses augment ordinary distillation data, and fresh rollouts renew the frontier estimate as the policy changes. Experiments on ALFWorld, WebShop, and ScienceWorld report higher mean autonomous performance than the compared distillation baselines across the evaluated student scales and teacher configurations, with ablations indicating complementary effects of candidate selection and guidance filtering. In one ALFWorld timing setting, the additional collection cost is offset by a shorter time to a target success rate.
BibTeX
@misc{jiang2026frontier,
  author = {Zheng Jiang and Yiming Chen and Houde Qian and Ling Li and Chaoyang Li and Lifeng Sun},
  title = {{Frontier-OPD: Discovering the Edge of Agent Capability and Teaching Agents to Go Beyond}},
  year = {2026},
  eprint = {2610.32802},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2610.32802},
}
Figure