WebAnchor: Anchoring Agent Planning to Stabilize Long-Horizon Web Reasoning

Xinmiao Yu, Liwen Zhang, Xiaocheng Feng, Yong Jiang, Bing Qin, Pengjun Xie, Jingren Zhou

TL;DR

WebAnchor introduces Anchor-GRPO, a two-stage reinforcement learning framework designed to stabilize long-horizon web reasoning agents. By identifying the "plan anchor" phenomenon—where the very first planning step disproportionately dictates the success of a complex task—the authors decouple planning and execution, optimizing the initial step with dense, experience-driven rubrics before refining execution via sparse rewards.

What Problem Does This Paper Address?

Large Language Model (LLM) agents excel at short-term tasks but frequently stumble during "deep research" assignments that require long-horizon web browsing, multi-step planning, and tool interaction. In these extended trajectories, errors compound rapidly, leading to catastrophic strategy drift.

The authors identify a critical bottleneck called the plan anchor phenomenon: the initial planning step acts as a structural anchor for the entire trajectory. If the agent makes a mistake in its very first step (e.g., bad task decomposition or poor tool selection), it triggers a cascading failure that is nearly impossible to recover from.

Figure 1: Illustrating the plan anchor phenomenon, the critical role of the first step in task accuracy, and the use of plan rubrics to guide optimization.

Standard reinforcement learning (RL) algorithms like Group Relative Policy Optimization (GRPO) struggle here because they distribute rewards uniformly across the trajectory or rely on sparse terminal rewards. This treats every action equally, failing to prioritize the disproportionate impact of the crucial first step.

Methods

To address this, the authors propose Anchor-GRPO, a two-stage RL framework that separates planning from execution within a single policy model.

Figure 2: Anchor-GRPO framework. Stage 1 optimizes the initial plan using the Rubrics Model, providing dense rewards. Stage 2 refines the trajectory with sparse re- wards, ensuring alignment with the plan.
  1. Stage 1: Anchor Plan Optimization
    The model acts as a Planner. To prevent downstream actions from interfering with planning updates, the authors introduce LLM Masking, which zeroes out gradients for all tokens generated after the first step. The initial plan is optimized using a dense reward from a Plan Rubrics Learner. This learner distills planning principles from past successful and failed self-play trajectories, iteratively refining a 6-dimensional rubric (Goal Alignment, Subgoal Coverage, Tool Appropriateness, Logical Ordering, Actionability, and Clarity/Conciseness) with human-in-the-loop calibration.

  2. Stage 2: Trajectory Level Executor Optimization
    The model transitions to an Executor, interacting with the web environment using Search and Visit tools. In this stage, it is optimized using a sparse, binary task-completion reward (1 for an exact match of the final answer, 0 otherwise), training the executor to strictly align its actions with the high-quality plan established in Stage 1.

Datasets

The framework was evaluated across four challenging web-based information-seeking benchmarks:

  • BrowseComp / BrowseComp_en: A benchmark evaluating web browsing capabilities.
  • BrowseComp-Zh: The Chinese language counterpart of BrowseComp.
  • GAIA: A multimodal, tool-use benchmark for general AI assistants.
  • XBench-DeepSearch: A benchmark specifically tailored for complex, deep-research web tasks.

Key Insights and Ideas

  • Asymmetric Step Importance: Not all reasoning steps are created equal. In long-horizon web tasks, early credit assignment is paramount because initial missteps cause irreversible drift in the tool-use trajectory.
  • Decoupled Training in a Single Model: Instead of maintaining separate planner and executor models, Anchor-GRPO achieves functional decoupling through masked credit assignment, updating only the planning logits in Stage 1 and execution logits in Stage 2.
  • Dense Rubrics Beat Naive Rewards: Evaluating plans across structured dimensions (like subgoal coverage and tool appropriateness) provides a much more stable and effective gradient for RL than simple binary success metrics or naive plan rewards.

Key Results

  • State-of-the-Art Performance: WebAnchor-30B achieved an impressive 76.4% Pass@1 on GAIA, outperforming proprietary models like OpenAI-o3 (70.5%) and specialized open-source agents like WebSailor-32B (53.2%).
  • Improved Web Browsing: On BrowseComp, WebAnchor-30B hit 46.0% Pass@1, outperforming baseline GRPO (42.0%) and First-step GRPO (41.3%).
  • First-Step Importance Confirmed: Ablation studies showed that optimizing the first step specifically yielded a 43.3% Pass@1, compared to just 33.0% when optimizing a random intermediate step.
  • Robust Scalability: The framework demonstrated strong scaling properties, with performance monotonically increasing alongside model size (from 3B to 30B) and context length (from 16k to 64k).

Limitations and Future Work

  • Domain Specificity: The evaluation focused strictly on web-browsing and information-seeking agents. The authors note that the plan anchor phenomenon likely applies to other domains (like code generation or robotics), but this has yet to be experimentally verified.
  • Rubric Optimization: While the Plan Rubrics Learner is highly effective, the authors acknowledge there is still room to improve the fine-grained criteria to capture even more subtle planning errors.
  • Alternative Optimization: Future work will explore integrating other advanced optimization techniques to further stabilize the initial planning phase.

Summary updated

Abstract

Large Language Model(LLM)-based agents have shown strong capabilities in web information seeking, with reinforcement learning (RL) becoming a key optimization paradigm. However, planning remains a bottleneck, as existing methods struggle with long-horizon strategies. Our analysis reveals a critical phenomenon, plan anchor, where the first reasoning step disproportionately impacts downstream behavior in long-horizon web reasoning tasks. Current RL algorithms, fail to account for this by uniformly distributing rewards across the trajectory. To address this, we propose Anchor-GRPO, a two-stage RL framework that decouples planning and execution. In Stage 1, the agent optimizes its first-step planning using fine-grained rubrics derived from self-play experiences and human calibration. In Stage 2, execution is aligned with the initial plan through sparse rewards, ensuring stable and efficient tool usage. We evaluate Anchor-GRPO on four benchmarks: BrowseComp, BrowseComp-Zh, GAIA, and XBench-DeepSearch. Across models from 3B to 30B, Anchor-GRPO outperforms baseline GRPO and First-step GRPO, improving task success and tool efficiency. Notably, WebAnchor-30B achieves 46.0% pass@1 on BrowseComp and 76.4% on GAIA. Anchor-GRPO also demonstrates strong scalability, getting higher accuracy as model size and context length increase.
BibTeX
@inproceedings{yu2026webanchor,
  author = {Xinmiao Yu and Liwen Zhang and Xiaocheng Feng and Yong Jiang and Bing Qin and Pengjun Xie and Jingren Zhou},
  title = {{WebAnchor: Anchoring Agent Planning to Stabilize Long-Horizon Web Reasoning}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2026},
  doi = {10.18653/v1/2026.findings-acl.2058},
}
Figure