SkillShift-Bench: Benchmarking Skill Learning under Environment Evolution for Web Agents

Bo-Yu Chen, Zhi Zhou, Yu-Feng LiarXiv 2026 · arXiv

TL;DR

SkillShift-Bench is a new benchmark designed to test how well web agents can reuse learned skills as websites evolve. The researchers found that even minor changes in a website's UI or runtime conditions cause significant performance drops, and they introduced MPCR, a test-time adaptation method that helps agents recover by intelligently reranking actions based on current page feasibility.

What Problem Does This Paper Address?

Web agents are increasingly capable of learning "skills"—reusable sequences of actions—to handle complex, long-horizon tasks like "find my first cancelled order." However, these agents are typically trained and tested in static, "closed" environments.

In the real world, the web is a moving target. Websites update their layouts, change their underlying DOM structure, or introduce transient interruptions like pop-up modals. When a website evolves, a learned skill might remain semantically relevant (the goal hasn't changed), but it becomes "locally non-executable" because the button moved, the locator changed, or a pop-up is blocking the click.

Existing benchmarks often fail to isolate these specific types of "environment evolution," making it hard to diagnose exactly why an agent failed. This paper fills that gap by introducing a controlled environment to study the "skill-transfer gap."

Methods

The authors introduce two major components: a benchmark and an adaptation algorithm.

1. SkillShift-Bench

This benchmark evaluates agents across three distinct "shift modes" that mimic real-world website evolution:

  • Contextual Shift: Changes in routes, navigation, or page structure (e.g., moving a menu item).
  • Perceptual Shift: Changes in visual or DOM signals (e.g., changing CSS classes or button styling) while keeping the functionality intact.
  • Execution Shift: Injecting runtime blockers (like modals) that intercept interactions, requiring the agent to recover and resume.
Figure 1: Three controlled shift modes in SkillShift-Bench. Aligned variants preserve task intent, task-critical entities, required functionality, and success criteria, while Contextual, Perceptual, and Execution Shifts respectively target skill invocation, grounding evidence, and runtime feasibility.

2. MPCR (Masked Prior-Guided Candidate Reranking)

To solve the failures identified by the benchmark, the authors propose MPCR. Instead of relearning the entire skill library (which is expensive), MPCR adapts at "test-time." * Propose: The agent suggests candidate actions based on its learned skills (the "prior"). * Verify: It checks the current page state to see if those actions are actually feasible (e.g., "Is the button actually there?"). * Score & Rank: It uses a heuristic scoring system and an LLM-based pairwise reranker to select the best executable action that still aligns with the original skill.

Figure 2: Execution flow of MPCR within the web- agent loop.

Datasets

The benchmark is instantiated across three mature, self-hosted open-source web systems:

  • GitLab: Developer collaboration tasks.
  • Magento: E-commerce administration.
  • WordPress: Content management.

The researchers created 355 natural language tasks across these sites. They evaluated four state-of-the-art skill-learning methods: AWM, ASI, SkillWeaver, and WALT.

Key Insights and Ideas

  • Execution Shift is the Hardest: Runtime interruptions (Execution Shift) exert the strongest pressure on agents, causing the most dramatic drops in success rates compared to layout or styling changes.
  • The "Local Executability" Mismatch: Most failures aren't because the agent forgot what to do, but because it couldn't figure out how to do the next step in a slightly different environment.
  • Adapt the Executor, Not the Library: You don't always need to retrain your agent when a website updates. By adding a "feasibility-aware" layer to the action selection process, agents can navigate around many environment shifts.
  • Regularization is Critical: MPCR stays "anchored" to the learned skill library. This prevents the agent from getting distracted by irrelevant but "feasible" actions on the page.

Key Results

The benchmark revealed a systematic degradation in performance across all existing methods as environments evolved.

  • Baseline Fragility: For the ASI method, success rates plummeted from 53.65% in the source environment to just 15.79% under Execution Shift in the target environment.
  • MPCR Gains: MPCR consistently improved performance across all shift types. On average, it increased success rates from 37.90% to 41.26% (an 8.87% relative improvement).
  • Pressure-Dependent Success: The more "fragile" the environment became, the more MPCR helped. Under the toughest Execution Shift conditions, relative gains reached up to 24.55%.
Success Rate profiles under controlled environment evolution

Limitations and Future Work

  • Scope: The current benchmark focuses on English desktop web interactions. Future versions could expand to mobile, multi-lingual, and accessibility-oriented settings.
  • Model Capability: The study primarily used highly capable LLMs (like DeepSeek-V3.2 and GPT-5 mini). It remains to be seen how well smaller, more resource-constrained models would handle these shifts.
  • Recovery Mechanisms: While MPCR helps select the best action, the authors note that more advanced "recovery" mechanisms (that specifically target dismissing obstructions) could be a fruitful area for future research.

Summary updated

Abstract

Skill learning enables web agents to reuse prior experience for long-horizon interactive tasks. However, learned skills are typically evaluated in closed environments, overlooking the fact that real-world web deployment involves continuous shifts in invocation context, perceptual grounding, and execution feasibility. To this end, we introduce SkillShift-Bench, the first benchmark for evaluating skill learning with controlled open-environment shifts. SkillShift-Bench simulates Contextual Shift, Perceptual Shift, and Execution Shift across three self-hosted web systems, and evaluates agents with diagnostics covering task success, interaction cost, skill use, and skill failure. Comprehensive experiments reveal systematic robustness degradation under controlled shifts, characterized by lower success rates, higher interaction costs, and more frequent skill failures, with Execution Shift exerting the strongest pressure. We further formulate shifted skill execution as a nonstationary local executability problem and provide both a dynamic-regret analysis and a practical test-time adaptation method, demonstrating the feasibility of improving learned-skill execution without relearning the skill library. Code, tasks, environments, documentation, and the leaderboard are available at https://skillshift-bench.github.io/.
BibTeX
@misc{chen2026skillshift,
  author = {Bo-Yu Chen and Zhi Zhou and Yu-Feng Li},
  title = {{SkillShift-Bench: Benchmarking Skill Learning under Environment Evolution for Web Agents}},
  year = {2026},
  eprint = {2610.33528},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2610.33528},
}
Figure