Web agents get much faster and more accurate when they stop clicking through pages one screenshot at a time and instead call small reusable programs ("tools") that were learned from a website in advance. The catch is that websites change. When a button is renamed or a URL parameter disappears, a cached tool can quietly do the wrong thing without raising any error. The agent, which trusts its tools the way a language model trusts its training data, then reports success on a task it never completed. This is knowledge obsolescence moved from facts to skills: the agent's procedural memory has gone out of date, and its hallucinated success is the visible symptom.
We propose to treat every learned tool as a belief with an expiry condition. When a tool is discovered, we record an executable contract from its successful runs. The contract is a set of checks on the page after the tool runs: which elements, URL patterns, and rendered regions changed, and how. Each time the tool is called, its contract is re-run against the live page. A failed contract marks the tool stale, and a dependency graph spreads that staleness to other tools that rely on the same page structures. The agent then re-discovers only the affected tools, not the whole site. The core bet is that cheap deterministic code checks, recorded from evidence, catch silent tool failures that both runtime exceptions and multimodal-model judges miss. It is also that targeted repair costs a small fraction of full offline re-discovery.
Where this sits
Two systems supply the starting point and both name the gap we target. WALT: Web Agents that Learn Tools reverse-engineers site features into deterministic scripts and parameterized URLs, reaching 52.9% success on VisualWebArena and 50.1% on WebArena. Its authors say "schema drift" breaks these tools and leaves two options: slow agentic clicking or a full re-discovery pipeline. Agent JIT Compilation compiles requests into graphs of code tools with preconditions and postconditions. It gets up to a 10.4× speedup, and its invariant protocol cuts plan failures from 80% to 43%. It also needs a 25–90 minute offline warm-up per site and notes that redesigns break cached tools. Its postconditions are written to check the success of a planned call. They are not designed to detect that the tool itself has decayed. We build on both and add what neither has: an online mechanism that detects tool staleness and repairs it locally.
The detection side draws on two findings about why model-based checking is unreliable. Let's Think in Two Steps shows that multimodal judges agree with flawed trajectories. Its Self-Grounded Verification asks the judge to describe what success should look like before it sees the trajectory, which reduces this bias and adds about 20 points in online supervision. The authors also report that residual errors come from the judge misreading small interface elements. That is the perception–reasoning bottleneck that CharXiv documents for charts, where one misread tick mark turns sound reasoning into confident fabrication. CodePercept shows that executable code is a more exact perceptual intermediate than captions for static STEM images. We carry that lesson to interactive pages: check state through queries that execute, and fall back to a model's reading of pixels only where nothing executable exists.
The propagation side borrows from Can Agent Memory Systems Track Evolving State?. That paper's StateMem stores facts with explicit dependency links and uses a deterministic rechecker to flag stale values. The motivation is its finding that models anchor on outdated but salient facts even when the correct value is in context, and that chain-of-thought makes this worse. We expect the same trap in action spaces: a tool that worked many times is the "loud" fact. StateMem works only on conversational text. We apply its dependency-tracking idea to code tools that share page selectors and URL schemas.
Executable Code Actions Elicit Better LLM Agents motivates why exceptions are not enough. CodeAct self-debugs from tracebacks, but its authors observe that agents assume the contents of variables without inspecting them. Logical failures that raise no exception go unnoticed, and a stale tool clicking the wrong-but-existing element is exactly that kind of failure.
Core Claim
We claim three things. First, under realistic site drift, most tool failures are silent: the tool returns normally, and the agent's final answer is a hallucinated success. Second, contracts recorded from discovery-time evidence detect these failures with higher precision and recall than a strong multimodal judge, at negligible latency. Third, contract failures combined with a dependency graph localize which tools need repair well enough that targeted repair recovers most of the pre-drift success rate. It should do so at a small fraction of the cost of full re-discovery.
Research Program
The first component is contract recording. During WALT-style discovery, every successful run of a tool is logged with page state before and after the call. That state includes the DOM (the page's element tree), the URL, and screenshots. From the differences across runs we fit a contract: a conjunction of predicates that held on every successful run and that vary with the tool's arguments. Examples are "an element matching this role and text pattern exists before the call," "the URL gains parameter q equal to the argument," and "the result list region changes and contains the argument string." Predicates are drawn first from DOM and URL queries. Rendered-region predicates are used only for visual content that the DOM does not expose, such as product images on VisualWebArena. These are checked by pixel difference against a region bound to a stable selector. A multimodal model is called only as a last resort. Recording contracts from evidence, rather than having an LLM write them, is the key design choice, and we ablate it directly.
The second component is a dependency graph. Each tool is linked to the selectors, URL templates, and form schemas its code and contract touch. When a contract fails, the failing predicate identifies the broken structure. Every tool depending on that structure is pre-emptively marked suspect, in the same spirit as StateMem's rechecker. The agent is told explicitly in its context which tools are stale, so that it does not keep calling a familiar broken tool.
The third component is targeted repair. A stale tool is re-discovered by a short exploration seeded with the old script, the failed predicate, and a before/after difference of the relevant page region. Success is defined as passing the old contract's semantic predicates, such as "results contain the query," under new structural bindings. Repaired tools get new contracts.
Proposed Experiments
No drift benchmark exists in the file, so we build one. The WebArena and VisualWebArena sites are self-hosted, which lets us apply controlled redesigns to their templates. We will use four drift types: renaming labels and identifiers, moving elements, renaming or removing URL parameters, and restructuring forms and visual layout. Each type comes in a mild and a severe version. Tools are discovered on the original sites, and agents are then evaluated on the drifted sites with the original task sets.
Metrics are task success; silent-failure rate, meaning the fraction of tool calls that return normally but leave the page in a wrong state, labeled with the benchmarks' programmatic evaluators; false-success rate in agent final reports; staleness detection precision and recall at the tool level; and repair cost in wall-clock time and tokens, compared against the 25–90 minute warm-up reported for Agent JIT.
Baselines
The naive reference point is WALT's tools used unchanged on drifted sites, with fallback to agentic clicking only when a tool raises an exception. This measures how much drift silently hurts. The reported no-drift ceiling for this baseline is WALT at 52.9% on VisualWebArena and 50.1% on WebArena. The step-by-step reference is GPT-4o with Set-of-Marks, which VisualWebArena reports at 19.78% overall success. It is immune to tool staleness, so it is the floor any tool-based method must stay above under drift.
The strongest competitor for detection is WALT tools with a Self-Grounded Verification judge after each tool call, triggering fallback or re-discovery when the judge says the call failed. It is the strongest because it is the best published defense against agents accepting their own flawed steps. The file reports its gain as about +20 points in online supervision; it gives no number for SGV on drifted sites. We will run it with the same base model, the same tool library, and a per-call token budget at least matching our method's total repair-plus-check budget. Beating it means higher detection recall at equal or better precision, and higher end-to-end success on severe drift, at lower latency.
The strongest competitor for repair is full offline re-discovery of the drifted site with WALT's pipeline. It bounds the success we can hope to recover. Beating it means matching its success within a few points at a small fraction of its cost.
Three ablations isolate the key ingredients. Replacing recorded contracts with LLM-written postconditions, as in Agent JIT's protocol, tests whether evidence-based recording matters. Removing the dependency graph, so that only the tool whose contract failed is repaired, tests propagation. Restricting contracts to DOM and URL predicates tests whether the rendered-region channel matters at all.
Why not just…?
Why not just rely on runtime exceptions and fall back to clicking? This is how CodeAct-style self-debugging works, and its authors already observe logical failures that raise nothing. Our silent-failure measurement on the naive baseline tests this directly. If most drift failures turn out to raise exceptions, the detection half of this proposal is unnecessary, and we would report that.
Why not just use a multimodal judge such as Self-Grounded Verification after every call? It is a real threat on mild, visually obvious drift. Its authors, however, attribute remaining errors to misread fine interface details, and CharXiv shows that fine perceptual reading is exactly where strong models fail. The head-to-head detection comparison, broken down by drift type, settles this. We expect the judge to be competitive on layout moves and to lose on renamed parameters and labels, where the page looks plausible but is wrong.
Why not just periodically re-run full discovery? It works but is expensive: the reported warm-up is 25–90 minutes per site. It also does not know when to run, so between refreshes stale tools still produce false successes. Our cost-versus-success curve compares scheduled full refreshes against contract-triggered targeted repair.
Why not just check the DOM and drop vision entirely? This may well be sufficient for most tools. The DOM-only ablation answers the question, and we would happily simplify the method if the rendered-region channel adds nothing outside image-dependent VisualWebArena tasks.
Decisive checks
The proposal stands or falls on three results. Silent failures must be a large share of drift failures for the naive baseline. Recorded contracts must beat the Self-Grounded Verification judge on detection under severe drift. Targeted repair must approach full re-discovery success at a small fraction of its cost. If the second check fails, the contribution reduces to a drift benchmark and a cost study. If the third check fails, the dependency graph and repair loop are the parts to rethink.
References
- WALT: Web Agents that Learn Tools
- Agent JIT Compilation for Latency-Optimizing Web Agent Planning and Scheduling
- Let's Think in Two Steps: Mitigating Agreement Bias in MLLMs with Self-Grounded Verification
- CharXiv: Charting Gaps in Realistic Chart Understanding in Multimodal LLMs
- CodePercept: Code-Grounded Visual STEM Perception for MLLMs
- Can Agent Memory Systems Track Evolving State?
- Executable Code Actions Elicit Better LLM Agents
- VisualWebArena: Evaluating Multimodal Agents on Realistic Visual Web Tasks