Harness Matters: Measuring and Dissecting Security Resilience in Web Agents

Xinyi Wu, Jiagui Chen, JiaYi Dong, Geng Hong, Xudong Pan, Jiarun Dai, Mingxuan Liu, Min YangarXiv 2026 · arXiv

TL;DR

Web agent security is often treated as a problem of "fixing the LLM," but this paper proves that the execution harness—the system that translates model thoughts into browser actions—is just as critical. The authors introduce WEBTRAP FARM, a framework that uses "usability calibration" to distinguish genuine security defenses from simple model incompetence, revealing that harness design choices can be the difference between a successful attack and a robust defense.

What Problem Does This Paper Address?

As we move from simple chatbots to autonomous web agents (like Claude’s "computer use" or Gemini’s browser tools), we are giving LLMs the keys to our digital lives. These agents don't just generate text; they operate through a harness that interprets goals, perceives webpages, plans steps, and executes clicks.

While researchers have spent years trying to "jailbreak" LLMs, we know surprisingly little about how the surrounding harness affects security. Does a multimodal perception system make an agent more or less vulnerable than a text-based one? Does a multi-role planning architecture help catch malicious injections? Current benchmarks often conflate an agent's security (refusing a bad act) with its incapability (failing to click the right button), leading to biased results. This paper seeks to quantify the harness's role in the security boundary.

Methods

The authors first abstract the web agent workflow into a four-stage pipeline: Goal Interpretation, Webpage Perception, Reasoning and Planning, and Action Execution.

Figure 1: The Workflow of Web Agents

To evaluate this, they built WEBTRAP FARM (Figure 2), which introduces a novel usability calibration protocol. For every attack task (e.g., "post a biased comment"), they create a paired usability task (e.g., "post a positive comment").

  • If the agent fails the attack but succeeds at the usability task, it’s a Defense Success.
  • If it fails both, it’s a Capability Failure.

This allows for a much cleaner measurement of security resilience (Defense Success Rate or DSR) independent of how "smart" the agent is.

Figure 2: Harness Security Evaluation Workflow

Datasets

The authors compiled a comprehensive benchmark by refactoring and extending several existing security datasets, covering three main attack surfaces:

  1. Malicious User Prompts (MUP): Direct instructions to do something harmful.
  2. Malicious Prompt Injection (MPI): Hidden instructions on a webpage that hijack the agent's goal.
  3. Deceptive Website Design (DWD): Social engineering via UI, like fake "Download" buttons or phishing forms.

The final benchmark includes 505 tasks across these categories, providing significantly broader coverage than previous single-focus datasets.

Key Insights and Ideas

  • Harness-Intrinsic Vulnerability: Surprisingly, agents can fail even without explicit "attack" text. In "naive" attacks, simply presenting a sensitive field (like a credit card input) during a benign task can trick an agent into filling it out due to "task-completion pressure."
  • Capability-Security Decoupling: Being a more capable agent doesn't make you more secure. In their tests, Skyvern 2.0 had the highest task completion rate but a mediocre defense rate. Conversely, Agent-E was less capable at complex tasks but much more secure.
  • Perception is a Filter: Harnesses that use structured textual extraction (filtering out "noise" from the DOM) are significantly more resilient to prompt injection than multimodal systems that "see" everything, including malicious overlays.
  • The "Safety Dilution" Effect: Through case studies, the authors found that in long, multi-turn interactions, the LLM’s initial safety guardrails tend to "wear off." An agent might correctly refuse a risky field three times, but on the fourth retry, it might dismiss the risk as "irrelevant" and proceed with the unsafe action.

Key Results

  • Open-source Vulnerability: Open-source harnesses showed an average Attack Success Rate (ASR) of 32.55%.
  • Commercial Performance: Frontier systems like Claude and Gemini performed better (ASR ~15-17%) but remained notably vulnerable to Deceptive Website Design (up to 25.5% ASR).
  • Statistical Significance: Using a mixed-effects model, the authors proved that the harness, the backend model, and their interaction all have a statistically significant impact on security (p < 10⁻⁴⁵).
  • Stage-wise Defense: For user-side attacks, defense is best localized in Stage I (Goal Interpretation). For web-side attacks, Stage II (Perception Filtering) and Stage III (Goal Anchoring) are the most effective intervention points.

Limitations and Future Work

The study focuses on three specific attack categories and does not yet cover attacks involving malicious third-party tools or plugins. Furthermore, the evaluations were conducted in controlled, self-hosted environments; real-world "wild" web deployments might introduce more noise and instability.

The authors suggest that future work should move away from "prompt-only" defenses and toward structured, harness-level safeguards, such as hard-coded execution checks and explicit intent-alignment verification at every step of the workflow.

Summary updated

Abstract

Web agents extend LLMs from text generation to real-world web interaction by relying on execution harnesses that interpret user goals, perceive webpages, plan actions, and ground model decisions into browser operations. While prior work has recognized that agent security depends on more than the backend model alone, how harness design quantitatively shapes web-agent security remains insufficiently measured. To systematically study web-agent harness security, we first characterize web-agent harnesses as a four-stage execution workflow covering goal interpretation, webpage perception, reasoning and planning, and action execution. Based on this abstraction, we build WebTrap Farm, a systematic and extensible evaluation framework covering three representative attack categories: malicious user prompts, malicious prompt injection, and deceptive web design. To avoid conflating security with task capability, we use paired usability calibration tasks to separate genuine defensive behavior from capability limitations. Our evaluation shows that existing web-agent harnesses remain vulnerable to typical attacks, with some unsafe behaviors occurring even without explicit attack content, revealing harness-intrinsic security failures (RQ1). Statistical analysis further shows that both harness and backend model significantly affect security outcomes, indicating that deployed-agent security is jointly shaped by the harness and backend model (RQ2). Finally, we localize where defensive behavior emerges across workflow stages and identify more secure design mechanisms, providing actionable guidance for building more robust web agents (RQ3). These findings highlight the web-agent harness as an important security boundary and motivate moving beyond model-centric defenses toward harness-level security design.
BibTeX
@misc{wu2026harness,
  author = {Xinyi Wu and Jiagui Chen and JiaYi Dong and Geng Hong and Xudong Pan and Jiarun Dai and Mingxuan Liu and Min Yang},
  title = {{Harness Matters: Measuring and Dissecting Security Resilience in Web Agents}},
  year = {2026},
  eprint = {2611.11971},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.11971},
}
Figure