TL;DR
FocusAgent introduces a simple, two-stage framework that uses a lightweight LLM as a dynamic retriever to prune web page accessibility trees (AxTrees) before passing them to a web agent. This method reduces observation context sizes by over 50% and slashes token costs while maintaining strong task performance. Crucially, this pruning step doubles as a security sanitizer, dropping the success rate of malicious prompt-injection attacks from over 80% to under 1%.
What Problem Does This Paper Address?
Large Language Model (LLM) web agents are increasingly used to automate complex, multi-step tasks on the internet. However, modern web pages are massive. Even when compressed from raw HTML/DOM into an Accessibility Tree (AxTree)—which represents the page's user interface elements—the text can easily span tens of thousands of tokens.
This massive context window creates three major bottlenecks:
- High Computational Cost: Processing massive inputs at every single navigation step quickly racks up expensive API bills.
- Context Saturation: LLMs can get distracted or overwhelmed by noisy, irrelevant information on a page, leading to poor action planning.
- Security Vulnerabilities: Long pages expose agents to indirect prompt injection attacks, where malicious text hidden in ads, banners, or pop-ups hijacks the agent's instructions to steal data or perform unauthorized actions.
Existing solutions rely on basic truncation (cutting off the bottom of the page) or standard embedding-based vector search. Truncation frequently deletes critical information, while embedding models struggle because web navigation requires an understanding of dynamic state and planning context, not just static semantic similarity to the user's initial goal.
Methods
FocusAgent solves this with a two-stage pipeline that decouples page filtering from action execution:
- Stage 1: LLM-Based Retrieval: The framework first numbers every line of the raw AxTree. A lightweight, inexpensive LLM (like GPT-4.1-mini) acts as a retriever. Guided by the user's goal, it uses Chain-of-Thought (CoT) reasoning to identify and output the specific line ranges necessary to complete the task.
- Soft Prompting Strategy: To ensure the retriever doesn't accidentally delete critical elements, the authors employ a "soft" prompting strategy. The retriever is explicitly instructed to prioritize recall—meaning if it is uncertain whether a line is relevant, it should keep it rather than risk task failure.
- Stage 2: Action Prediction: The irrelevant lines are stripped out, replaced by a simple placeholder indicating how many lines were pruned. This clean, compact AxTree is then passed to the primary backbone agent (e.g., GPT-4.1 or Claude-3.7-Sonnet), which decides on the next action (click, scroll, fill, etc.).
Datasets
The authors evaluated FocusAgent using the BrowserGym ecosystem across two primary web navigation benchmarks:
- WorkArena L1: A benchmark focused on enterprise knowledge work tasks (e.g., editing forms, sorting lists, ordering items in a service catalog).
- WebArena: A highly realistic web environment containing tasks spread across various categories like Shopping, Gitlab, Maps, and Reddit.
- DoomArena: A specialized security framework used to test web agents against adversarial prompt injections, specifically banner and pop-up attacks.
Key Insights and Ideas
- State and Planning Matter More Than Keywords: Traditional retrievers (like BM25 or embedding models) fail because they only look for text matching the user's goal. FocusAgent succeeds because an LLM retriever can infer that structural elements—like a shopping cart or a navigation sidebar—are necessary for the plan, even if they aren't explicitly mentioned in the goal text.
- Retriever History is Distracting: Counterintuitively, the authors found that providing the interaction history to the lightweight retriever degraded performance. Smaller models get confused by long interaction histories; they perform better when evaluating the current page state in isolation.
- Security via Sanitization: By forcing an LLM to actively select only what is relevant to the user's goal, adversarial prompt injections (like fake "ALERT" pop-ups trying to redirect the agent) are naturally ignored and pruned out. The attack text never even reaches the main backbone agent.
Key Results
- Massive Context Reduction: FocusAgent successfully pruned AxTree observations by 50% to 61% on average across benchmarks.
- Maintained Performance: On WorkArena L1, FocusAgent paired with a GPT-5-mini retriever achieved a 53.2% success rate, nearly matching the unpruned baseline agent's 53.6%, but at a fraction of the token cost.
- Drastic Security Improvements: In pop-up attack scenarios on DoomArena, standard agents suffered from massive vulnerabilities. FocusAgent dropped the Attack Success Rate (ASR) from over 80% down to less than 1% for both GPT-4.1 and Qwen backbone models.
- The Latency Trade-Off: The primary downside is speed. Because the system requires two sequential LLM calls per step (one to prune, one to act), FocusAgent increased per-step latency by an average of 7.6 seconds.
Limitations and Future Work
The most prominent limitation is the latency overhead introduced by the two-stage design, making it less ideal for real-time applications where sub-second responses are mandatory. Additionally, the system remains sensitive to prompt engineering; if the retriever is too aggressive, it can trap the agent in infinite loops (e.g., scrolling endlessly because an expand button was pruned away).
Future work includes exploring ways to optimize this latency and investigating more sophisticated, adaptive attacks that might attempt to mimic valid structural page elements to bypass the LLM retriever.
Summary updated