TL;DR
AutoSurfer is a novel framework that automatically generates high-quality training data for web-automation agents by systematically exploring websites using a breadth-first strategy. By grounding task synthesis and trajectory refinement in actual navigation paths, it significantly reduces model hallucinations and produces highly diverse training datasets. When used to fine-tune open-source vision-language models, AutoSurfer achieves a state-of-the-art 24.23% task completion accuracy on the WebArena benchmark, outperforming previous synthetic data generation methods.
What Problem Does This Paper Address?
Building multimodal Large Language Model (LLM) agents capable of automating complex web tasks (like booking a flight or managing a code repository) is a major goal in AI. However, these agents frequently fail on complex or unfamiliar websites because they lack high-quality web interaction data for training.
Existing automated data generation methods fall short in two main ways:
- Incomplete Exploration: They rely on random-walk exploration or propose tasks solely from a website's homepage, missing deeper functionalities hidden behind multi-level menus or dropdowns.
- Hallucinated Tasks: Because they analyze web pages in isolation, they often synthesize ambiguous or impossible tasks, leading to broken and unreliable training trajectories.
As a result, current agents struggle to map user intent to the correct sequence of graphical user interface (GUI) actions.
Methods
AutoSurfer introduces a three-stage pipeline designed to mimic how a human explores and learns a new website: Comprehensive Exploration, Complex Task Synthesis, and Trajectory Refinement.
- Systematic Breadth-First Exploration: Instead of wandering randomly, AutoSurfer maintains a queue of discovered pages and action traces. It recursively expands hidden UI elements like menus and dropdowns. Crucially, it differentiates between fixed elements (like search bars) and dynamic elements (like product lists), sampling only a few dynamic elements to avoid redundant exploration. It also propagates knowledge across pages to prevent duplicating tasks.
- Trajectory-Grounded Task Synthesis: Rather than inventing tasks from isolated page content, AutoSurfer uses the entire navigation path leading up to a page to synthesize both action-oriented and information-seeking tasks. This ensures the generated tasks are realistic and achievable.
- Trajectory-Guided Refinement: The initial exploration paths can be messy or sub-optimal. AutoSurfer feeds these raw trajectories as "hints" to a web agent, which replays the task to optimize the path, remove redundant actions, and clean up the task descriptions.
Finally, these refined trajectories are converted into a supervised fine-tuning (SFT) dataset in ShareGPT format to train website-specific LLMs (wLLMs).
Datasets
The authors evaluated AutoSurfer using the WebArena benchmark, a highly realistic, self-hostable web environment. To evaluate single-website agent performance efficiently, they focused on four distinct domains, selecting one task per template:
- CMS (Content Management System): 57 tasks
- Gitlab (Code Repository): 56 tasks
- Reddit (Forum): 26 tasks
- Shopping (E-commerce): 55 tasks
Key Insights and Ideas
- Human-like Exploration Matters: Shifting from random walks to a systematic breadth-first search allows the system to uncover deep site functionalities that prior methods completely missed.
- Context Cures Hallucination: Grounding task generation in a full exploration trajectory rather than a single page view ensures that the synthesized tasks match real user behavior and site capabilities.
- The Exploration-Refinement Gap: Exploration data alone is insufficient for training because the agent operates in a reduced action space (e.g., it rarely needs to scroll when just mapping a page). Task-specific refinement is necessary to force the agent to use the full action space (like scrolling and inputting data), providing much cleaner supervision for next-action learning.
Key Results
The authors fine-tuned Qwen2.5-VL-7B-Instruct using data generated by AutoSurfer and compared it against other state-of-the-art data generators:
- Overall Accuracy: AutoSurfer achieved an overall task completion accuracy of 24.23% (when trained on combined website data) and 23.71% (when training website-specific models, or wLLMs). This significantly outperforms prior methods like SynthAgent (19.59%), OS-Genesis (13.40%), and Explorer (4.64%).
- Domain Success: AutoSurfer showed massive improvements in specific domains, particularly on Reddit, where the website-specific model achieved 34.62% accuracy compared to SynthAgent's 15.38%.
- Task Diversity: Using text embeddings to calculate pairwise cosine similarity, AutoSurfer achieved a task diversity score of 0.7774, outperforming both OS-Genesis and SynthAgent (0.7459), proving it generates a broader distribution of unique tasks.
Limitations and Future Work
While AutoSurfer marks a notable advancement, some limitations remain:
- Single-Website Focus: The current implementation is primarily optimized for single-website tasks and excludes multi-domain workflows (such as tasks requiring Google Maps or cross-site navigation).
- Baseline Agent Constraints: To isolate the impact of data quality, the authors used a relatively simple LLM-based agent for evaluation. Consequently, the absolute accuracies across all tested models remain low compared to commercial, heavily engineered agent frameworks.
For future work, the authors suggest using AutoSurfer’s refined trajectories as an initialization step for reinforcement learning (RL), combining offline synthetic data with online interaction rewards to tackle longer-horizon, multi-website tasks.
Summary updated