Abstract
Graphical User Interface (GUI) agents can automate complex tasks across digital environments, but their development is hindered by the scarcity of high-quality trajectory data for training. Existing approaches rely on expensive human annotation, making them unsustainable at scale. We propose AgentTrek, a scalable data synthesis pipeline that generates web agent trajectories by leveraging publicly available tutorials. Our three-stage method: (1) automatically harvests and filters tutorial-like texts from the internet using a specialized classification model, (2) transforms these texts into structured task specifications with step-by-step instructions, and (3) employs a visual-language model (VLM) agent to execute these instructions in real environments, while a VLM-based evaluator verifies trajectory correctness. The synthesized trajectories encompass multiple modalities, including text-based HTML observations with function-calling API actions, and vision-based screenshot observations with pixel-level actions. This multimodal data, enriched with chain-of-thought reasoning, enables agents to achieve state-of-the-art performance on both textual web browsing benchmarks (e.g., WebArena) and visual web grounding and browsing benchmarks (e.g., ScreenSpot Web and Multimodal Mind2Web). Furthermore, our fully automated approach significantly reduces data collection costs, achieving a cost of just $0.55 per high-quality trajectory without human annotators. Our work demonstrates that guided replay using web tutorials is a practical and scalable strategy for training advanced GUI agents, paving the way for more capable and autonomous digital assistants.
Introduction
Building intelligent agents capable of navigating Graphical User Interfaces (GUIs)—such as booking flights or managing CRM systems—requires massive amounts of high-quality "trajectory" data. A trajectory consists of a sequence of observations (screenshots or HTML), reasoning steps, and actions (clicks, typing). Currently, obtaining this data is a significant bottleneck. High-quality trajectories typically necessitate expensive human annotation, while existing synthetic methods often lack realism because they rely on models "imagining" environments rather than interacting with them. Consequently, GUI agents frequently struggle with long-horizon tasks, precise visual grounding, and generalization to unfamiliar websites.
The AgentTrek Pipeline
AgentTrek addresses these challenges by turning the vast, untapped knowledge in online "how-to" guides into a structured training school for agents. The pipeline consists of three main stages:
1. Automatic Tutorial Collection
The system harvests millions of potential tutorials from massive web corpora like RedPajama. It employs a rule-based pre-filter and a specialized FastText classifier to identify genuine "how-to" content. Once identified, a Large Language Model (LLM), such as GPT-4o-mini, transforms these raw texts into structured task specifications, including high-level goals, prerequisites, and granular step-by-step instructions.
2. Guided Replay for Trajectory Generation
Instead of merely reading the tutorial, a Vision-Language Model (VLM) agent executes the task in a live browser environment via BrowserGym. The structured tutorial serves as a "cheat sheet," guiding the agent through complex workflows and menus. During this "replay," the system records multimodal data: HTML, accessibility trees, screenshots, and the agent's internal chain-of-thought reasoning.
3. VLM-Based Evaluation
To ensure data quality, a separate VLM evaluator reviews the recorded trajectory. It verifies whether the agent successfully achieved the goal and adhered to the instructions, effectively filtering out failed attempts or hallucinated successes.
Experimental Results and Insights
The authors generated the AgentTrek dataset, containing 10,398 successful trajectories across 127 websites and 11 task categories (e.g., e-commerce, productivity, social media).
State-of-the-Art Performance
A Qwen2.5-32B model fine-tuned on AgentTrek achieved a 22.40% success rate on WebArena, significantly outperforming the original GPT-4o (13.10%). On the ScreenSpot benchmark, the AgentTrek-trained model more than doubled the visual grounding performance of the base Qwen2-VL model, rising from 30.7% to 67.4% average accuracy.
Scaling and Cost Efficiency
The research demonstrates a clear scaling law: agent performance improves steadily as the volume of AgentTrek data increases. Crucially, 10,000 synthetic trajectories can approach the performance levels of human-annotated data. Furthermore, at $0.55 per trajectory, AgentTrek is orders of magnitude cheaper than human annotation, which can cost tens of dollars per sample.
Conclusion and Future Work
AgentTrek proves that web tutorials are a form of "distilled expertise" that can effectively guide agents through complex real-world environments. While highly effective, the approach faces challenges such as temporal decay (websites changing over time) and domain concentration. Future work may explore "dynamic adaptation," where agents learn to update tutorial instructions themselves when encountering broken steps during replay. Overall, AgentTrek provides a scalable and practical strategy for training the next generation of autonomous digital assistants.
Summary updated