WEPO: Web Element Preference Optimization for LLM-based Web Navigation

Jiarun Liu, Jia Hao, Chunhong Zhang, Zheng HuAAAI 2025 · DOI

TL;DR

This paper introduces WEPO, a novel framework that leverages preference learning to improve LLM-based web navigation agents. By training LLMs to distinguish between relevant and irrelevant HTML elements using a distance-based sampling method, WEPO significantly enhances task success rates and operation accuracy, achieving state-of-the-art performance on the Mind2Web benchmark.

What Problem Does This Paper Address?

Autonomous web navigation agents, powered by Large Language Models (LLMs), are rapidly advancing. However, current approaches often struggle to effectively utilize the inherent redundancy of HTML elements on web pages for training. Existing methods either rely on traditional reinforcement learning with complex reward functions or supervised learning that may not fully capture nuanced user preferences. This paper addresses the challenge of efficiently training LLM agents to understand and interact with complex web environments by proposing a method that learns from human-like preferences, specifically focusing on how to select the correct HTML elements for a given task.

Methods

The core of WEPO is Web Element Preference Optimization, which integrates Direct Preference Optimization (DPO), a preference learning technique, into web navigation tasks. Instead of relying on explicit reward functions, WEPO trains LLMs by presenting them with pairs of "preferred" and "dis-preferred" element selections.

The novelty lies in how these preference pairs are generated:

  • Distance-Based Negative Sampling: WEPO samples "non-salient" or less relevant HTML elements as negative examples. This is achieved by analyzing the DOM tree structure and identifying elements that are functionally or semantically distant from the correct element for a given task. This heuristic approach is more efficient than random sampling and leverages the inherent structure of web pages.
  • HTML Pruning: To manage the complexity of web pages, WEPO employs an element-centric approach to prune HTML snippets, focusing on a key element and its ancestors within a defined depth and width. This helps the model concentrate on relevant parts of the page.
  • DPO Integration: The sampled preference pairs (correct element vs. sampled negative elements) are used to fine-tune the LLM using the DPO objective. This objective maximizes the likelihood of selecting the preferred element and minimizes the likelihood of selecting the dis-preferred ones, effectively aligning the agent's actions with user intent.

Datasets

The primary dataset used for training and evaluation is Mind2Web. This benchmark is noted for its diversity, encompassing 137 real-world websites and 2350 multi-step tasks across 31 domains, making it suitable for evaluating generalization capabilities.

Key Insights and Ideas

  • Preference Learning for Web Navigation is Under-Explored: While preference learning has been successful in NLP tasks, its application to web navigation was largely absent. WEPO demonstrates its significant potential in this domain.
  • HTML Structure is Rich with Implicit Preference Signals: The hierarchical and often redundant nature of HTML provides a fertile ground for learning preferences without explicit human annotation for every single interaction.
  • Distance-Based Sampling is Key to Efficiency: Simply sampling random elements as negatives is less effective. WEPO's distance-based approach, informed by DOM tree proximity, significantly improves the learning process and sample efficiency.
  • Element Selection is Crucial for Overall Performance: The paper highlights that improving the agent's ability to select the correct HTML element is a primary driver of success, even more so than just accurately predicting the input or select values for those elements.

Key Results

WEPO achieved state-of-the-art performance on the Mind2Web benchmark.

  • Overall Improvement: The Llama3-8B-WEPO model (8 billion parameters) achieved a 13.8% improvement over the WebAgent baseline and a 5.3% improvement over the CogAgent baseline.
  • Outperforming Baselines: WEPO significantly outperformed traditional supervised fine-tuning (SFT) methods, including the MindAct baseline by 20.0% and WebAgent by 13.8%.
  • Efficiency: WEPO achieved these results with smaller model parameters and faster inference times compared to some larger models.
  • Generalization: The method demonstrated strong generalization across cross-domain, cross-task, and cross-website test sets.
  • Distance-Based Sampling Advantage: The distance-based sampling method showed a 2.3% to 3% higher SSR compared to random sampling.

Limitations and Future Work

  • Theoretical Analysis: The authors acknowledge the need for deeper theoretical analysis to fully understand why preference optimization is so effective for web navigation.
  • HTML Encoder: Future work could benefit from a dedicated HTML encoder that better understands hierarchical and nested structures, potentially inspired by models like WebFormer and HTML-T5.
  • Large Context Lengths: The current implementation has not been tested with extremely large context lengths (e.g., 128k tokens), which could be crucial for handling complex web pages.
  • Real-World Generalization: While WEPO shows promise, its generalization across highly diverse, real-world web interfaces with varying designs and technologies needs further investigation. The authors plan to test WEPO on other benchmarks like WebLINX and WebVoyager.

Summary updated

BibTeX
@inproceedings{liu2025wepo,
  author = {Jiarun Liu and Jia Hao and Chunhong Zhang and Zheng Hu},
  title = {{WEPO: Web Element Preference Optimization for LLM-based Web Navigation}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2025},
  doi = {10.1609/aaai.v39i25.34863},
}
Figure