On the Fine-Grained Planning Abilities of VLM Web Agents

Surgan Jandial, Yinong Oliver Wang, Andrea Bajcsy, Fernando De la TorreEMNLP 2025

TL;DR

Vision-Language Models (VLMs) show promise as web agents, but their ability to plan complex sequences of actions remains poorly understood. This paper introduces a novel evaluation framework to assess VLMs' fine-grained planning skills, revealing that current models struggle significantly, even with basic temporal reasoning and plan assessment, highlighting a critical gap for reliable web agent development.

What Problem Does This Paper Address?

Modern AI agents, particularly those powered by Vision-Language Models (VLMs), are increasingly tasked with navigating and interacting with the web to complete user-defined goals. While VLMs excel at perception (understanding visual and textual content) and action execution, their ability to plan – to devise a coherent sequence of actions to achieve a goal – is a crucial but understudied component. Existing evaluations often focus on overall task success rates, which mask underlying issues in planning capabilities. This paper aims to bridge this gap by providing a fine-grained evaluation of VLMs' planning skills, specifically their temporal understanding and plan assessment abilities, to understand their readiness as reliable web agents.

Methods

The authors propose a novel evaluation framework that breaks down VLM planning into two key dimensions:

  1. Temporal Understanding: This dimension assesses the VLM's ability to understand the order of events and predict future states.
  2. Temporal Ordering: VLMs are presented with a task and two screenshots representing different steps in a web trajectory. They must identify which screenshot represents an earlier step.
  3. Future State Prediction: VLMs are given an initial form state and two action plans. They must determine if both plans lead to the same final state before submission.
  4. Plan Assessment: This dimension evaluates the VLM's ability to reason about and select the most effective plan.
  5. Real Web Environment Study: VLMs are presented with a task, a webpage screenshot, and four action plans. They must choose the plan that most effectively accomplishes the task.
  6. Edge Case Test Study: Similar to the real web environment study, but uses synthetic data with controlled errors (e.g., pre-filled fields, incorrect checkbox selections) to challenge VLMs' robustness.

The evaluation uses both instruction-based prompting and Chain-of-Thought (CoT) prompting to assess different reasoning styles.

Datasets

The paper leverages existing datasets and creates new benchmarks:

  • Existing Datasets:
    • MiniWob++: Used for generating synthetic data for future state prediction and edge case studies.
  • GUIAct: Used for temporal ordering tasks.
  • MM-M2W: Used for real-world web environment plan assessment.
  • Newly Introduced Benchmarks: The paper repurposes and augments these datasets to create specific evaluation tasks for temporal ordering, future state prediction, and plan assessment, including synthetic edge cases designed to challenge VLMs.

Key Insights and Ideas

  • VLMs struggle with basic temporal reasoning: Even simple tasks of identifying which of two screenshots comes earlier in a web interaction are challenging for most VLMs, suggesting a lack of genuine temporal understanding.
  • Spurious heuristics over semantic logic: VLMs often rely on superficial cues (like input order) rather than the actual semantic logic of web interactions to make decisions.
  • Plan assessment is a significant bottleneck: VLMs perform poorly when asked to choose the most effective plan, especially in challenging edge cases or when plans involve subtle differences in action sequences.
  • Scaling doesn't guarantee planning improvement: While some models show performance gains with increased size, this doesn't consistently translate to better fine-grained planning abilities.
  • GUI-specific models don't necessarily outperform generalists in planning: Despite excelling in GUI navigation, these models can struggle with general reasoning tasks when evaluated on planning.

Key Results

  • Temporal Ordering: Most VLMs fail to surpass a random guess baseline (50%), with human raters achieving ~96.2%. Only Qwen2-VL 72B showed consistent performance above random guess (65.89%).
  • Future State Prediction: Models generally perform poorly, often near random guess (50%), with human performance at 97%. Performance is better on shorter action sequences.
  • Plan Assessment (Real Web): Many VLMs score below 50%, with better performance when options are semantically distinct rather than just shuffled.
  • Plan Assessment (Edge Case): Performance drops significantly, with most models below 50% and GUI-specific models achieving very low accuracy.
  • Correlation with Task Success: A strong correlation was observed between planning skill scores and overall task success rates, reinforcing the importance of evaluating these fine-grained skills.

Limitations and Future Work

  • Resource Constraints: The authors acknowledge limitations in evaluating proprietary models (e.g., GPT-40-mini) due to computational costs.
  • Data Reliance: The use of repurposed and synthetic data, while effective for controlled evaluation, may not fully capture the diversity of real-world web scenarios.
  • Future Directions: The authors suggest incorporating these fine-grained skills into the pre-training of VLMs, focusing on smaller, more deployable models. They also call for community efforts to develop more sophisticated training and evaluation paradigms for VLM planning capabilities.

Summary updated

BibTeX
@inproceedings{jandial2025fine,
  author = {Surgan Jandial and Yinong Oliver Wang and Andrea Bajcsy and Fernando De la Torre},
  title = {{On the Fine-Grained Planning Abilities of VLM Web Agents}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2025},
}
Figure