Evaluating Vision-Language Models as Evaluators in Path Planning

Mohamed Aghzal, Xiang Yue, Erion Plaku, Ziyu Yao

Abstract

While Vision-Language Models (VLMs) demonstrate high-level reasoning capabilities, their application in robotics is often hindered by spatial perception limitations. This paper introduces PATHEVAL, a benchmark designed to evaluate VLMs as critics in path planning. Our findings reveal a significant "vision bottleneck": models struggle to perceive low-level geometric details from images alone, often performing near random unless provided with explicit textual path descriptors. We analyze the performance of state-of-the-art models like GPT-4o and Qwen2-VL, highlighting the discrepancy between their semantic reasoning and visual grounding.

Introduction

Large Language Models (LLMs) and VLMs have shown promise in understanding complex instructions, yet they struggle with end-to-end motion planning, such as generating precise coordinate sequences for navigation. A potential alternative is using these models as plan evaluators or "critics." In this role, a VLM would select the optimal path from candidates generated by traditional algorithms (e.g., RRT), leveraging its pre-trained knowledge to match path characteristics with specific scenario requirements (e.g., a heavy truck requiring wide turns).

However, effective evaluation requires three distinct capabilities: abstracting scenario requirements, perceiving geometric properties from visual input, and integrating this information for decision-making. This study investigates whether current VLMs possess the requisite spatial reasoning or if they rely on "hallucinated" interpretations of visual data.

The PATHEVAL Benchmark

To systematically evaluate these capabilities, we developed PATHEVAL, a controllable benchmark comprising 14,550 path-pair comparison tasks across 1,150 environments. Paths are generated using the RRT-Connect algorithm and visualized in both 2D top-down and 3D perspective views.

Comparison of 2D and 3D path visualizations in the PATHEVAL benchmark.

The benchmark decomposes the evaluation task into three levels:

  1. Attribute Abstraction: Identifying relevant metrics (e.g., smoothness, clearance) for a given scenario.
  2. Low-level Perception: Distinguishing geometric differences between paths from visual renders.
  3. Information Integration: Combining scenario knowledge with visual observations to select the superior path.

Each path is characterized by seven ground-truth metrics, including minimum/average clearance, path length, smoothness, and sharp turn counts.

Results and Analysis

Our evaluation of nine state-of-the-art models, including GPT-4o and Qwen2-VL, yielded several key insights:

The Vision Bottleneck

Most models performed near random (approx. 50% accuracy) when relying solely on visual input. However, performance improved dramatically when ground-truth metrics were provided as text. For instance, Qwen2-VL's accuracy increased from 50.2% to 73.1% with textual descriptors. This suggests that while models understand the logic of the evaluation task, they cannot accurately see the paths.

Hallucinated Reasoning and Encoder Ambiguity

Models frequently generated confident but incorrect justifications, claiming a path had fewer turns when the image showed the opposite. Probing experiments on vision encoders (CLIP, SigLip) revealed that they produce nearly identical feature vectors for distinct paths in the same environment, rendering them mathematically indistinguishable to the model.

2D vs. 3D Modalities

Contrary to expectations, 3D renders did not improve performance. GPT-4o was 4% more accurate on 2D images than 3D, likely because perspective shifts in 3D introduce visual complexities that further confuse spatial reasoning.

Conclusion

The "vision bottleneck" remains a primary obstacle for VLMs in robotics. Current general-purpose vision encoders, trained on natural images, lack the precision required for geometric path analysis. Future progress requires task-specific discriminative adaptation of vision encoders or architectural innovations like "chain-of-visual-thought" to bridge the gap between high-level reasoning and low-level perception.

Summary updated

BibTeX
@inproceedings{aghzal2025evaluating,
  author = {Mohamed Aghzal and Xiang Yue and Erion Plaku and Ziyu Yao},
  title = {{Evaluating Vision-Language Models as Evaluators in Path Planning}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2025},
  url = {https://openreview.net/forum?id=SPdMNZtXBE},
}
Figure