Abstract
Despite their promise to perform complex reasoning, large language models (LLMs) have been shown to have limited effectiveness in end-to-end planning. This has inspired an intriguing question: if these models cannot plan well, can they still contribute to the planning framework as a helpful plan evaluator? In this work, we generalize this question to consider LLMs augmented with visual understanding, i.e., Vision-Language Models (VLMs). We introduce PathEval, a novel benchmark evaluating VLMs as plan evaluators in complex path-planning scenarios. Succeeding in the benchmark requires a VLM to be able to abstract traits of optimal paths from the scenario description, demonstrate precise low-level perception on each path, and integrate this information to decide the better path. Our analysis of state-of-the-art VLMs reveals that these models face significant challenges on the benchmark. We observe that the VLMs can precisely abstract given scenarios to identify the desired traits and exhibit mixed performance in integrating the provided information. Yet, their vision component presents a critical bottleneck, with models struggling to perceive low-level details about a path. Our experimental results show that this issue cannot be trivially addressed via end-to-end fine-tuning; rather, task-specific discriminative adaptation of these vision encoders is needed for these VLMs to become effective path evaluators.
Introduction
Robotic path planning typically involves a dual-stage process: generating a candidate path and evaluating it against specific constraints. While traditional algorithms like RRT-Connect excel at generating valid trajectories, defining evaluation criteria for complex, commonsense scenarios—such as a firefighting robot prioritizing coverage or a heavy truck avoiding sharp turns—remains a significant challenge.
Researchers have hypothesized that Vision-Language Models (VLMs) could serve as "critics" or reward models, leveraging their pre-trained knowledge to evaluate paths described in natural language. However, the visual precision required to distinguish between two similar paths in a complex environment is immense. This paper investigates whether state-of-the-art (SOTA) VLMs can bridge the gap between high-level semantic reasoning and low-level geometric perception.
The PathEval Benchmark
To rigorously test these capabilities, we developed the PathEval Benchmark, which consists of 14,550 path pairs across 1,150 distinct environments. The benchmark includes 15 diverse decision-making contexts, ranging from warzone navigation to heavy traffic management.
Each path pair is rendered in both 2D top-down and 3D perspective views. Paths are mapped to seven ground-truth descriptors, including clearance (min/max/avg), path length, smoothness, and the frequency of sharp turns. Success in PathEval requires a model to perform three distinct tasks: Attribute Abstraction (identifying relevant metrics for a text-based scenario), Low-level Perception (comparing metrics between paths in an image), and Information Integration (combining scenario needs with visual observations to select the optimal path).
Results and the "Vision Bottleneck"
Our evaluation of 9 SOTA models, including GPT-4o, LLaVA-NeXT, and Qwen2-VL, reveals a stark contrast between reasoning and perception.
Reasoning vs. Perception
VLMs are remarkably proficient at the linguistic component of the task. For instance, models correctly identified that a "large truck" requires fewer sharp turns over 92% of the time. However, performance plummeted when the models were required to extract these metrics from images. Most open-source VLMs performed near the 50% random baseline on the core benchmark. Even the top performer, GPT-4o, achieved only 66.5% accuracy on 2D images.
The Verbalization Jump
A key finding is that when models are provided with textual ground-truth descriptors (e.g., "Path 1 has 4 sharp turns, Path 2 has 1"), their accuracy skyrockets. GPT-4o’s performance jumped to nearly 89%, and Qwen2-VL improved from ~50% to 73%. This confirms that the primary failure mode is not logical reasoning, but a fundamental inability to "see" low-level geometric details.
Encoder Blindness
Probing experiments on standard vision encoders (like CLIP) showed that feature vectors for two distinct paths are often nearly identical, with cosine similarities exceeding 0.9. To these encoders, the paths are visually indistinguishable, leading the language head to generate "hallucinated justifications"—plausible-sounding but factually incorrect explanations for their choices.
Conclusion
The PathEval benchmark demonstrates that current general-purpose vision encoders are not yet equipped for the high-precision discriminative tasks required in robotics. Simply fine-tuning VLMs end-to-end does not resolve this perception gap. Future work must focus on developing or adapting vision backbones specifically for geometric and spatial discrimination to enable VLMs to function as effective path evaluators.
Summary updated