Abstract
Multimodal large language models (MLLMs) have demonstrated powerful capabilities in general spatial understanding and reasoning. However, their fine-grained spatial understanding and reasoning capabilities in complex urban scenarios have not received significant attention. To fill this gap, we focus primarily on road markings as a typical example of fine-grained spatial elements, given the essential role of the integrated road traffic network they form within cities. We propose RoadBench, a systematic benchmark that comprehensively evaluates MLLMs' fine-grained spatial understanding and reasoning capabilities using Bird's-Eye View (BEV) and First-Person View (FPV) image inputs. This benchmark comprises six tasks consisting of 9,121 strictly manually verified test cases. These tasks form a systematic evaluation framework that bridges local spatial understanding to global reasoning. After evaluating 14 mainstream MLLMs, we confirm that RoadBench is a challenging benchmark; in certain tasks, model performance falls short of simple rule-based or random selection baselines. These findings contribute to the advancement of spatial understanding capabilities for MLLMs.
The Challenge of Fine-Grained Spatial Reasoning
While MLLMs have become impressively good at describing general scenes, they often stumble when asked for precision. In urban navigation and autonomous driving, "general" descriptions are insufficient. Precision is required to determine exactly how many lanes are available, which ones allow specific maneuvers like U-turns, and whether a vehicle is on a main highway or a service road.
Existing benchmarks mostly focus on whole-image understanding or identifying isolated objects. There is a significant gap in evaluating fine-grained spatial reasoning—the ability to understand thin, elongated structures like road markings that follow complex traffic laws. RoadBench fills this gap by focusing on the topological and regulatory logic of urban roads.
The RoadBench Evaluation Framework
The authors developed a systematic evaluation framework consisting of six distinct tasks across two perspectives: Bird’s-Eye View (BEV) from satellite imagery and First-Person View (FPV) from in-vehicle cameras. The tasks are designed hierarchically, moving from local perception to global reasoning.
Core Tasks:
- Lane Counting (BEV & FPV): Determining the number of lanes in a specific direction.
- Lane Designation Recognition (BEV & FPV): Identifying if a lane is for straight travel, left turns, or U-turns.
- Road Network Correction (BEV): Redrawing "noisy" map lines to match actual road geometry seen in satellite images.
- Road Type Classification (FPV): Distinguishing between main roads and service roads based on environmental cues.
Dataset and Methodology
The RoadBench dataset contains 9,121 manually verified test cases. Data was aggregated from OpenStreetMap (OSM), Google Maps satellite imagery, and crowd-sourced in-vehicle camera databases across multiple major cities. Unlike many automated datasets, every label in RoadBench underwent rigorous human verification to ensure ground-truth accuracy. Additionally, all sensitive information, such as faces and license plates, was manually masked.
The researchers also experimented with different visual prompting methods, such as using red arrows versus color-coded lines, to determine how the visual presentation of the prompt affects model performance.
Key Insights and Experimental Results
The evaluation of 14 mainstream models, including the GPT-5 series, Gemini 2.5, Qwen2.5-VL, and LLaMA-3.2-Vision, revealed several critical shortcomings in current MLLM capabilities:
- The "Coordinate Crisis": MLLMs struggle with precise numerical coordinates. In the Road Network Correction task, models failed to pinpoint exact junction locations, leading to unstable map outputs.
- Hallucination vs. Vision: Models often relied on "typical" road layouts learned during training rather than the actual image provided. For example, a model might hallucinate a left-turn lane based on common road patterns even if the markings are absent in the image.
- Perspective Bias: Models generally performed better on FPV (driver's view) than BEV (satellite view), likely reflecting a bias in training data which contains more dashcam-style photos than specialized satellite imagery.
- The Baseline Barrier: In the BEV Lane Designation task, the top-performing model (GPT-5) achieved 58.6% accuracy, which was lower than a simple rule-based baseline (60.5%) that assumes the leftmost lane is always for left turns.
Conclusion and Future Directions
RoadBench demonstrates that even state-of-the-art MLLMs struggle with the "fine print" of urban streets. While GPT-5-Mini emerged as a leader, its F1-score of 0.369 in BEV Lane Counting highlights that the task is far from solved. Future work will focus on improving coordinate precision for map-making applications, expanding the benchmark to include adverse weather conditions that obscure road markings, and developing models that better integrate expert traffic knowledge with raw visual perception.
Summary updated