RoadBench: Benchmarking MLLMs on Fine-Grained Spatial Understanding and Reasoning under Urban Road Scenarios

Jun Zhang, Jie Feng, Long Chen, Junhui Wang, Zhicheng Liu, Depeng Jin, Yong Li

Abstract

Multimodal large language models (MLLMs) have demonstrated powerful capabilities in general spatial understanding and reasoning. However, their fine-grained spatial understanding and reasoning capabilities in complex urban scenarios have not received significant attention. To fill this gap, we focus primarily on road markings as a typical example of fine-grained spatial elements, given the essential role of the integrated road traffic network they form within cities. We propose RoadBench, a systematic benchmark that comprehensively evaluates MLLMs' fine-grained spatial understanding and reasoning capabilities using Bird's-Eye View (BEV) and First-Person View (FPV) image inputs. This benchmark comprises six tasks consisting of 9,121 strictly manually verified test cases. These tasks form a systematic evaluation framework that bridges local spatial understanding to global reasoning. After evaluating 14 mainstream MLLMs, we confirm that RoadBench is a challenging benchmark; in certain tasks, model performance falls short of simple rule-based or random selection baselines. These findings contribute to the advancement of spatial understanding capabilities for MLLMs.

The Challenge of Fine-Grained Spatial Reasoning

While MLLMs have become impressively good at describing general scenes, they often stumble when asked for precision. In urban navigation and autonomous driving, "general" descriptions are insufficient. Precision is required to determine exactly how many lanes are available, which ones allow specific maneuvers like U-turns, and whether a vehicle is on a main highway or a service road.

Comparison of Bird's-Eye View (BEV) and First-Person View (FPV) road marking analysis.

Existing benchmarks mostly focus on whole-image understanding or identifying isolated objects. There is a significant gap in evaluating fine-grained spatial reasoning—the ability to understand thin, elongated structures like road markings that follow complex traffic laws. RoadBench fills this gap by focusing on the topological and regulatory logic of urban roads.

The RoadBench Evaluation Framework

The authors developed a systematic evaluation framework consisting of six distinct tasks across two perspectives: Bird’s-Eye View (BEV) from satellite imagery and First-Person View (FPV) from in-vehicle cameras. The tasks are designed hierarchically, moving from local perception to global reasoning.

Overview of the six evaluation tasks in RoadBench across different spatial scopes and viewpoints.

Core Tasks:

  1. Lane Counting (BEV & FPV): Determining the number of lanes in a specific direction.
  2. Lane Designation Recognition (BEV & FPV): Identifying if a lane is for straight travel, left turns, or U-turns.
  3. Road Network Correction (BEV): Redrawing "noisy" map lines to match actual road geometry seen in satellite images.
  4. Road Type Classification (FPV): Distinguishing between main roads and service roads based on environmental cues.

Dataset and Methodology

The RoadBench dataset contains 9,121 manually verified test cases. Data was aggregated from OpenStreetMap (OSM), Google Maps satellite imagery, and crowd-sourced in-vehicle camera databases across multiple major cities. Unlike many automated datasets, every label in RoadBench underwent rigorous human verification to ensure ground-truth accuracy. Additionally, all sensitive information, such as faces and license plates, was manually masked.

The researchers also experimented with different visual prompting methods, such as using red arrows versus color-coded lines, to determine how the visual presentation of the prompt affects model performance.

Key Insights and Experimental Results

The evaluation of 14 mainstream models, including the GPT-5 series, Gemini 2.5, Qwen2.5-VL, and LLaMA-3.2-Vision, revealed several critical shortcomings in current MLLM capabilities:

  • The "Coordinate Crisis": MLLMs struggle with precise numerical coordinates. In the Road Network Correction task, models failed to pinpoint exact junction locations, leading to unstable map outputs.
  • Hallucination vs. Vision: Models often relied on "typical" road layouts learned during training rather than the actual image provided. For example, a model might hallucinate a left-turn lane based on common road patterns even if the markings are absent in the image.
  • Perspective Bias: Models generally performed better on FPV (driver's view) than BEV (satellite view), likely reflecting a bias in training data which contains more dashcam-style photos than specialized satellite imagery.
  • The Baseline Barrier: In the BEV Lane Designation task, the top-performing model (GPT-5) achieved 58.6% accuracy, which was lower than a simple rule-based baseline (60.5%) that assumes the leftmost lane is always for left turns.

Conclusion and Future Directions

RoadBench demonstrates that even state-of-the-art MLLMs struggle with the "fine print" of urban streets. While GPT-5-Mini emerged as a leader, its F1-score of 0.369 in BEV Lane Counting highlights that the task is far from solved. Future work will focus on improving coordinate precision for map-making applications, expanding the benchmark to include adverse weather conditions that obscure road markings, and developing models that better integrate expert traffic knowledge with raw visual perception.

Summary updated

Abstract

Multimodal large language models (MLLMs) have demonstrated powerful capabilities in general spatial understanding and reasoning. However, their fine-grained spatial understanding and reasoning capabilities in complex urban scenarios have not received significant attention in the fields of both research and industry. To fill this gap, we focus primarily on road markings as a typical example of fine-grained spatial elements under urban scenarios, given the essential role of the integrated road traffic network they form within cities. Around road markings and urban traffic systems, we propose RoadBench, a systematic benchmark that comprehensively evaluates MLLMs' fine-grained spatial understanding and reasoning capabilities using BEV and FPV image inputs. This benchmark comprises six tasks consisting of 9,121 strictly manually verified test cases. These tasks form a systematic evaluation framework that bridges understanding at local spatial scopes to global reasoning. They not only test MLLMs' capabilities in recognition, joint understanding, and reasoning but also assess their ability to integrate image information with domain knowledge. After evaluating 14 mainstream MLLMs, we confirm that RoadBench is a challenging benchmark for MLLMs while revealing significant shortcomings in existing MLLMs' fine-grained spatial understanding and reasoning capabilities within urban scenarios. In certain tasks, their performance even falls short of simple rule-based or random selection baselines. These findings, along with RoadBench itself, will contribute to the comprehensive advancement of spatial understanding capabilities for MLLMs. The benchmark code, example datasets, and raw evaluation results are available in the supplementary material.
BibTeX
@misc{zhang2025roadbench,
  author = {Jun Zhang and Jie Feng and Long Chen and Junhui Wang and Zhicheng Liu and Depeng Jin and Yong Li},
  title = {{RoadBench: Benchmarking MLLMs on Fine-Grained Spatial Understanding and Reasoning under Urban Road Scenarios}},
  year = {2025},
  eprint = {2511.18011},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2511.18011},
}
Figure