Pothole Risk Assessment with Geometrically Grounded Vision Language Models

Abdalmalek Aburaddaha, Zaid A. El Shair, Ali Al-Bustami, Samir A. RawashdeharXiv 2026 · arXiv

TL;DR

This paper introduces a five-stage hybrid framework that combines deterministic geometric computer vision with Vision-Language Models (VLMs) to assess pothole severity and generate natural language driving warnings. By grounding VLMs like Gemini 2.5 Pro in measurements such as area and wheel-path position, the system produces specific, actionable alerts that human raters preferred 3.5:1 over standard image-only baselines.

What Problem Does This Paper Address?

Potholes are a multi-billion dollar headache for infrastructure and vehicle safety. While modern computer vision can detect where a pothole is, it struggles to determine how severe it is or what a driver should do about it.

Judging depth from a single RGB image (monocular depth estimation) is an "ill-posed" problem—shadows and lighting can easily trick a model into seeing a shallow stain as a deep cavity. When researchers try to use Vision-Language Models (VLMs) for this, the models tend to be over-conservative, issuing generic "high danger" warnings for almost everything to be safe. This leads to "alarm fatigue," where drivers eventually ignore the system. The authors aim to bridge this gap by "grounding" the VLM's reasoning in actual geometric measurements.

Methods

The authors propose a modular, five-stage pipeline designed for offline infrastructure assessment:

  1. Perspective Transformation: Warps the raw dashcam footage to reduce foreshortening, making distant potholes easier to analyze and focusing the model's "attention" on the road surface.
  2. Zero-Shot Segmentation: Uses Meta’s Segment Anything Model (SAM) to delineate pothole boundaries without needing task-specific training.
  3. Geometric Feature Extraction: Extracts a six-dimensional feature vector (Area, Perimeter, Circularity, Aspect Ratio, Solidity, and Spatial Position) from the segmentation masks.
  4. VLM Severity Classification & Risk Scoring: Feeds the image and the geometric JSON data to Gemini 2.5 Pro. The VLM classifies the pothole (Low/Medium/High severity), and a composite risk score is calculated based on depth, size, and proximity to the vehicle's wheel path.
  5. Natural Language Warning Generation: The VLM generates a professional warning and a specific maneuver suggestion (e.g., "Reduce speed to 15 mph and steer slightly right").
The five-stage modular framework overview

Datasets

  • Nienaber Pothole Dataset: The primary source, containing 1,119 dashcam scenes and over 3,000 annotated potholes from South African roads.
  • ENPAD (Extended Nienaber Pothole Assessment Dataset): A new resource introduced by the authors, adding the pipeline's generated masks, geometric features, and VLM-based warnings to the original dataset.
  • PothRGBD: An independent dataset with actual physical depth measurements (in millimeters) used to validate whether the VLM's "perceived severity" correlates with reality.

Key Insights and Ideas

  • Geometric Anchoring: The VLM isn't treated as a standalone oracle. Instead, geometric features act as "anchors." By providing the model with the actual pixel area and position, it can no longer "speculate" wildly based on ambiguous visual cues like shadows.
  • Perceived vs. Physical Severity: A major finding was that both VLMs and humans are relatively poor at estimating exact metric depth from single images. The authors argue that VLM outputs should be treated as "perceived visual severity" categories rather than precise depth measurements.
  • Redistribution, Not Just Sensitivity: Grounding the model didn't just make it "quieter." It moved 26.7% of scenes out of the "high-urgency" tier (reducing false alarms) while simultaneously doubling the number of "critical" assignments for large, deep potholes directly in the wheel path.
Comparison of urgency distributions between image-only and hybrid methods

Key Results

  • Human Preference: In a blind study, human raters preferred the geometrically grounded warnings over image-only ones by a ratio of 3.5:1.
  • Quality Metrics: The hybrid approach showed significant gains in clarity (+0.83) and safety appropriateness (+0.73) on a 5-point Likert scale.
  • Rule Adherence: Ablation studies showed that the model's ability to follow the "decision rubric" came almost entirely from the per-pothole geometric measurements, not from the text of the rules themselves.
  • Cost-Effectiveness: The system operates at approximately $0.105 per scene in API fees, making it viable for large-scale offline infrastructure monitoring.

Limitations and Future Work

  • Latency: The current system takes 18–23 seconds per scene, dominated by API calls and heavy segmentation models. It is currently strictly an offline tool. Future work involves distilling these models for real-time, on-vehicle use.
  • Monocular Ambiguity: Despite the grounding, the lack of true 3D data (Stereo or LiDAR) remains a bottleneck for perfect depth assessment.
  • Contextual Blindness: The maneuver suggestions currently do not account for surrounding traffic or adjacent lane occupancy.
  • Generalization: The model was tested primarily on one geographic region; its performance on different road materials (like concrete) or in different climates (like snowy regions) remains to be seen.

Summary updated

Abstract

Potholes impose billions of dollars in annual vehicle damage, yet detectors report only where potholes are, not how severe they are or what to do. Judging severity from a single RGB image is ill-posed, and ungrounded vision-language model (VLM) assessments tend to be over-conservative. We present a five-stage modular framework for the offline assessment of already-detected potholes that combines geometric measurements with VLM reasoning: perspective transformation, zero-shot SAM segmentation, geometric feature extraction, Gemini 2.5 Pro severity classification with composite risk scoring, and natural-language warning generation. We evaluate it on 3,092 potholes in 1,119 dashcam scenes against independent references. Severity agreement with a prediction-free three-annotator panel is weak (Cohen's $κ= 0.139$, 95% CI [0.00, 0.28]; $n = 101$) and far below the annotators' agreement with one another ($κ= 0.605$); against physically measured depth on an independent RGB-D dataset, it is also weak ($κ= 0.098$), although the predicted classes preserve the order of measured depths. The severity classes are therefore perceived visual categories, not depth measurements. Geometric grounding does change the warnings: relative to an image-only baseline, it moves 26.7% of scenes out of the high-urgency tier and raises critical-tier assignments by 109% (Stuart-Maxwell $p < 10^{-15}$), and in a blind study, three raters preferred the grounded warnings 3.5:1. A six-arm prompt ablation attributes the hybrid prompt's rule adherence to the per-pothole measurements rather than to the rubric or worked examples. At USD 0.105 per scene, it suits large-scale offline infrastructure assessment; its maneuver suggestions are advisory research outputs, not validated vehicle guidance.
BibTeX
@misc{aburaddaha2026pothole,
  author = {Abdalmalek Aburaddaha and Zaid A. El Shair and Ali Al-Bustami and Samir A. Rawashdeh},
  title = {{Pothole Risk Assessment with Geometrically Grounded Vision Language Models}},
  year = {2026},
  eprint = {2611.16061},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.16061},
}
Figure