From Cloud to Edge: A Fully Local Pipeline for Pothole Risk Assessment and Driver Warning Generation

Abdalmalek Aburaddaha, Samir A. RawashdeharXiv 2026 · arXiv

TL;DR

Researchers have successfully migrated a complex pothole risk assessment pipeline from expensive, high-latency cloud APIs to a fully local system running on a single GPU. By combining optimized segmentation, distilled "student" models for severity grading, and grammar-constrained vision-language models (VLMs), the new pipeline is 16x faster, runs offline, and produces warnings that human raters actually judge to be safer than those generated by the cloud.

What Problem Does This Paper Address?

While detecting a pothole is a solved computer vision problem, assessing its risk and providing actionable advice to a driver is much harder. Previous state-of-the-art systems relied on commercial cloud-hosted VLMs (like Gemini 2.5 Pro) to interpret road geometry and write warnings.

However, cloud dependence is a dealbreaker for real-world automotive applications. A 20-second round-trip latency and a $0.10 per-scene cost make real-time warnings impossible. Furthermore, cloud models are "black boxes" that can change without notice and require a constant internet connection—something a vehicle in a remote area or a parking garage cannot guarantee. The authors sought to prove that a fully local "edge" pipeline could match or exceed cloud performance.

Methods

The authors rebuilt a five-stage pipeline to run locally:

  1. Perspective Correction: Standardizes the dashcam view so pothole sizes are comparable across the frame.
  2. Optimized Segmentation: Interestingly, the authors found that lightweight "Mobile" versions of the Segment Anything Model (SAM) failed to accurately capture small road defects. Instead, they kept the heavy SAM ViT-H weights but optimized them using fp16 half-precision and graph compilation (torch.compile), achieving a 2.7x speedup without losing mask fidelity.
  3. Geometric Feature Extraction: Measures area, circularity, and lane position.
  4. Knowledge Distillation for Severity: To replace the cloud model's grading, they trained an ensemble of six ConvNeXt-Tiny "student" models to mimic the cloud model’s labels. This reduced inference time from seconds to just 28 milliseconds.
  5. Constrained Advisory Generation: They used a local 8-billion parameter VLM (Qwen3-VL-8B) to write the final warning. Crucially, they used grammar-constrained decoding (via XGrammar) to force the model to output valid JSON and—more importantly—to restrict its advice to a closed vocabulary of safe, in-lane maneuvers (e.g., "reduce speed" rather than "swerve into the next lane").

Datasets

  • ENPAD-A (New): A dataset of 1,119 scenes and 3,097 potholes, each with perception records and structured driver action labels.
  • South African Road Dataset: The original source for the dashcam imagery.
  • PothRGBD: An independent dataset used to test how the distilled models handle different camera geometries (finding that distillation is highly camera-specific).
  • Human Consensus Set: A small, high-quality set of 38 potholes graded by multiple humans to provide a "gold standard" for severity.

Key Insights and Ideas

  • Safety through Constraint: The local model was rated safer than the cloud model not because it was "smarter," but because it was more restricted. The cloud model occasionally suggested risky maneuvers (like straddling a center line) that a single camera cannot justify. The local system’s "closed maneuver vocabulary" acted as a safety guardrail.
  • The "Reasoning-First" Prompting: In constrained JSON generation, the order of fields matters. By forcing the model to write its "reasoning" field before its "maneuver" field, the authors improved accuracy. This allows the model to process the visual evidence before committing to a final classification.
  • The LLM-as-a-Judge Failure: The study highlights a major pitfall in current AI research. An automated LLM judge (Gemini Flash) preferred the cloud warnings, while human raters preferred the local ones. The LLM judge likely rewarded "fluent, varied language," whereas humans prioritized "caution and concreteness."

Key Results

  • Speed: The end-to-end pipeline time dropped from 20.1 seconds to 1.27 seconds per scene (a 16x speedup).
  • Cost: Marginal cost per scene dropped from $0.105 to $0.
  • Human Evaluation: In a blind study, three independent raters gave the local warnings a safety score of 4.18/5, compared to 3.65/5 for the cloud system ($p=0.001$).
  • Severity Accuracy: The distilled ConvNeXt ensemble achieved a Cohen’s $\kappa$ of 0.268, very close to the cloud model’s 0.321, while being 170x faster.

Limitations and Future Work

The authors acknowledge that the distilled severity grader is camera-specific. If the camera height or angle changes significantly, the model needs to be re-distilled using a small batch of new cloud labels.

Future work aims to deploy this pipeline on embedded vehicle hardware (like NVIDIA Orin), integrate a real-time object detector at the start of the pipeline, and expand the "closed vocabulary" to include more nuanced free-text explanations while maintaining safety constraints.

Summary updated

Abstract

Vision-language models can turn road images into clear, geometry-aware pothole warnings, but current systems depend on commercial cloud services. Our earlier five-stage pipeline needed two cloud calls per scene, taking about 20 s and USD 0.105, and could not run offline. We rebuild every stage to run on a single local GPU and evaluate it with the same data, prompts, and protocol. Segmentation keeps the original Segment Anything weights but runs them in half precision with graph compilation, which makes it 2.7x faster without changing any downstream risk class; five lightweight variants could not match its masks on small road defects. Severity is graded by six ConvNeXt students distilled from 3092 cloud labels. They reach Cohen's kappa = 0.268 against human consensus at 28 ms per pothole, close to the cloud model's 0.321, within the camera setup they were trained for. A local 8-billion-parameter vision-language model writes the warnings under a grammar that admits only in-lane maneuvers. Three blind, independent raters scored these warnings safer than the cloud warnings (4.18 vs. 3.65 of 5; p = 0.001; 50 scenes) and equally clear, whereas an automated LLM judge ranked them the other way. The complete pipeline runs at 1.27 s per scene, 16x faster than the cloud system, offline and at no per-scene cost.
BibTeX
@misc{aburaddaha2026from,
  author = {Abdalmalek Aburaddaha and Samir A. Rawashdeh},
  title = {{From Cloud to Edge: A Fully Local Pipeline for Pothole Risk Assessment and Driver Warning Generation}},
  year = {2026},
  eprint = {2611.19737},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.19737},
}
Figure