Can Foundation Models Reliably Identify Spatial Hazards? A Case Study on Curb Segmentation

Diwei Sheng, Giles Hamilton-Fletcher, Mahya Beheshti, Chen Feng, John-Ross RizzoarXiv 2024 · DOI · arXiv

Abstract

For pedestrians with blindness and low vision (PBLV), curbs are essential yet hazardous urban features. While foundation models have demonstrated remarkable capabilities in general computer vision tasks, their reliability in high-stakes spatial hazard identification remains under-explored. This paper evaluates state-of-the-art foundation models on a novel, pedestrian-centric curb segmentation dataset. We introduce a "Modified-lang-seg" approach that improves accuracy by filtering overlapping detections. Our findings reveal that while foundation models offer high sensitivity, they suffer from significant false-positive rates and prohibitive latency, highlighting a critical gap between current AI capabilities and the real-time requirements of assistive technology.

Introduction

In the context of independent mobility for the PBLV community, the curb is a dual-purpose landmark: it defines the boundary between safe pedestrian zones and vehicular traffic, but it also represents a primary tripping hazard. Effective assistive technology must go beyond simple object detection; it requires precise curb segmentation to identify the exact pixel-level geometry and orientation of these features.

Recent advancements in "foundation models"—AI systems trained on massive, multi-modal datasets—suggest a potential solution for niche spatial tasks. However, these models are often evaluated on general-purpose or driver-centric datasets (e.g., BDD100K), which do not reflect the unique perspective of a pedestrian. This study benchmarks leading foundation models to determine if they can reliably serve as the backbone for assistive navigation systems.

Methodology

The research evaluates three primary foundation model architectures, each representing a different approach to open-vocabulary segmentation:

  1. LISA (Large Language Instructed Segmentation Assistant): A model that leverages a Large Language Model (LLM) to perform "reasoning segmentation" based on complex text prompts.
  2. Lang-seg: A combinatorial pipeline that utilizes GroundingDINO for text-based object detection and the Segment Anything Model (SAM) to generate precise masks for those detections.
  3. OVSEG (Open-Vocabulary Semantic Segmentation): A model based on CLIP that performs pixel-level classification across a wide range of categories.

The Modified-lang-seg Approach

During initial testing, the researchers observed that standard Lang-seg often generated multiple, overlapping bounding boxes for a single curb, leading to fragmented or redundant segmentation masks. To address this, they developed Modified-lang-seg. This version introduces a confidence-based filtering step: when multiple bounding boxes overlap, the system retains only the box with the highest confidence score before passing the data to SAM. This simple refinement aims to reduce noise and improve the precision of the final output.

The Curb Segmentation Dataset

A significant contribution of this work is the introduction of a new, high-quality Curb Segmentation Dataset specifically designed for pedestrian safety. Unlike existing datasets that focus on the view from a vehicle's dashboard, this dataset features:

  • Pedestrian Perspective: 2,133 images captured at chest level in New York City.
  • Balanced Composition: 1,620 "positive" images containing curbs at various angles and 513 "negative" images featuring curb-like distractors (e.g., windowsills, cafe barriers, and sidewalk cracks).
  • Precise Annotation: Manual polygonal annotations provide a ground truth for exact shape evaluation, moving beyond the limitations of rough bounding boxes.

Results and Discussion

Segmentation Performance

The Modified-lang-seg approach emerged as the top performer, achieving a Mean Intersection over Union (mIoU) of 40.3%. This significantly outperformed the base Lang-seg (36.0%) and LISA (26.6%). The modification also improved precision to 0.7035, demonstrating that intelligent filtering is essential for reducing the "over-segmentation" common in generalist models.

The "Generalist" Trap

While models like OVSEG showed high sensitivity, they suffered from extremely low specificity. These models tend to over-segment, often mistaking the edges of buildings or sidewalks for curbs. In a safety-critical application, a high rate of false positives (hallucinating hazards) can make a device frustrating and unusable for a pedestrian.

The Latency Gap

Perhaps the most significant finding was the computational overhead. The fastest model, OVSEG, required 3.7 seconds per frame, while LISA took as long as 10.85 seconds. For a pedestrian walking at a normal pace, a multi-second delay in hazard detection is unacceptable. Even Modified-lang-seg, despite its accuracy, required 4.77 seconds per image, highlighting that these models are currently too slow for real-time deployment on wearable hardware.

Conclusion

Foundation models show promise in identifying complex spatial hazards like curbs, but they are not yet "plug-and-play" solutions for assistive technology. The high latency and tendency toward false positives remain significant hurdles.

Future research should focus on "Combinatorial AI": pairing the high-level reasoning of foundation models with lightweight, mobile-optimized architectures. By using the pedestrian-centric dataset provided in this study to fine-tune efficient models (such as YOLOv8-nano or MobileSAM), the gap between laboratory performance and real-world utility can finally be bridged, moving us closer to reliable, real-time safety tools for the PBLV community.

Summary updated

BibTeX
@misc{sheng2024can,
  author = {Diwei Sheng and Giles Hamilton-Fletcher and Mahya Beheshti and Chen Feng and John-Ross Rizzo},
  title = {{Can Foundation Models Reliably Identify Spatial Hazards? A Case Study on Curb Segmentation}},
  year = {2024},
  eprint = {2406.07202},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2406.07202},
}
Figure