Post-Hurricane Debris Segmentation Using Fine-Tuned Foundational Vision Models

Kooshan Amini, Yuhao Liu, Jamie E. Padgett, Guha Balakrishnan, Ashok Veeraraghavan

Abstract

Rapid debris removal is critical for disaster recovery, yet manual analysis of aerial imagery remains a significant bottleneck. This paper presents fCLIPSeg, a specialized model developed by researchers at Rice University that leverages foundational vision-language models for automated debris segmentation. By fine-tuning CLIPSeg on a curated dataset of 1,200 multi-annotated images across multiple hurricane events, the authors demonstrate high-accuracy, event-agnostic performance. The model achieves a Dice score of 0.70 on unseen disaster regions, showcasing its potential for rapid, scalable deployment in emergency response scenarios.

Introduction

Following a hurricane, the immediate restoration of emergency services and road networks depends on the swift identification and removal of debris. While agencies like NOAA can capture high-resolution aerial imagery within days of an event, the manual interpretation of thousands of images creates a massive operational bottleneck.

Automating this process is technically challenging due to the "visually messy" nature of disaster zones. Debris is often fragmented and composed of diverse materials such as wood, metal, and shingles. Furthermore, its appearance is highly sensitive to environmental factors, including lighting conditions, camera angles, and background contrast (e.g., debris on sand versus grass). Traditional AI models often fail to generalize across different geographic regions or disaster events because the "visual signature" of the wreckage changes.

Visual challenges in debris detection, ranging from image quality and lighting to debris density and composition.

Methodology

Rather than training a model from scratch, the researchers utilized CLIPSeg, a foundational vision model pre-trained on vast amounts of image-text data. This approach allows the model to leverage a broad "common sense" understanding of visual concepts.

The team introduced several key technical innovations:

  • Decoder Fine-Tuning: The core CLIP encoders were kept frozen to preserve general knowledge, while only the decoder was trained to map high-level features to specific debris masks.
  • Visual Prompt Engineering: Training images were "engineered" by darkening or blurring backgrounds to highlight debris, helping the model focus on relevant features.
  • Consensus Annotation: To ensure high-quality ground truth, each image in the training set was labeled by multiple human annotators, with their inputs averaged to create a mathematical consensus.
  • Interpolated Embeddings: Data augmentation techniques blended text prompts (e.g., "debris at high density") with visual prompts to enhance model robustness.

Datasets and Experimental Setup

The study introduced a new open-source dataset comprising approximately 1,200 manually annotated aerial RGB images (50m x 50m crops). The data covers three major hurricanes:

  1. Hurricane Ian (2022): Estero Island, FL.
  2. Hurricane Ike (2008): Galveston Island, TX.
  3. Hurricane Ida (2021): Grand Isle, LA.

To test the model's ability to generalize to "unseen events," it was trained on data from Hurricanes Ian and Ike and then evaluated on Hurricane Ida.

Results

The results demonstrate that foundational models are remarkably "debris-ready" with minimal fine-tuning. Key performance metrics include:

  • Generalization: fCLIPSeg achieved a Dice score of 0.70 on the unseen Hurricane Ida dataset, significantly outperforming baseline models.
  • Precision: The model showed a dramatic improvement in precision for high-density debris (0.87) compared to the base CLIPSeg model (0.10).
  • Operational Efficiency: The model is computationally lightweight, capable of processing approximately 4,000 images (covering an entire island) in just 2 minutes and 19 seconds on a single modern GPU.
  • Low False Positives: The model demonstrated near-perfect performance in debris-free areas, a critical requirement for avoiding the misallocation of recovery resources.

Limitations and Future Work

Despite its success, fCLIPSeg faces challenges in detecting very small, isolated fragments or debris obscured by tree cover. It also occasionally confuses debris with similar-looking textures, such as certain types of vegetation.

Future research will focus on:

  • 3D Volume Estimation: Integrating LiDAR or stereo-photogrammetry to estimate debris volume, which is more useful for logistics than 2D footprints.
  • Global Scaling: Expanding the dataset to include international regions with different construction materials and environmental contexts.
  • Refining Detection: Improving the model's sensitivity to tiny fragments and its ability to distinguish between debris and natural mimicry.

Summary updated

Abstract

Timely and accurate detection of hurricane debris is critical for effective disaster response and community resilience. While post‐disaster aerial imagery is readily available, robust debris segmentation solutions applicable across multiple disaster regions remain limited. Since environmental and imaging conditions vary by region and alter debris's visual signatures, developing a debris segmentation model suitable for different hazard scenarios is challenging, especially given the scarcity of training data. This study addresses these challenges by relying on vast pretrained knowledge from the foundation vision model Contrastive Language‐Image Pre‐Training (CLIP) and fine‐tuning a lightweight decoder on labeled debris imagery, thereby achieving robust performance in segmenting non‐vegetative debris within complex urban backgrounds with a relatively small, high‐quality dataset. To enable fine‐tuning, this work introduces an open‐source dataset comprising approximately 1200 manually annotated aerial RGB images of structural debris from Hurricanes Ian, Ida, and Ike. To mitigate human biases and enhance data quality, labels from multiple annotators are strategically aggregated into consensus annotations. The resulting fine‐tuned model, named CLIPSeg‐debris, achieves a Dice score of 0.86 on data from Hurricane Ida—a disaster event entirely excluded during training—outperforming baseline models. This approach thereby enables broader adoption across multiple hazard scenarios and effectively adapts to various backgrounds, sensor resolutions, and debris conditions.
BibTeX
@misc{amini2025post,
  author = {Kooshan Amini and Yuhao Liu and Jamie E. Padgett and Guha Balakrishnan and Ashok Veeraraghavan},
  title = {{Post-Hurricane Debris Segmentation Using Fine-Tuned Foundational Vision Models}},
  year = {2025},
  eprint = {2504.12542},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2504.12542},
}
Figure