Abstract
Timely and accurate detection of hurricane-induced debris is critical for effective disaster response and enhanced community resilience. While post-disaster aerial imagery is widely available, robust debris segmentation methods applicable across diverse disaster regions remain limited. Developing a generalized solution is challenging due to variations in environmental and imaging conditions that alter the visual signatures of debris, further complicated by the scarcity of high-quality labeled training data. This study addresses these challenges by fine-tuning pre-trained foundational vision models, achieving robust performance despite using a relatively small yet high-quality dataset. Specifically, this work introduces an open-source dataset comprising approximately 1,200 manually annotated aerial RGB images collected after Hurricanes Ian, Ida, and Ike. To mitigate human biases and enhance labeling quality, annotations from multiple experts are strategically aggregated, complemented by visual prompt engineering. The resulting fine-tuned model, named fCLIPSeg, achieves a Dice score of 0.70 on imagery from Hurricane Ida—a disaster event completely excluded from training—with virtually no false positives in debris-free regions. This research provides the first event-agnostic debris segmentation model deployable with standard RGB imagery, making it particularly well-suited for rapid, large-scale post-disaster impact assessments and recovery planning.
Introduction
Hurricane-induced debris poses a significant threat to life safety and economic recovery, blocking emergency routes and accounting for approximately 36% of total recovery costs. While high-resolution aerial imagery is often provided by agencies like NOAA immediately following a storm, the manual analysis required to identify debris fields creates a critical bottleneck in disaster response.
Automating this process is difficult because debris is visually heterogeneous. Its appearance varies significantly based on lighting, camera resolution, and the underlying terrain (e.g., sand vs. pavement). Most existing models are "event-specific," failing when applied to new geographic regions or different storm events. This research introduces fCLIPSeg, a generalized, event-agnostic solution for debris segmentation using standard RGB imagery.
Methodology
The proposed framework leverages CLIPSeg, a foundational image segmentation model that utilizes Contrastive Language-Image Pre-training (CLIP) encoders. By fine-tuning only the transformer-based decoder while keeping the large CLIP encoders frozen, the model learns to specialize in debris textures without requiring massive amounts of training data.
A key innovation in this study is Visual Prompt Engineering. To accelerate learning and improve focus on debris signatures, training images were modified by blurring and darkening the background (dimming). This technique forces the model to prioritize the visual features of the debris itself rather than the surrounding environment.
Additionally, the team employed Consensus Annotation, where labels from multiple experts were aggregated via per-pixel averaging to create a robust ground truth, and Embedding Interpolation to enhance model robustness through data augmentation.
Datasets and Evaluation
The model was trained and validated using a new open-source dataset of approximately 1,200 manually annotated aerial RGB images (50x50m ground resolution). The dataset includes imagery from:
- Hurricane Ian (2022): High-resolution data from Florida.
- Hurricane Ike (2008): Lower-resolution data from older sensors in Texas.
- Hurricane Ida (2021): Held out entirely as an independent test set to evaluate event-agnostic generalization.
Results
fCLIPSeg demonstrated significant improvements over baseline models:
- Generalization: The model achieved a Dice score of 0.70 on the unseen Hurricane Ida test set, compared to 0.30 for the baseline CLIPSeg.
- Reliability: In debris-free areas, the model achieved a 0.99 Dice score, indicating a near-zero false positive rate.
- Precision: High-density debris piles were identified with a precision of 0.87.
- Efficiency: The model processes approximately 9.6 images per second. In a real-world test, it segmented the 10 km² area of Estero Island, FL, in just 2 minutes and 19 seconds.
Discussion and Conclusion
This study demonstrates that foundational vision models can be effectively specialized for disaster response tasks with relatively small, high-quality datasets. By focusing on event-agnostic features, fCLIPSeg provides a scalable tool for rapid post-disaster assessment. Future work will focus on incorporating 3D data (LiDAR) to estimate debris volume, expanding the dataset to include international geographic regions, and developing predictive models to estimate debris distribution before landfall.
Summary updated