Abstract
AI-based disaster damage assessment (ADA) models often suffer from a lack of generality, failing to accurately classify damage when deployed on disaster types outside their training distribution. This paper introduces CollabGeneral, a framework designed to bridge this "generality gap" by combining a generality-aware deep learning optimization strategy with crowdsourced human intelligence. Using Subjective Logic to manage uncertainty and fuse diverse inputs, CollabGeneral demonstrates superior robustness and accuracy across multiple real-world disaster datasets compared to traditional deep learning and domain adaptation baselines.
The Problem of Model Generality
Current AI solutions for disaster response are typically "event-specific." While a model might excel at identifying structural damage from an earthquake, it often fails when tasked with assessing flood or hurricane damage. This is due to "specificity"—the tendency of models to overfit to the visual features of a single disaster type.
A primary challenge is visual ambiguity. For instance, the grey skies of a storm can look remarkably similar to the smoke plumes of a wildfire, yet they represent different disaster contexts. While humans can intuitively generalize the concept of "damage" across these contexts, they are slower and more expensive than AI. The goal of CollabGeneral is to combine the speed of AI with the broad reasoning capabilities of human intelligence.
The CollabGeneral Framework
The proposed framework consists of two core modules designed to optimize both machine learning and human-machine collaboration.
1. Generality-aware Deep Optimization (GDO)
Standard deep learning models optimize for accuracy on a specific training set. CollabGeneral introduces a dual-loss function to encourage broader applicability:
- Accuracy-aware loss ($L_1$): Maintains high performance on the primary task.
- Generality-aware loss ($L_2$): Penalizes the model if its performance fluctuates significantly across different disaster types.
The system uses a budget-constrained multi-armed bandit approach to explore various network architectures and parameters, finding the optimal balance between specificity and generality.
2. Subjective Logic-driven Crowd-AI Fusion (SCF)
To maintain cost-efficiency, CollabGeneral does not send every image to human workers. Instead, it identifies "high-divergence" images—those where different AI model instances disagree most (high Shannon entropy). These selected images are sent to crowdsourced workers via platforms like Amazon Mechanical Turk.
The fusion of AI and human labels is handled via Subjective Logic. Unlike binary logic, Subjective Logic explicitly models "opinions" through three dimensions: belief, disbelief, and uncertainty. This allows the system to weigh the reliability of each source (AI or human) and the discriminative difficulty of each image, ensuring that high-confidence, correct insights prevail over uncertain or erroneous ones.
Experimental Setup and Datasets
The framework was evaluated using four real-world datasets from CrisisNLP:
- Hurricane Irma (2017): Wind and coastal damage.
- Ecuador Earthquake (2016): Urban structural damage.
- Nepal Earthquake (2015): Rural structural damage.
- Sri Lanka Flooding (2017): Water-based damage.
Images were classified into three severity levels: severe, medium, and no/minor damage.
Key Results
CollabGeneral consistently outperformed state-of-the-art baselines, including standard DNNs (ResNet, DenseNet, VGG) and specialized domain adaptation models (GTA, VS).
Key findings include:
- Superior Generalization: On the Sri Lanka Flooding dataset, CollabGeneral improved the F1-Score by 5.22% and the Kappa-Score by 8.41% over the best baseline.
- Robustness: The performance gap between CollabGeneral and standard models widened as the number of disaster types increased, proving its scalability.
- Overall Accuracy: The framework achieved an overall F1-score of 0.8436, significantly higher than the 0.7326 achieved by a standard ResNet-A baseline.
Conclusion and Future Work
CollabGeneral demonstrates that integrating generality-aware optimization with uncertainty-driven human-AI collaboration can significantly improve disaster damage assessment. While crowdsourcing introduces latency and cost, the framework's active sampling strategy minimizes these impacts. Future research may extend this approach to other domains facing domain-shift challenges, such as misinformation detection and intelligent transportation systems.
Summary updated