On Optimizing Model Generality in AI-based Disaster Damage Assessment: A Subjective Logic-driven Crowd-AI Hybrid Learning Approach

Yang Zhang, Ruohan Zong, Lanyu Shang, Huimin Zeng, Zhenrui Yue, Na Wei, Dong Wang

Abstract

AI-based disaster damage assessment (ADA) models often suffer from a lack of generality, failing to accurately classify damage when deployed on disaster types outside their training distribution. This paper introduces CollabGeneral, a framework designed to bridge this "generality gap" by combining a generality-aware deep learning optimization strategy with crowdsourced human intelligence. Using Subjective Logic to manage uncertainty and fuse diverse inputs, CollabGeneral demonstrates superior robustness and accuracy across multiple real-world disaster datasets compared to traditional deep learning and domain adaptation baselines.

The Problem of Model Generality

Current AI solutions for disaster response are typically "event-specific." While a model might excel at identifying structural damage from an earthquake, it often fails when tasked with assessing flood or hurricane damage. This is due to "specificity"—the tendency of models to overfit to the visual features of a single disaster type.

Visual ambiguity in disaster imagery: Storm clouds in a hurricane can appear similar to smoke in wildfire events, leading to misclassification by specific models.

A primary challenge is visual ambiguity. For instance, the grey skies of a storm can look remarkably similar to the smoke plumes of a wildfire, yet they represent different disaster contexts. While humans can intuitively generalize the concept of "damage" across these contexts, they are slower and more expensive than AI. The goal of CollabGeneral is to combine the speed of AI with the broad reasoning capabilities of human intelligence.

The CollabGeneral Framework

The proposed framework consists of two core modules designed to optimize both machine learning and human-machine collaboration.

1. Generality-aware Deep Optimization (GDO)

Standard deep learning models optimize for accuracy on a specific training set. CollabGeneral introduces a dual-loss function to encourage broader applicability:

  • Accuracy-aware loss ($L_1$): Maintains high performance on the primary task.
  • Generality-aware loss ($L_2$): Penalizes the model if its performance fluctuates significantly across different disaster types.

The system uses a budget-constrained multi-armed bandit approach to explore various network architectures and parameters, finding the optimal balance between specificity and generality.

2. Subjective Logic-driven Crowd-AI Fusion (SCF)

To maintain cost-efficiency, CollabGeneral does not send every image to human workers. Instead, it identifies "high-divergence" images—those where different AI model instances disagree most (high Shannon entropy). These selected images are sent to crowdsourced workers via platforms like Amazon Mechanical Turk.

The fusion of AI and human labels is handled via Subjective Logic. Unlike binary logic, Subjective Logic explicitly models "opinions" through three dimensions: belief, disbelief, and uncertainty. This allows the system to weigh the reliability of each source (AI or human) and the discriminative difficulty of each image, ensuring that high-confidence, correct insights prevail over uncertain or erroneous ones.

Experimental Setup and Datasets

The framework was evaluated using four real-world datasets from CrisisNLP:

  • Hurricane Irma (2017): Wind and coastal damage.
  • Ecuador Earthquake (2016): Urban structural damage.
  • Nepal Earthquake (2015): Rural structural damage.
  • Sri Lanka Flooding (2017): Water-based damage.

Images were classified into three severity levels: severe, medium, and no/minor damage.

Key Results

CollabGeneral consistently outperformed state-of-the-art baselines, including standard DNNs (ResNet, DenseNet, VGG) and specialized domain adaptation models (GTA, VS).

Performance comparison of CollabGeneral against various baselines across Ecuador Earthquake, Sri Lanka Flooding, and Overall metrics.

Key findings include:

  • Superior Generalization: On the Sri Lanka Flooding dataset, CollabGeneral improved the F1-Score by 5.22% and the Kappa-Score by 8.41% over the best baseline.
  • Robustness: The performance gap between CollabGeneral and standard models widened as the number of disaster types increased, proving its scalability.
  • Overall Accuracy: The framework achieved an overall F1-score of 0.8436, significantly higher than the 0.7326 achieved by a standard ResNet-A baseline.

Conclusion and Future Work

CollabGeneral demonstrates that integrating generality-aware optimization with uncertainty-driven human-AI collaboration can significantly improve disaster damage assessment. While crowdsourcing introduces latency and cost, the framework's active sampling strategy minimizes these impacts. Future research may extend this approach to other domains facing domain-shift challenges, such as misinformation detection and intelligent transportation systems.

Summary updated

Abstract

This paper focuses on the AI-based damage assessment (ADA) applications that leverage state-of-the-art AI techniques to automatically assess the disaster damage severity using online social media imagery data, which aligns well with the ''disaster risk reduction'' target under United Nations' Sustainable Development Goals (UN SDGs). This paper studies an ADA model generality problem where the objective is to address the limitation of current ADA solutions that are often optimized only for a single disaster event and lack the generality to provide accurate performance across different disaster events. To address this limitation, we work with domain experts and local community stakeholders in disaster response to develop CollabGeneral, a subjective logic-driven crowd-AI collaborative learning framework that integrates AI and crowdsourced human intelligence into a principled learning framework to address the ADA model generality problem. Extensive experiments on four real-world ADA datasets demonstrate that CollabGeneral consistently outperforms the state-of-the-art baselines by significantly improving the ADA model generality across different disasters.
BibTeX
@inproceedings{zhang2023optimizing,
  author = {Yang Zhang and Ruohan Zong and Lanyu Shang and Huimin Zeng and Zhenrui Yue and Na Wei and Dong Wang},
  title = {{On Optimizing Model Generality in AI-based Disaster Damage Assessment: A Subjective Logic-driven Crowd-AI Hybrid Learning Approach}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2023},
  doi = {10.24963/ijcai.2023/701},
}
Figure