Calibrated Semi-Supervised Models for Disaster Response based on Training Dynamics

Khushboo Gupta, Nikita Gautam, Tiberiu Sosea, Doina Caragea, Cornelia CarageaProceedings of the ... International ISCRAM Conference 2025 · DOI

Abstract

Despite advancements in semi-supervised learning (SSL) techniques that can be used when labeled data is limited, many SSL approaches still face challenges related to miscalibration. Calibration is crucial for ensuring the accuracy, reliability, and robustness of uncertainty estimates. In this work, we analyze the calibration performance of various SSL methods in the disaster response domain. Our results show that traditional self-training (ST) and mixup-based SSL methods often suffer from high Expected Calibration Error (ECE) despite achieving competitive F1 scores. In contrast, a newly introduced approach in the disaster domain, AUM-ST-Mixup, significantly improves calibration, achieving the lowest ECE across all settings. This improvement suggests that incorporating uncertainty-aware selection via Area Under the Margin (AUM) alongside mixup regularization enhances both predictive performance and model confidence alignment. Our findings highlight the importance of calibration-aware SSL methods, paving the way for more trustworthy model predictions in low-resource settings.

Introduction

In high-stakes disaster response, machine learning models often suffer from "miscalibration"—they tend to be overconfident even when their predictions are incorrect. When disaster strikes, emergency responders rely on social media to identify casualties, infrastructure damage, and urgent needs. However, training deep learning models for these tasks requires massive amounts of labeled data, which is rarely available in the early hours of a crisis.

Semi-supervised learning (SSL) attempts to bridge this gap by using a small amount of labeled data to "pseudo-label" a much larger pool of unlabeled data. However, if the initial model is poorly calibrated, it produces noisy pseudo-labels with high confidence. This leads to a feedback loop where the model "hallucinates" its way into extreme overconfidence, resulting in high Expected Calibration Error (ECE). In a disaster scenario, a model that is 99% sure about a false report of a collapsed bridge is a liability, not an asset.

Methodology: AUM-ST-Mixup

To address these challenges, this research introduces AUM-ST-Mixup, a framework that combines three key components to improve both accuracy and calibration:

  1. Self-Training (ST): An iterative process where a "teacher" model labels unlabeled data to train a "student" model.
  2. Area Under the Margin (AUM): Instead of relying on static confidence scores, AUM tracks training dynamics. It measures the difference between the logit of the assigned label and the next highest logit over several epochs. This helps distinguish between "easy-to-learn" samples (likely correct) and "hard-to-learn" or noisy samples (likely incorrect).
  3. Mixup Regularization: AUM scores are used to categorize pseudo-labeled data. The model then "mixes" (linearly interpolates) labeled data and easy pseudo-labels with hard/noisy samples. This forces the model to learn smoother decision boundaries, which significantly improves calibration.

Experimental Setup and Datasets

The study utilizes the HumAID dataset, a large-scale collection of human-annotated disaster tweets covering 10 distinct events, such as the 2017 Hurricane Irma and the 2018 Kerala Floods. The dataset includes over 77,000 tweets across categories like "Requests or urgent needs" and "Infrastructure damage."

Class distribution across disaster events in the HumAID dataset.

To simulate real-world "low-resource" constraints, the authors created subsets with only 5, 10, 25, and 50 labeled examples per class to evaluate how models perform when data is extremely scarce.

Key Results and Discussion

The proposed AUM-ST-Mixup approach demonstrated superior calibration across all tested scenarios. While traditional SSL methods like standard Self-Training (ST) often achieve high F1 scores, they frequently suffer from the "F1-Calibration Trap," where they are "precisely wrong" about their own certainty.

Comparison of F1 and ECE metrics across various SSL methods and labeling regimes.

As shown in the results, AUM-ST-Mixup dominated the calibration metrics:

  • Superior Calibration: In the most constrained setting (5 labels per class), AUM-ST-Mixup achieved an ECE of 0.099, significantly lower than the 0.360 for standard ST and 0.417 for MixMatch.
  • Competitive Performance: In the 5-label setting, AUM-ST-Mixup also outperformed ST in terms of F1 score (0.529 vs 0.471). While ST achieved a slightly higher F1 in the 50-label setting (0.647 vs 0.633), AUM-ST-Mixup maintained a much better balance between accuracy and confidence alignment.
  • Dynamics over Static Scores: The results confirm that training dynamics (AUM) are a more reliable proxy for truth than raw confidence scores in uncalibrated models.

Conclusion and Future Work

AUM-ST-Mixup provides a robust solution for low-resource disaster response by ensuring models are not only accurate but also "honest" about their confidence. While there is a slight accuracy trade-off in higher-data settings and some computational overhead due to tracking training dynamics, the gains in reliability are paramount for high-stakes applications.

Future research will explore the integration of Large Language Models (LLMs) into this SSL framework and investigate whether ensemble methods can further stabilize calibration in extremely low-resource environments.

Summary updated

Abstract

Despite advancements in semi-supervised learning (SSL) techniques that can be used when labeled data is limited, many SSL approaches still face challenges related to miscalibration. Calibration is crucial for ensuring the accuracy, reliability, and robustness of uncertainty estimates. In this work, we analyze the calibration performance of various SSL methods in the disaster response domain. Our results show that traditional self-training (ST) and mixup-based SSL methods often suffer from high Expected Calibration Error (ECE) despite achieving competitive F1 scores. In contrast, a newly introduced approach in the disaster domain, AUM-ST-Mixup, significantly improves calibration, achieving the lowest ECE across all settings. This improvement suggests that incorporating uncertainty-awareselection via Area Under the Margin (AUM) alongside mixup regularization enhances both predictive performance and model confidence alignment. Our findings highlight the importance of calibration-aware SSL methods, paving the way for more trustworthy model predictions in low-resource settings.
BibTeX
@inproceedings{gupta2025calibrated,
  author = {Khushboo Gupta and Nikita Gautam and Tiberiu Sosea and Doina Caragea and Cornelia Caragea},
  title = {{Calibrated Semi-Supervised Models for Disaster Response based on Training Dynamics}},
  booktitle = {Proceedings of the ... International ISCRAM Conference},
  year = {2025},
  doi = {10.59297/5xkjq067},
}
Figure