Abstract
Social media platforms provide a wealth of real-time information during forensic investigations and natural disasters. However, the sheer volume of data and the prevalence of irrelevant content create a significant challenge for human analysts. This paper presents an interactive, human-in-the-loop framework that leverages Bayesian Graph Neural Networks (BGNNs) and a diversity-aware active learning strategy to sift through massive datasets. By effectively estimating model uncertainty and ensuring sample diversity, the proposed system achieves high classification accuracy with as few as 18 to 50 manual annotations, significantly reducing the human workload in time-critical scenarios.
Introduction
In the immediate aftermath of a major event, such as a hurricane or an earthquake, social media is inundated with multimodal content. While keyword-based retrieval can aggregate relevant posts, the resulting datasets are often "noisy," containing a high percentage of irrelevant information. Traditional deep learning models, which typically require thousands of labeled examples, are ill-suited for these fast-moving forensic scenarios where labels are scarce and time is of the essence.
This research addresses the "few-shot" sifting problem: building a reliable classifier that can learn from a minimal set of human-labeled examples. The goal is to provide analysts with a tool that not only identifies informative content but also provides a trustworthy measure of its own predictive uncertainty, allowing for more effective human-AI collaboration.
Methodology: A Human-in-the-Loop Pipeline
The proposed framework utilizes a multi-stage pipeline designed for both computational efficiency and high interactivity. The process begins with multimodal feature extraction using a Contrastive Language-Image Pre-training (CLIP) model, which maps both text and images into a shared vector space. These features are then used to construct a k-Nearest Neighbors (k-NN) graph, where each post is a node connected to its 16 most similar neighbors. This graph structure allows the model to propagate information from a small set of labeled nodes to the rest of the unlabeled dataset.
At the core of the system is a Bayesian Graph Neural Network (BGNN) based on GraphSAGE layers. To estimate uncertainty, the model employs Monte Carlo (MC) Dropout, running multiple inference passes to generate a distribution of predictions. This uncertainty is then leveraged by a diversity-aware active learning strategy called BALD-KMeans. Unlike standard active learning that might select redundant samples, BALD-KMeans uses clustering to ensure that the posts presented to the human analyst for labeling are both highly uncertain and representative of different regions in the data distribution.
Furthermore, the framework incorporates "Event Augmentation," where unlabeled data from unrelated events is used to help the model better define the boundaries of the target class. Finally, a pseudo-labeling mechanism automatically assigns labels to the most confident predictions, further accelerating the learning process.
Experimental Evaluation and Results
The framework was evaluated using the CrisisMMD dataset, which includes social media posts from seven major natural disasters, such as Hurricane Irma and the Mexico Earthquake. The primary task was to classify posts as "informative" or "not informative" for humanitarian or forensic purposes.
Experimental results demonstrate that the system is highly effective even with extremely limited supervision. With only 18 manual labels, the BGNN achieved an F1-score of approximately 73.6%. Increasing the label count to 50—a task requiring only a few minutes of human effort—pushed the F1-score to 77.8%.
An ablation study (see Figure 5) confirms that each component of the pipeline contributes significantly to the overall performance. Removing event augmentation or replacing the diversity-aware selection with random sampling led to noticeable drops in accuracy. In terms of efficiency, the initial setup (training and graph construction) takes roughly 117 seconds on an RTX 6000 GPU, with subsequent active learning loops completing in approximately 121 seconds, making the system viable for real-time deployment.
Discussion and Conclusion
This work highlights the importance of uncertainty and diversity in interactive machine learning. By combining Bayesian inference with graph-based representation, the system effectively identifies the "needles in the haystack" of social media data. The inclusion of data from disparate events proved to be a powerful regularizer, providing a denser background that makes relevant information easier to distinguish.
Future work will focus on improving graph scalability for datasets containing millions of posts and exploring more complex heterogeneous graph structures that incorporate metadata such as hashtags and geographic proximity. Additionally, real-world testing with forensic analysts is planned to evaluate the impact of human factors like fatigue and bias on the active learning loop.
Summary updated