TL;DR
This paper introduces an interactive system for forensic analysts to sift through social media data for important events. By combining Bayesian Graph Neural Networks (BGNNs) with active learning and pseudo-labeling, the system significantly reduces the manual annotation effort required to train effective multimodal classification models.
What Problem Does This Paper Address?
Forensic investigations often rely on sifting through vast amounts of social media data, which can be time-consuming and overwhelming. Manually identifying relevant posts containing images and text related to specific events is a major bottleneck. While machine learning can automate this, traditional deep learning models require extensive labeled data, which is often unavailable in forensic scenarios. This paper addresses the challenge of building accurate multimodal classification models for event sifting with limited human annotation, proposing an interactive, human-in-the-loop approach.
Methods
The core of the proposed method is an interactive pipeline that leverages Bayesian Graph Neural Networks (BGNNs). Here's a breakdown of the key techniques:
- Bayesian Graph Neural Networks (BGNNs): The authors adapt Monte Carlo Dropout to create a Bayesian version of Graph Neural Networks (specifically, GraphSAGE). This allows the model to not only make predictions but also quantify its uncertainty about those predictions, which is crucial for active learning.
- Active Learning (BALD-based): Instead of relying solely on uncertainty, the system incorporates BALD (Bayesian Active Learning by Disagreement), enhanced with a diversity metric. This helps select the most informative posts for human annotation, ensuring a good balance between uncertainty and data diversity.
- Instance Selection: For initial annotation, KMeans clustering is used to select representative points from different clusters, ensuring a diverse starting set.
- Pseudo-Labeling: The system leverages highly confident predictions from the BGNN to automatically label unlabeled data. This "pseudo-labeling" further enriches the training set, especially when BALD scores are low (indicating high confidence).
- Event Augmentation: To improve robustness and generalization, the training set is augmented with unlabeled data from other forensic events. These are treated as a distinct "another event" class during training, helping the model better distinguish between informative and non-informative posts related to the target event.
- Feature Extraction: A CLIP model is used to extract multimodal features (image and text) from social media posts, creating a unified representation. These features are then used to build a k-NN graph for the GNN.
Datasets
The primary dataset used for evaluation is CrisisMMD [19]. This dataset comprises seven different types of crisis events (hurricanes, wildfires, earthquakes, floods) with varying sizes. The authors filtered the dataset to include only posts with images and text, and removed duplicates. They specifically focused on posts with images attached to them.
Key Insights and Ideas
- BGNNs are effective for forensic event sifting: The paper demonstrates that BGNNs are well-suited for this task, outperforming standard neural networks.
- Active learning and pseudo-labeling are valuable, but context matters: The effectiveness of active learning (BALD) and pseudo-labeling varies depending on the stage of the annotation process and the specific event.
- Augmenting with data from other events helps: Including unlabeled data from unrelated events improves classification performance, likely by helping the model better understand what constitutes "non-informative" data.
- Diversity in active learning is important: Simply picking the most uncertain samples might lead to redundant annotations. Incorporating diversity in instance selection is key.
- Few-shot learning is critical: The proposed methods are particularly effective in low-data regimes, which is characteristic of real-world forensic scenarios.
Key Results
- The proposed BGNN-based method, when combined with augmentation, instance selection, active learning, and pseudo-labeling, achieved the highest F1-scores across different annotation stages (18, 34, and 50 labeled instances).
- In the few-shot learning scenario (18 labeled instances), augmentation and instance selection provided a significant improvement of almost 4 percentage points compared to random selection.
- Pseudo-labeling showed a considerable improvement when moving from 18 to 34 labeled instances, highlighting its importance.
- The performance gains from different configurations became smaller as more labeled data became available, indicating the method's strength in low-data scenarios.
- The BGNN model outperformed other models of similar size (MLP, GNN, BMLP) in terms of F1-score.
- The average runtime for the first iteration (including training, feature extraction, and graph spanning) was around 117 seconds, with subsequent iterations taking about 121 seconds.
Limitations and Future Work
- Runtime: While efficient for interactive use, further research into faster graph construction methods is desirable, especially for larger datasets.
- Instance Selection: The authors suggest exploring community algorithms as an alternative to KMeans for instance selection and investigating the use of graph features (like betweenness and degree) for influence-based selection.
- Graph Construction: Exploring alternative methods for graph construction, potentially incorporating edge features or using heterogeneous graphs, could improve results.
- Human Evaluation: The current experiments were conducted in a simulated environment. Evaluating the method with actual human forensic analysts would be valuable to identify potential real-world adjustments.
Summary updated