Introduction
Rumor detection on social media has become a critical yet challenging task, often described as a "cat-and-mouse" game between spreaders and detectors. Modern rumors are increasingly sophisticated, utilizing provocative imagery and complex social interactions to gain traction. While existing models have attempted to integrate text, images, or social graphs, they frequently encounter three primary hurdles:
- Incomplete Social Graphs: Privacy settings and API limitations often result in missing links within social networks.
- Indiscriminate Aggregation: Standard Graph Neural Networks (GNNs) typically fail to distinguish between supportive and debunking interactions.
- Modality Misalignment: Fusing heterogeneous data, such as pixel-dense images and short text snippets, into a coherent representation remains difficult.
To address these challenges, we present the Multi-modal Feature-enhanced Attention Network (MFAN), a framework designed to capture the nuanced dynamics of rumor propagation across multiple modalities.
Methodology: The MFAN Architecture
The MFAN framework employs a multi-layered approach to extract and fuse features from diverse data sources.
Feature Extraction
The model begins by processing raw inputs:
- Textual Features: A multi-scale CNN captures local semantic information across different word-group sizes.
- Visual Features: Images are processed through a pre-trained ResNet50 to extract high-level visual representations.
Enhanced Social Graph Learning
A core innovation of MFAN is its treatment of social interactions.
- Hidden Link Inference: To combat the "missing link" problem, the model calculates cosine similarity between nodes. If two nodes exhibit high similarity, a potential edge is inferred, enriching the sparse social graph.
- Signed Graph Attention (Signed GAT): Unlike traditional GNNs that assume all connections are positive (homophily), MFAN uses a "signed" mechanism. This allows the model to account for negative correlations, such as users camouflaging their intent or comments that actively oppose a post.
Hierarchical Fusion and Alignment
The extracted features are integrated using a Cross-modal Co-attention mechanism. This allows different modalities to inform one another—for instance, using text to identify relevant regions within an image. Furthermore, a Self-supervised Alignment Loss is introduced to ensure that representations from the textual-visual view and the social graph view are semantically aligned, preventing any single modality from dominating the learning process.
Experimental Evaluation
The performance of MFAN was evaluated on two prominent benchmarks: Weibo (Chinese social media) and PHEME (English tweets from breaking news events). The datasets were filtered to include only multimedia posts containing both text and images, resulting in 2,018 posts for PHEME and 1,467 for Weibo.
Key Results
MFAN consistently outperformed state-of-the-art baselines, including EBGCN and GLAN, across all evaluated metrics.
As shown in the results table, MFAN achieved an accuracy of 88.73% on PHEME and 88.95% on Weibo. These results represent a significant improvement over previous methods. Ablation studies further confirmed the necessity of each component; removing visual or graph features led to a performance drop of approximately 3-4%, demonstrating that the multi-modal approach is essential for robust detection.
Discussion and Future Work
The success of MFAN highlights several key insights:
- The Importance of Negative Neighbors: Capturing oppositional social dynamics is crucial for identifying rumors.
- Alignment as a Regularizer: Forcing cross-modal alignment helps the model learn a more "universal" representation of truth.
- Graph Completion: Inferring potential links significantly boosts performance in the face of sparse metadata.
Despite its strengths, MFAN has limitations. The current study focuses exclusively on posts with both text and images, whereas real-world rumors may be text-only or video-based. Additionally, the computational cost of inferring hidden links at scale and the static nature of the social graph are areas for future improvement. Future iterations could benefit from modeling the temporal propagation paths of rumors as they evolve.
Summary updated