MFAN: Multi-modal Feature-enhanced Attention Networks for Rumor Detection

Jiaqi Zheng, Xi Zhang, Sanchuan Guo, Quan Wang, Wenyu Zang, Yongdong Zhang

Introduction

Rumor detection on social media has become a critical yet challenging task, often described as a "cat-and-mouse" game between spreaders and detectors. Modern rumors are increasingly sophisticated, utilizing provocative imagery and complex social interactions to gain traction. While existing models have attempted to integrate text, images, or social graphs, they frequently encounter three primary hurdles:

  1. Incomplete Social Graphs: Privacy settings and API limitations often result in missing links within social networks.
  2. Indiscriminate Aggregation: Standard Graph Neural Networks (GNNs) typically fail to distinguish between supportive and debunking interactions.
  3. Modality Misalignment: Fusing heterogeneous data, such as pixel-dense images and short text snippets, into a coherent representation remains difficult.

To address these challenges, we present the Multi-modal Feature-enhanced Attention Network (MFAN), a framework designed to capture the nuanced dynamics of rumor propagation across multiple modalities.

Methodology: The MFAN Architecture

The MFAN framework employs a multi-layered approach to extract and fuse features from diverse data sources.

The MFAN architecture, illustrating the multi-modal feature extraction and hierarchical fusion process.

Feature Extraction

The model begins by processing raw inputs:

  • Textual Features: A multi-scale CNN captures local semantic information across different word-group sizes.
  • Visual Features: Images are processed through a pre-trained ResNet50 to extract high-level visual representations.

Enhanced Social Graph Learning

A core innovation of MFAN is its treatment of social interactions.

  • Hidden Link Inference: To combat the "missing link" problem, the model calculates cosine similarity between nodes. If two nodes exhibit high similarity, a potential edge is inferred, enriching the sparse social graph.
  • Signed Graph Attention (Signed GAT): Unlike traditional GNNs that assume all connections are positive (homophily), MFAN uses a "signed" mechanism. This allows the model to account for negative correlations, such as users camouflaging their intent or comments that actively oppose a post.

Hierarchical Fusion and Alignment

The extracted features are integrated using a Cross-modal Co-attention mechanism. This allows different modalities to inform one another—for instance, using text to identify relevant regions within an image. Furthermore, a Self-supervised Alignment Loss is introduced to ensure that representations from the textual-visual view and the social graph view are semantically aligned, preventing any single modality from dominating the learning process.

Experimental Evaluation

The performance of MFAN was evaluated on two prominent benchmarks: Weibo (Chinese social media) and PHEME (English tweets from breaking news events). The datasets were filtered to include only multimedia posts containing both text and images, resulting in 2,018 posts for PHEME and 1,467 for Weibo.

Key Results

MFAN consistently outperformed state-of-the-art baselines, including EBGCN and GLAN, across all evaluated metrics.

Performance comparison of MFAN against state-of-the-art rumor detection methods on PHEME and Weibo datasets.

As shown in the results table, MFAN achieved an accuracy of 88.73% on PHEME and 88.95% on Weibo. These results represent a significant improvement over previous methods. Ablation studies further confirmed the necessity of each component; removing visual or graph features led to a performance drop of approximately 3-4%, demonstrating that the multi-modal approach is essential for robust detection.

Discussion and Future Work

The success of MFAN highlights several key insights:

  • The Importance of Negative Neighbors: Capturing oppositional social dynamics is crucial for identifying rumors.
  • Alignment as a Regularizer: Forcing cross-modal alignment helps the model learn a more "universal" representation of truth.
  • Graph Completion: Inferring potential links significantly boosts performance in the face of sparse metadata.

Despite its strengths, MFAN has limitations. The current study focuses exclusively on posts with both text and images, whereas real-world rumors may be text-only or video-based. Additionally, the computational cost of inferring hidden links at scale and the static nature of the social graph are areas for future improvement. Future iterations could benefit from modeling the temporal propagation paths of rumors as they evolve.

Summary updated

Abstract

Rumor spreaders are increasingly taking advantage of multimedia content to attract and mislead news consumers on social media. Although recent multimedia rumor detection models have exploited both textual and visual features for classification, they do not integrate the social structure features simultaneously, which have shown promising performance for rumor identification. It is challenging to combine the heterogeneous multi-modal data in consideration of their complex relationships. In this work, we propose a novel Multi-modal Feature-enhanced Attention Networks (MFAN) for rumor detection, which makes the first attempt to integrate textual, visual, and social graph features in one unified framework. Specifically, it considers both the complement and alignment relationships between different modalities to achieve better fusion. Moreover, it takes into account the incomplete links in the social network data due to data collection constraints and proposes to infer hidden links to learn better social graph features. The experimental results show that MFAN can detect rumors effectively and outperform state-of-the-art methods.
BibTeX
@inproceedings{zheng2022mfan,
  author = {Jiaqi Zheng and Xi Zhang and Sanchuan Guo and Quan Wang and Wenyu Zang and Yongdong Zhang},
  title = {{MFAN: Multi-modal Feature-enhanced Attention Networks for Rumor Detection}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2022},
  doi = {10.24963/ijcai.2022/335},
}
Figure