Abstract The prevalence of online misinformation, termed “fake news”, has exponentially escalated in recent years. These deceptive information, often rich with multimodal content, can easily deceive individuals into spreading them via various social media platforms. This has made it a hot research topic to automatically detect multimodal fake news. Existing works made a great progress on inter-modality feature fusion or semantic interaction yet largely ignore the importance of intra-modality entities and feature aggregation. This imbalance causes them to perform erratically on data with different emphases. In the realm of authentic news, the intra-modality contents and the inter-modality relationship should be in mutually supportive relationships. Inspired by this idea, we propose an innovative approach to multimodal fake news detection (IFIS), incorporating both intra-modality feature aggregation and inter-modality semantic fusion. Specifically, the proposed model implements a entity detection module and utilizes attention mechanisms for intra-modality feature aggregation, whereas inter-modality semantic fusion is accomplished via two concurrent Co-attention blocks. The performance of IFIS is extensively tested on two datasets, namely Weibo and Twitter, and has demonstrated superior performance, surpassing various advanced methods. The experimental results validate the capability of our proposed approach in offering the most balanced performance for multimodal fake news detection tasks.
Introduction
Detecting fake news is no longer just a text-processing challenge; modern misinformation is "multimodal," blending deceptive text with doctored or misleading images. While recent research has made strides in inter-modality fusion (how text and images interact), many existing models neglect intra-modality aggregation—the internal relationships within a single modality. This neglect often leads to erratic performance across different datasets. For instance, a model might excel when text is the primary signal but fail when the visual-textual relationship is more nuanced.
In authentic news, internal content and cross-modal relationships are typically mutually supportive. The IFIS (Intra-modality Feature aggregation and Inter-modality Semantic fusion) framework aims to bridge this gap by refining features within each modality before attempting to fuse them, mimicking the human cognitive process of cross-referencing information.
Methodology
The IFIS framework consists of four primary modules designed to process and combine textual and visual data effectively:
- Text Embedding: The model utilizes BERT (Bidirectional Encoder Representations from Transformers) to generate abstract representations of news text. Language-specific versions (Chinese for Weibo, English for Twitter) ensure high-quality semantic capture.
- Entity Feature Embedding: Rather than processing an entire image, which often includes noisy background data, IFIS uses Faster R-CNN to identify the top $k$ salient "entities" (objects or regions). These focused regions are then passed through a ResNet-50 to extract clean visual features.
- Feature Aggregation Module: This module employs Multi-head Self-Attention to model internal relationships. For images, it calculates how different detected entities relate to one another; for text, it captures long-distance dependencies between words. This step "cleans" the data within each modality before fusion.
- Semantic Fusion Module: To combine modalities, the paper introduces two parallel Co-attention blocks. This simulates human behavior: one block uses text to "query" the image features, while the other uses the image to "query" the text, identifying inconsistencies that often characterize fake news.
Experimental Evaluation
Datasets
The model was evaluated on two benchmark datasets:
- Weibo: A large-scale Chinese social media dataset containing approximately 4,779 real posts and 4,749 fake ones.
- Twitter: A dataset from the "Verifying Multimedia Use" task. This dataset is notably smaller and features a "many-to-one" relationship where multiple text posts share the same image, presenting a significant challenge for visual-heavy models.
Key Results
IFIS demonstrated superior and balanced performance across both benchmarks, outperforming advanced baseline models such as EANN, MVAE, and SAFE:
- Weibo Dataset: Achieved an accuracy of 0.896, surpassing the previous state-of-the-art (DIIF).
- Twitter Dataset: Achieved an accuracy of 0.838, outperforming the next best model (MPFN) by 0.5%.
Unlike previous models like MCAN, which showed up to an 8% accuracy swing between datasets, IFIS remained stable. This suggests that intra-modality aggregation makes the model significantly more robust to varying data distributions.
Discussion and Insights
Entities over Pixels
A critical finding of this research is that "classical" image extraction—dividing an image into a grid—often breaks semantic objects apart. By prioritizing object detection (Faster R-CNN), the model focuses on meaningful "entities," which reduces noise and improves detection accuracy. Parameter analysis revealed that selecting exactly $k=5$ visual entities provided the optimal balance between context and computational complexity.
Ablation Studies
Ablation studies confirmed the necessity of the dual-pronged approach. Removing the Entity detection, Aggregation, or Fusion modules resulted in immediate performance drops, validating that both internal refinement and cross-modal interaction are essential for peak performance.
Conclusion and Future Work
IFIS provides a balanced and robust solution for multimodal fake news detection by emphasizing both intra-modality aggregation and inter-modality fusion. While the current model is highly effective for text and static images, future work could expand its scope to include video, audio, or social network propagation structures. Additionally, the authors suggest that integrating external knowledge bases could further enhance the model's ability to identify factual inaccuracies that are not immediately apparent from the provided media alone.
Summary updated