A two-branch multimodal fake news detection model based on multimodal bilinear pooling and attention mechanism

Ying Guo, Hong Ge, Jinhong LiFrontiers of Computer Science 2023 · DOI

Introduction

The proliferation of fake news on social media presents a significant challenge, as deceptive content often combines sensationalist text with misleading imagery. Detecting such "multimodal" misinformation requires models that can effectively interpret the complex relationships between different data types. Current state-of-the-art (SOTA) models often struggle with two primary issues: ineffective feature fusion—typically relying on simple concatenation—and poor generalization across diverse news events.

The MBPAM Framework

To address these limitations, researchers have developed the Multimodal Bilinear Pooling and Attention Mechanism (MBPAM). This model moves beyond simple data "stacking" to capture nuanced, non-linear interactions between text and images.

The overall architecture of the MBPAM model, illustrating the feature extraction, fusion, and detection modules.

Two-Branch Feature Extraction

The MBPAM model employs a dual-branch approach for both text and image modalities to ensure that both global and fine-grained features are captured.

  • Text Processing: Utilizing a pre-trained BERT model, one branch extracts global sentence semantics. A parallel branch uses a 1D-CNN to capture multi-granularity word-level features, ensuring that specific linguistic "smoking guns" are not lost.
  • Image Processing: Using ResNet-50, the model similarly splits processing into regional information and deep image semantics.
Detailed view of the text feature extractor module, showing the sentence and word-level branches.

Advanced Fusion and Enhancement

The core innovation of MBPAM lies in its fusion strategy. Instead of simple addition, it utilizes Multimodal Compact Bilinear Pooling (MCBP). This technique calculates the outer product of text and image features, allowing for complex cross-modal synergy. To manage computational costs, the model employs the Count Sketch algorithm and Fast Fourier Transforms (FFT).

Furthermore, a self-attention mechanism is applied to each modality independently. This highlights the most relevant or suspicious parts of the content before the final fusion, enhancing the model's sensitivity to deceptive patterns.

Domain Adversarial Training

To ensure the model remains effective across different news topics (e.g., shifting from election news to health crises), a Domain Adversarial Module is integrated. This module penalizes the model if it can easily identify the specific news event associated with the data. This forces the feature extractor to focus on "event-agnostic" characteristics typical of fake news regardless of the subject matter.

Experimental Results

The MBPAM model was evaluated against two major benchmarks: the Weibo Dataset (Chinese) and the Twitter Dataset (English).

Dataset MBPAM Accuracy Previous Best (BDANN/Spotfake)
Weibo 90.4% 84.2% (BDANN)
Twitter 86.8% 83.0% (BDANN) / 77.7% (Spotfake)

Ablation studies confirmed that the bilinear pooling mechanism was the most critical component for these performance gains, particularly on the Weibo dataset.

Conclusion

The MBPAM model demonstrates that the relationship between text and images in fake news is multiplicative rather than additive. By capturing cross-modal synergies and employing adversarial training for better generalization, MBPAM sets a new standard for multimodal rumor detection. Future work will look to incorporate social context metadata and handle posts containing multiple images.

Summary updated

BibTeX
@article{guo2023two,
  author = {Ying Guo and Hong Ge and Jinhong Li},
  title = {{A two-branch multimodal fake news detection model based on multimodal bilinear pooling and attention mechanism}},
  journal = {Frontiers of Computer Science},
  year = {2023},
  doi = {10.3389/fcomp.2023.1159063},
}
Figure