Frequency Spectrum Is More Effective for Multimodal Representation and Fusion: A Multimodal Spectrum Rumor Detector

An Lao, Qi Zhang, Chongyang Shi, Longbing Cao, Kun Yi, Liang Hu, Duoqian Miao

Introduction

Detecting rumors on social media is a critical challenge that requires analyzing both text and images. Traditional methods typically operate in the spatial domain, using Convolutional Neural Networks (CNNs) or Transformers to process pixels and word sequences. However, these approaches often struggle to distinguish rumor-specific features from background noise and suffer from high computational complexity when fusing different modalities.

To address these limitations, this paper introduces the Frequency Spectrum Representation and fUsion network (FSRU). By shifting multimodal analysis from the spatial domain to the frequency domain using Fourier Transforms, FSRU achieves more discriminative representations and efficient cross-modal fusion, outperforming existing state-of-the-art models.

Methodology: The FSRU Architecture

The core innovation of FSRU is the use of the Discrete Fourier Transform (DFT) to process data. This shift allows the model to leverage the mathematical properties of the frequency domain for better feature extraction and integration.

The architecture of the FSRU model, highlighting the shift to the frequency domain for representation and fusion.

Key components of the FSRU framework include:

  • Unimodal Spectrum Compression (USC): Text and images are converted into frequency spectra and then compressed using learnable filter banks. This process emphasizes "veracity-relevant" components while suppressing irrelevant noise.
  • Cross-modal Spectrum Co-selection (CSC): This module employs an "Emphasize-and-Suppress" (E&S) mechanism where one modality acts as a filter for the other. This helps the model focus on complementary features indicative of rumors.
  • Dual Contrastive Learning: The model is optimized using two contrastive losses: intra-modal (separating rumors from non-rumors within one modality) and inter-modal (aligning corresponding text-image pairs).
  • Distribution Similarity Fusion: Instead of simple concatenation, FSRU uses Jensen-Shannon (JS) divergence to measure similarity between text and image distributions, adaptively weighting them for final classification.

Key Insights: Why the Frequency Domain?

Moving to the frequency domain provides several distinct advantages for rumor detection:

  1. Sparsity and Discriminative Power: In the frequency domain, signals often become sparse. As shown in the spectrogram comparisons, rumors and non-rumors exhibit distinct, centrally concentrated patterns that are easier for the model to identify than raw spatial data.
  2. Global Context: A single component in the frequency domain inherently contains information from the entire spatial signal. This provides a global view similar to Transformers but without the quadratic computational cost of self-attention.
  3. Efficiency: Point-wise multiplication in the frequency domain is mathematically equivalent to complex convolutions in the spatial domain, allowing for highly efficient cross-modal interaction.
Visual examples of raw data and their corresponding spectrograms for non-rumors and rumors.

Experimental Results

The researchers evaluated FSRU on two major benchmarks: Twitter and Weibo. FSRU consistently ranked at the top against eight competitive baselines, including CAFE and BMR.

  • Performance: FSRU achieved an accuracy of 95.2% on Twitter and 90.1% on Weibo.
  • Ablation Studies: Removing the frequency-domain modules (USC or CSC) led to significant performance drops, such as a ~3.5% decrease in accuracy on Weibo without USC, confirming the importance of the frequency-based approach.
  • Feature Separation: Qualitative analysis via t-SNE visualizations demonstrated that FSRU creates much clearer boundaries between rumor and non-rumor features compared to spatial-domain models.

Conclusion

FSRU demonstrates that the frequency domain is a powerful alternative for multimodal rumor detection. While the performance gain on Weibo was slightly lower due to image quality variations, the model's efficiency and accuracy make it a robust solution. Future work will focus on deeper theoretical analysis of frequency components and applying this architecture to other multimodal tasks like sentiment analysis.

Summary updated

Abstract

Multimodal content, such as mixing text with images, presents significant challenges to rumor detection in social media. Existing multimodal rumor detection has focused on mixing tokens among spatial and sequential locations for unimodal representation or fusing clues of rumor veracity across modalities. However, they suffer from less discriminative unimodal representation and are vulnerable to intricate location dependencies in the time-consuming fusion of spatial and sequential tokens. This work makes the first attempt at multimodal rumor detection in the frequency domain, which efficiently transforms spatial features into the frequency spectrum and obtains highly discriminative spectrum features for multimodal representation and fusion. A novel Frequency Spectrum Representation and fUsion network (FSRU) with dual contrastive learning reveals the frequency spectrum is more effective for multimodal representation and fusion, extracting the informative components for rumor detection. FSRU involves three novel mechanisms: utilizing the Fourier transform to convert features in the spatial domain to the frequency domain, the unimodal spectrum compression, and the cross-modal spectrum co-selection module in the frequency domain. Substantial experiments show that FSRU achieves satisfactory multimodal rumor detection performance.
BibTeX
@inproceedings{lao2024frequency,
  author = {An Lao and Qi Zhang and Chongyang Shi and Longbing Cao and Kun Yi and Liang Hu and Duoqian Miao},
  title = {{Frequency Spectrum Is More Effective for Multimodal Representation and Fusion: A Multimodal Spectrum Rumor Detector}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2024},
  doi = {10.1609/aaai.v38i16.29803},
}
Figure