Introduction
Detecting rumors on social media is a critical challenge that requires analyzing both text and images. Traditional methods typically operate in the spatial domain, using Convolutional Neural Networks (CNNs) or Transformers to process pixels and word sequences. However, these approaches often struggle to distinguish rumor-specific features from background noise and suffer from high computational complexity when fusing different modalities.
To address these limitations, this paper introduces the Frequency Spectrum Representation and fUsion network (FSRU). By shifting multimodal analysis from the spatial domain to the frequency domain using Fourier Transforms, FSRU achieves more discriminative representations and efficient cross-modal fusion, outperforming existing state-of-the-art models.
Methodology: The FSRU Architecture
The core innovation of FSRU is the use of the Discrete Fourier Transform (DFT) to process data. This shift allows the model to leverage the mathematical properties of the frequency domain for better feature extraction and integration.
Key components of the FSRU framework include:
- Unimodal Spectrum Compression (USC): Text and images are converted into frequency spectra and then compressed using learnable filter banks. This process emphasizes "veracity-relevant" components while suppressing irrelevant noise.
- Cross-modal Spectrum Co-selection (CSC): This module employs an "Emphasize-and-Suppress" (E&S) mechanism where one modality acts as a filter for the other. This helps the model focus on complementary features indicative of rumors.
- Dual Contrastive Learning: The model is optimized using two contrastive losses: intra-modal (separating rumors from non-rumors within one modality) and inter-modal (aligning corresponding text-image pairs).
- Distribution Similarity Fusion: Instead of simple concatenation, FSRU uses Jensen-Shannon (JS) divergence to measure similarity between text and image distributions, adaptively weighting them for final classification.
Key Insights: Why the Frequency Domain?
Moving to the frequency domain provides several distinct advantages for rumor detection:
- Sparsity and Discriminative Power: In the frequency domain, signals often become sparse. As shown in the spectrogram comparisons, rumors and non-rumors exhibit distinct, centrally concentrated patterns that are easier for the model to identify than raw spatial data.
- Global Context: A single component in the frequency domain inherently contains information from the entire spatial signal. This provides a global view similar to Transformers but without the quadratic computational cost of self-attention.
- Efficiency: Point-wise multiplication in the frequency domain is mathematically equivalent to complex convolutions in the spatial domain, allowing for highly efficient cross-modal interaction.
Experimental Results
The researchers evaluated FSRU on two major benchmarks: Twitter and Weibo. FSRU consistently ranked at the top against eight competitive baselines, including CAFE and BMR.
- Performance: FSRU achieved an accuracy of 95.2% on Twitter and 90.1% on Weibo.
- Ablation Studies: Removing the frequency-domain modules (USC or CSC) led to significant performance drops, such as a ~3.5% decrease in accuracy on Weibo without USC, confirming the importance of the frequency-based approach.
- Feature Separation: Qualitative analysis via t-SNE visualizations demonstrated that FSRU creates much clearer boundaries between rumor and non-rumor features compared to spatial-domain models.
Conclusion
FSRU demonstrates that the frequency domain is a powerful alternative for multimodal rumor detection. While the performance gain on Weibo was slightly lower due to image quality variations, the model's efficiency and accuracy make it a robust solution. Future work will focus on deeper theoretical analysis of frequency components and applying this architecture to other multimodal tasks like sentiment analysis.
Summary updated