Introduction
Multimodal fake news detection, which integrates information from text, images, audio, and video, has become a standard approach for identifying misinformation on social media. However, recent research identifies a significant challenge: modality disruption. This phenomenon occurs when certain modalities—often designed to be sensational or clickbait—introduce noise that confuses detection models. Instead of improving performance, the inclusion of additional data types can sometimes degrade accuracy.
As shown in the figure above, creators often use hyperbolic language or disjointed visual editing. In the "Fake News" example, the text claims a nuclear attack while the video shows a standard speech. Conversely, in the "Real News" example, the text describes an "intimate interaction" while the video depicts a man punching a kangaroo to save his dog. These discrepancies create "noise" that standard multimodal models struggle to filter.
The FND-MoE Framework
To address modality disruption, the researchers propose the Fake News Detection Mixture of Experts (FND-MoE) framework. This architecture is designed to selectively focus on the most reliable features across different modalities while actively suppressing disruptive noise.
The framework consists of several key stages:
- Specialized Feature Extraction: Modality-specific experts (BERT for text, VGG/Wav2Vec2.0 for audio, C3D/VGG19 for video) extract high-level features.
- Pairwise Cross-Attention: The model computes interactions between all pairs of modalities to identify complementary signals.
- Two-Pass Feature Selection: This core module filters the fused features to retain only the most informative vectors.
Two-Pass Feature Selection
The innovation of FND-MoE lies in its two-pass gating mechanism. Standard "Softmax" gating spreads attention across all inputs, meaning the model always "listens" to disruptive noise to some degree. FND-MoE improves on this with:
- Pass 1: Top-k Gating. An attention-based mechanism ranks all fused feature vectors and selects the top $k$ most relevant ones.
- Pass 2: Gumbel-Sigmoid Selection. Using the Gumbel-Max trick, the model makes discrete binary decisions—effectively "muting" disruptive modalities—while remaining trainable via backpropagation.
Experimental Results
The FND-MoE framework was evaluated on two major benchmarks: FakeSV (a large Chinese short video dataset) and FVC-2018 (a YouTube video dataset).
- FakeSV: Achieved 82.84% accuracy, a 3.45% improvement over the previous state-of-the-art (SOTA).
- FVC-2018: Achieved 88.44% accuracy, a 3.71% improvement over SOTA.
Ablation studies confirmed that the combination of Top-k selection and Gumbel-Sigmoid is critical. Without these filtering steps, the models were significantly more vulnerable to the "more is less" paradox, where adding modalities like video or audio to a text-based model actually decreased its performance.
Conclusion
Modality disruption highlights a fundamental weakness in current multimodal AI: the inability to distinguish between signal and sensationalized noise. By implementing a discrete, two-pass selection mechanism, FND-MoE demonstrates that "noise-canceling" is just as vital as feature fusion in the fight against digital misinformation. Future work will explore how these techniques can be adapted to counter increasingly sophisticated AI-generated deepfakes.
Summary updated