Exploring Modality Disruption in Multimodal Fake News Detection

Moyang Liu, Kaiying Yan, Yukun Liu, Ruibo Fu, Zhengqi Wen, Xuefei Liu, Chenxing LiarXiv 2025 · DOI · arXiv

Introduction

Multimodal fake news detection, which integrates information from text, images, audio, and video, has become a standard approach for identifying misinformation on social media. However, recent research identifies a significant challenge: modality disruption. This phenomenon occurs when certain modalities—often designed to be sensational or clickbait—introduce noise that confuses detection models. Instead of improving performance, the inclusion of additional data types can sometimes degrade accuracy.

Examples of modality disruption where text and visual content provide conflicting or sensationalized information.

As shown in the figure above, creators often use hyperbolic language or disjointed visual editing. In the "Fake News" example, the text claims a nuclear attack while the video shows a standard speech. Conversely, in the "Real News" example, the text describes an "intimate interaction" while the video depicts a man punching a kangaroo to save his dog. These discrepancies create "noise" that standard multimodal models struggle to filter.

The FND-MoE Framework

To address modality disruption, the researchers propose the Fake News Detection Mixture of Experts (FND-MoE) framework. This architecture is designed to selectively focus on the most reliable features across different modalities while actively suppressing disruptive noise.

The FND-MoE architecture, featuring specialized experts and a two-pass feature selection module.

The framework consists of several key stages:

  1. Specialized Feature Extraction: Modality-specific experts (BERT for text, VGG/Wav2Vec2.0 for audio, C3D/VGG19 for video) extract high-level features.
  2. Pairwise Cross-Attention: The model computes interactions between all pairs of modalities to identify complementary signals.
  3. Two-Pass Feature Selection: This core module filters the fused features to retain only the most informative vectors.

Two-Pass Feature Selection

The innovation of FND-MoE lies in its two-pass gating mechanism. Standard "Softmax" gating spreads attention across all inputs, meaning the model always "listens" to disruptive noise to some degree. FND-MoE improves on this with:

  • Pass 1: Top-k Gating. An attention-based mechanism ranks all fused feature vectors and selects the top $k$ most relevant ones.
  • Pass 2: Gumbel-Sigmoid Selection. Using the Gumbel-Max trick, the model makes discrete binary decisions—effectively "muting" disruptive modalities—while remaining trainable via backpropagation.

Experimental Results

The FND-MoE framework was evaluated on two major benchmarks: FakeSV (a large Chinese short video dataset) and FVC-2018 (a YouTube video dataset).

  • FakeSV: Achieved 82.84% accuracy, a 3.45% improvement over the previous state-of-the-art (SOTA).
  • FVC-2018: Achieved 88.44% accuracy, a 3.71% improvement over SOTA.

Ablation studies confirmed that the combination of Top-k selection and Gumbel-Sigmoid is critical. Without these filtering steps, the models were significantly more vulnerable to the "more is less" paradox, where adding modalities like video or audio to a text-based model actually decreased its performance.

Conclusion

Modality disruption highlights a fundamental weakness in current multimodal AI: the inability to distinguish between signal and sensationalized noise. By implementing a discrete, two-pass selection mechanism, FND-MoE demonstrates that "noise-canceling" is just as vital as feature fusion in the fight against digital misinformation. Future work will explore how these techniques can be adapted to counter increasingly sophisticated AI-generated deepfakes.

Summary updated

Abstract

The rapid growth of social media has led to the widespread dissemination of fake news across multiple content forms, including text, images, audio, and video. Compared to unimodal fake news detection, multimodal fake news detection benefits from the increased availability of information across multiple modalities. However, in the context of social media, certain modalities in multimodal fake news detection tasks may contain disruptive or over-expressive information. These elements often include exaggerated or embellished content. We define this phenomenon as modality disruption and explore its impact on detection models through experiments. To address the issue of modality disruption in a targeted manner, we propose a multimodal fake news detection framework, FND-MoE. Additionally, we design a two-pass feature selection mechanism to further mitigate the impact of modality disruption. Extensive experiments on the FakeSV and FVC-2018 datasets demonstrate that FND-MoE significantly outperforms state-of-the-art methods, with accuracy improvements of 3.45% and 3.71% on the respective datasets compared to baseline models.
BibTeX
@misc{liu2025exploring,
  author = {Moyang Liu and Kaiying Yan and Yukun Liu and Ruibo Fu and Zhengqi Wen and Xuefei Liu and Chenxing Li},
  title = {{Exploring Modality Disruption in Multimodal Fake News Detection}},
  year = {2025},
  eprint = {2504.09154},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2504.09154},
}
Figure