Detecting Fake News on Social Media via Multimodal Semantic Understanding and Enhanced Transformer Architectures

Meiling Xu, Feng Li, Zhuang Miao, Zhongchuan Han, Lei Wang, Gong WangTraitement 2025 · DOI

TL;DR

This paper introduces a novel framework for detecting fake news by combining multimodal semantic matching (text + images) with an enhanced Transformer architecture. By using Gaussian Mixture Models to filter out noisy data and incorporating structural embeddings to map news propagation trees, the model achieves a 91.2% detection accuracy, significantly outperforming text-only and standard multimodal baselines.

What Problem Does This Paper Address?

Social media has turned misinformation into a high-speed, multimodal phenomenon. While early detection efforts focused almost exclusively on text, modern fake news often relies on the interplay—or intentional mismatch—between provocative images and misleading captions.

Current multimodal detection models face two major hurdles:

  1. Semantic Mismatch: They struggle to capture the "deep" relationship between an image and its text. For example, a real photo used in a completely false context (a "hard" sample) can easily trick a standard neural network.
  2. Data Noise: Social media data is messy. Traditional models often overfit on "noise" (samples where text and image are completely unrelated by accident) or struggle with the heterogeneous nature of the data.

The authors aim to bridge these gaps by creating a system that doesn't just look at content in isolation but also understands how that content spreads through "propagation trees."

Methods

The authors propose a two-part framework:

1. Multimodal Semantic Matching

To ensure the model understands if an image and text actually belong together, they use a Siamese network structure based on CLIP (Contrastive Language-Image Pre-training).

  • Feature Extraction: They use a Vision Transformer (ViT-B/32) for images and a 12-layer Transformer for text.
  • Noise Filtering via GMM: This is a standout feature. The researchers used a Gaussian Mixture Model (GMM) to analyze the loss distribution of training samples. By identifying "simple" samples (low loss, clear semantics) versus "hard/noisy" samples (high loss), they could prioritize learning from high-quality data and reduce the weight of noisy samples that might lead to "erroneous memories" in the network.

2. Enhanced Transformer for Detection

Once the features are matched, they are fed into a specialized Transformer designed for social media's unique structure.

  • Structural Embeddings: Unlike standard Transformers that treat data as a sequence, this model uses horizontal and vertical position embeddings to represent a news item's place within a propagation tree (the web of retweets and replies).
  • Path-Level Attention: The model doesn't treat every retweet equally. It uses a dynamic gating mechanism and attention layers to identify which propagation paths are most indicative of fake news.
  • Path Oversampling: To handle the fact that some news stories have very few replies, they implemented an oversampling method to ensure the model receives a balanced amount of feedback for every event.

Datasets

The model was evaluated on three primary public datasets:

  • Image Verification Corpus: A standard benchmark for image-text consistency.
  • Weibo Dataset: A large-scale dataset from the Chinese social media platform, rich in propagation data.
  • Top News Dataset: Used to test the model's performance on trending, high-impact news cycles.

Key Insights and Ideas

  • The "Simple Sample" Priority: One of the most interesting findings is that "retaining key semantics" from simple, clear samples actually makes the model better at identifying complex fakes. By not letting the model get "confused" by noisy data early in training, it develops a more robust baseline for what "real" news looks like.
  • Structure is a Signal: Fake news spreads differently than real news. By encoding the hierarchy of the propagation tree (who replied to whom and when), the Transformer can spot patterns of misinformation spread that content analysis alone would miss.
  • Two-Stage Training: The authors didn't just train once. They used a first stage to "predict the negative impact" of specific samples and a second stage to retrain the main model with weighted losses. This "curriculum-style" approach significantly boosted noise robustness.

Key Results

The proposed model showed dominant performance across the board:

  • Detection Accuracy: Reached 91.2%, outperforming heavyweights like BERT (42.5%), RoBERTa (57.9%), and even specialized multimodal models like FLAVA (85.6%).
  • Image-Text Matching: Achieved an R@1 of 81.2 for Image-to-Text matching, proving that the GMM-based filtering effectively sharpened the model’s "eyes."
  • F1 Scores: The model was particularly effective at identifying "Unverifiable Fake News" (F1: 0.912) and "Verified True Fake News" (F1: 0.945), categories where other models typically struggle due to lack of clear textual cues.

Limitations and Future Work

Despite the high accuracy, the authors acknowledge a few "weak spots":

  • Short Texts: The model’s performance dips when the accompanying text is very short or when the propagation tree is sparse (few retweets/replies).
  • Computational Complexity: While not explicitly detailed, the multi-stage training and hierarchical attention mechanisms likely require significant compute compared to simpler architectures.
  • Real-Time Processing: Future work is needed to optimize the model for real-time detection in high-volume social media streams. The authors also suggest exploring even more diverse fusion techniques, such as cross-modal alignment, to further improve efficiency.

Summary updated

Abstract

With the rapid development of social media, multimodal news combining text and images has become a primary vehicle for the spread of misinformation due to its strong dissemination power and broad audience reach.Existing fake news detection methods overly rely on textual analysis, making it difficult to effectively capture the deep semantic relationships between visual and textual modalities.Moreover, traditional neural networks often suffer from limited robustness and weak noise resistance when handling heterogeneous multimodal data.Although multimodal learning has recently been introduced into this field, challenges such as insufficient text-image semantic matching and a lack of architectural innovation persist.To address these issues, this study proposes a detection framework that integrates multimodal semantic understanding with an enhanced Transformer architecture.Specifically, a cross-modal attention mechanism is constructed to strengthen the interactive representation of visual and textual features through semantic alignment.Additionally, a hierarchical Transformer structure with a dynamic gating mechanism is designed to optimize the multimodal information fusion strategy, significantly improving the model's adaptability to complex social media scenarios.Experiments conducted on public datasets demonstrate that the proposed method achieves a noticeable improvement in detection accuracy compared to traditional models, while also exhibiting superior recall and noise robustness.This research offers a more efficient technological pathway for fake news detection on social media and promotes the broader application of multimodal learning in the domain of information content security.
BibTeX
@inproceedings{xu2025detecting,
  author = {Meiling Xu and Feng Li and Zhuang Miao and Zhongchuan Han and Lei Wang and Gong Wang},
  title = {{Detecting Fake News on Social Media via Multimodal Semantic Understanding and Enhanced Transformer Architectures}},
  booktitle = {Traitement},
  year = {2025},
  doi = {10.18280/ts.420327},
}
Figure