Learning Complex Heterogeneous Multimodal Fake News via Social Latent Network Inference

Mingxin Li, Yuchen Zhang, Haowei Xu, Xianghua Li, Chao Gao, Zhen Wang

TL;DR

Detecting fake news on platforms like TikTok and Instagram is difficult because researchers often cannot see the "social cascades" (who shared what) that traditional detection methods rely on. This paper introduces HML, a framework that reconstructs these hidden social connections using temporal patterns and multimodal content, outperforming state-of-the-art models and even large language models (LLMs) in identifying sophisticated fake news.

What Problem Does This Paper Address?

Most fake news detection research has focused on platforms like Twitter (X) or Weibo, where "retweet" and "comment" chains are visible. These cascades provide a structural "fingerprint" of how news spreads. However, on modern video-centric platforms like TikTok or YouTube, these relationships are often obscured by platform mechanics or privacy settings.

Furthermore, fake news has evolved from simple text to "complex heterogeneous multimodal" content—think deepfakes, edited audio, and misleading video captions. Existing models often just "splice" these different data types together without understanding the deep relationships between them or the social context in which they exist. The authors aim to bridge this gap by inferring a latent social network where an explicit one is missing.

Methods

The HML (Complex Heterogeneous Multimodal Fake News Detection via Latent Network Inference) framework operates in three main stages:

  1. Social Latent Network Inference (Stage 1): The authors use an improved Hawkes Process (a self-exciting point process) to model how news items influence each other over time. Even without a "retweet" button, they can estimate the "maximum likelihood" of influence based on when news about a specific event appears. They also introduce Gaussian noise to simulate how older news is "forgotten" by the public over time.
  2. Self-supervised Multimodal Content Learning (Stage 2): To handle text, audio, and video, the model uses a "Pruning and Mask" strategy to augment data. It employs contrastive learning to align these modalities—ensuring that the features extracted from a video's audio track, its frames, and its description are consistent and robust against "noisy" or imbalanced data.
  3. Network-Augmented Graph Transformer (Stage 3): Finally, the inferred social relationships and the refined multimodal features are fed into a Heterogeneous Multimodal Fake News Graph Transformer. This model treats news items as nodes in a graph, using an attention mechanism to weigh different types of relationships (e.g., same author, same location, or similar content) to make a final "Fake" or "Real" classification.

Datasets

The researchers evaluated HML on several key benchmarks:

  • FakeSV: A comprehensive short-video fake news dataset.
  • FVC (Fake Video Corpus): A standard dataset for video-based rumor detection.
  • Twitter & Weibo: Used to demonstrate that their "Network Inference" module can be used as a "plugin" to improve detection even on platforms where social data is already available.

Key Insights and Ideas

  • Inferring the Invisible: The biggest "aha" moment is that you don't actually need a retweet graph to understand social influence. By modeling the temporal "heat" and popularity of an event, the model can mathematically reconstruct the likely connections between news items.
  • LLMs Aren't Enough (Yet): The study compared HML against GPT-4o and Doubao. Surprisingly, the LLMs performed significantly worse (often by 10-15% in accuracy). This is because LLMs struggle to grasp the structural social context and the specific distribution of news dissemination that a specialized graph-based model can capture.
  • Cross-Modal Discrepancy: Fake news often has "cracks" where the audio doesn't perfectly match the video or the text. The self-supervised contrastive learning stage is specifically designed to catch these fine-grained inconsistencies.

Key Results

  • State-of-the-Art Performance: HML achieved 89.14% accuracy on the FakeSV dataset and 91.02% on the FVC dataset, outperforming previous leaders like MMVD and FakingRecipe by roughly 3-4%.
  • The "Plugin" Effect: When the authors added their Network Inference (NI) module to older, established models (like EANN or CAFE), those models saw an immediate performance boost of up to 10%.
  • Ablation Success: Removing the "Stage 1" latent network inference caused accuracy to drop by about 8%, proving that the social structure is just as important as the content itself for detection.

Limitations and Future Work

While HML is powerful, the authors note a few areas for growth:

  • LLM Adaptation: Since LLMs currently struggle with this task, the authors suggest future work should focus on how to better "prompt" or fine-tune LLMs to understand multimodal graph structures.
  • Real-time Constraints: The paper focuses on batch processing of datasets; applying this latent inference in a real-time "streaming" environment where news is breaking by the second remains a challenge.
  • Computational Complexity: Building and transforming heterogeneous graphs for millions of news items requires significant GPU resources (the study used 4x NVIDIA RTX A6000s). Improving the efficiency of the graph transformer for larger scales is a likely next step.

Summary updated

Abstract

With the diversification of online social platforms, news dissemination has become increasingly complex, heterogeneous, and multimodal, making the fake news detection task more challenging and crucial. Previous works mainly focus on obtaining social relationships of news via retweets, limiting the accurate detection when real cascades are inaccessible. Given the proven assessment of the spreading influence of events, this paper proposes a method called HML (Complex Heterogeneous Multimodal Fake News Detection method via Latent Network Inference). Specifically, an improved social latent network inference strategy is designed to estimate the maximum likelihood of news influences under the same event. Meanwhile, a novel heterogeneous graph is built based on social attributes for multimodal news under different events. Further, to better aggregate the relationships among heterogeneous multimodal features, this paper proposes a self-supervised-based multimodal content learning strategy, to enhance, align, fuse and compare heterogeneous modal contents. Based above, a personalized heterogeneous graph representation learning is designed to classify fake news. Extensive experiments demonstrate that the proposed method outperforms the SOTA in real social media news datasets.
BibTeX
@inproceedings{li2025learning,
  author = {Mingxin Li and Yuchen Zhang and Haowei Xu and Xianghua Li and Chao Gao and Zhen Wang},
  title = {{Learning Complex Heterogeneous Multimodal Fake News via Social Latent Network Inference}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2025},
  url = {https://openreview.net/forum?id=ZZAEM0cXEE},
}
Figure