FineFake: A Knowledge-Enriched Dataset for Fine-Grained Multi-Domain Fake News Detection

Ziyi Zhou, Xiaoming Zhang, Litian Zhang, Jiacheng Liu, Chaozhuo Li

Abstract

Existing benchmarks for fake news detection have significantly contributed to the advancement of models in assessing the authenticity of news content. However, these benchmarks typically focus solely on news pertaining to a single semantic topic or originating from a single platform, thereby failing to capture the diversity of multi-domain news in real scenarios. In order to understand fake news across various domains, external knowledge and fine-grained annotations are indispensable to provide precise evidence and uncover the diverse underlying strategies for fabrication. To address this gap, we introduce a novel multi-domain knowledge-enhanced benchmark with fine-grained annotations, named FineFake. FineFake encompasses 16,909 data samples spanning six semantic topics and eight platforms. Each news item is enriched with multi-modal content, potential social context, semi-manually verified common knowledge, and fine-grained annotations that surpass conventional binary labels. Furthermore, we propose a knowledge-enhanced domain adaptation network (KEAN) and demonstrate its effectiveness through extensive experiments.

Beyond Binary: The Need for Fine-Grained Detection

Most existing fake news benchmarks suffer from "tunnel vision," focusing on single topics (e.g., US Elections) or single platforms (e.g., Twitter). This limitation hinders a model's ability to handle domain shift—the phenomenon where a model trained on political rumors fails when applied to health news or different social media environments.

Moreover, traditional datasets rely on binary "Real" vs. "Fake" labels. This oversimplification ignores the specific fabrication strategies used. A news item might be fake due to fabricated text, manipulated images, or a mismatch between the two. Without fine-grained annotations and external factual evidence, models cannot explain why a piece of content is deemed untrustworthy.

Figure 1: Illustration of platform and topic shifts, highlighting the entity ambiguity problem (e.g., "Apple" as a company vs. a fruit).

The FineFake Dataset

The FineFake dataset addresses these gaps by providing 16,909 samples from diverse sources, including fact-checking sites (Snopes), social media (Twitter, Reddit), and official news outlets (CNN, NYT).

Key features include:

  • Knowledge Enrichment: Entities are linked to Wikidata, providing essential context for disambiguation (e.g., distinguishing Apple Inc. from the fruit).
  • Fine-Grained Taxonomy: News is categorized into six types: Real, Text-based fake, Image-based fake, Text-image inconsistency, Content-knowledge inconsistency, and Others.
  • Multi-Domain Scope: The dataset spans six topics (Politics, Entertainment, Business, Health, Society, Conflict) and eight platforms.

KEAN: Knowledge-Enhanced Domain Adaptation

To leverage this dataset, we developed the Knowledge-Enhanced Domain Adaptation Network (KEAN). KEAN is designed to maintain robustness across varying news domains by focusing on invariant fabrication characteristics rather than domain-specific jargon.

Figure 4: The KEAN model architecture, integrating multimodal encoders with external knowledge graph embeddings.

The architecture consists of:

  • Multimodal Encoders: Utilizing CLIP to extract features from text and images.
  • Knowledge Graph Encoder: Using TransE to embed Wikidata subgraphs, providing the model with factual "common sense."
  • Adversarial Training: A Domain-Adversarial Neural Network (DANN) approach ensures the model learns domain-invariant features.
  • Label Shift Re-weighting: A module to account for varying fake news priors across different platforms (e.g., higher on social media vs. official news).

Key Results and Insights

Extensive experiments on FineFake reveal several critical insights:

  • SOTA Performance: KEAN achieved a state-of-the-art F1-score of 0.797 in binary classification, outperforming standard multimodal models.
  • The Value of Knowledge: Incorporating external knowledge consistently improved performance by 1-3% across various metrics.
  • Cross-Domain Robustness: In "Dual Domain Adaptation" tasks (changing both topic and platform), KEAN maintained significantly higher accuracy than baselines, proving the effectiveness of its adversarial training.
  • Platform-Specific Strategies: Analysis shows that social media platforms like Twitter have a higher proportion of visual fakes compared to official news sources.

Conclusion

FineFake represents a significant step toward more realistic and interpretable fake news detection. By moving beyond binary labels and integrating external knowledge, the benchmark allows for a deeper understanding of fabrication strategies across diverse domains. Future work will focus on expanding the dataset to include video content and scaling the knowledge alignment process.

The FineFake project is publicly accessible at https://github.com/Accuser907/FineFake.

Summary updated

Abstract

Existing benchmarks for fake news detection have significantly contributed to the advancement of models in assessing the authenticity of news content. However, these benchmarks typically focus solely on news pertaining to a single semantic topic or originating from a single platform, thereby failing to capture the diversity of multi-domain news in real scenarios. In order to understand fake news across various domains, the external knowledge and fine-grained annotations are indispensable to provide precise evidence and uncover the diverse underlying strategies for fabrication, which are also ignored by existing benchmarks. To address this gap, we introduce a novel multi-domain knowledge-enhanced benchmark with fine-grained annotations, named \textbf{FineFake}. FineFake encompasses 16,909 data samples spanning six semantic topics and eight platforms. Each news item is enriched with multi-modal content, potential social context, semi-manually verified common knowledge, and fine-grained annotations that surpass conventional binary labels. Furthermore, we formulate three challenging tasks based on FineFake and propose a knowledge-enhanced domain adaptation network. Extensive experiments are conducted on FineFake under various scenarios, providing accurate and reliable benchmarks for future endeavors. The entire FineFake project is publicly accessible as an open-source repository at \url{https://github.com/Accuser907/FineFake}.
BibTeX
@article{zhou2024finefake,
  author = {Ziyi Zhou and Xiaoming Zhang and Litian Zhang and Jiacheng Liu and Xi Zhang and Chaozhuo Li},
  title = {{FineFake: A Knowledge-Enriched Dataset for Fine-Grained Multi-Domain Fake News Detection}},
  journal = {Information Fusion},
  year = {2024},
  doi = {10.1016/j.inffus.2026.104253},
}
Figure