FakeClaim: A Multiple Platform-driven Dataset for Identification of Fake News on 2023 Israel-Hamas War

Gautam Kishore Shahi, Amit Kumar Jaiswal, Thomas MandlarXiv 2024 · arXiv

TL;DR

Researchers have released FakeClaim, the first publicly available dataset specifically targeting misinformation surrounding the 2023 Israel-Hamas war. By combining YouTube video metadata and user comments with ground-truth data from 60 fact-checking organizations, they achieved an 87% Macro F1 score in identifying fake content, demonstrating that user engagement is a powerful signal for automated debunking.

What Problem Does This Paper Address?

During geopolitical conflicts, the volume of misinformation—ranging from mislabeled old footage to AI-generated "deepfakes"—skyrockets. While fact-checking organizations work tirelessly to debunk these claims, the sheer scale of social media makes manual verification a losing battle.

The authors identify a critical gap: most existing fake news datasets are limited to a single platform (like X/Twitter) or lack the "connective tissue" between a specific claim, the social media post propagating it, and the expert evidence used to debunk it. This paper introduces a multi-platform dataset and a methodology to bridge these elements, specifically focusing on the challenging medium of video on YouTube.

Methods

The researchers employed a multi-stage pipeline to build and test their dataset:

  1. Data Collection (AMUSED Framework): They used the AMUSED framework to scrape 1,370 fact-checked articles from 60 organizations globally.
  2. Claim Extraction: They extracted 1,499 specific claims across platforms like Facebook, X, YouTube, TikTok, Instagram, and Telegram.
  3. Feature Engineering: For the YouTube subset, they collected video titles, metadata, and over 166,000 user comments. They also calculated sentiment polarity for comments using TextBlob to capture the "emotional temperature" of the discussion.
  4. Classification: They benchmarked three pre-trained embedding models:
  5. GN-Swivel-20D: A 20-dimensional model trained on Google News.
  6. Universal Sentence Encoder (USE): A high-dimensional model designed for sentence-level semantics.
  7. RoBERTa: A robustly optimized BERT approach.

The research framework is visualized in the following figure:

Fig. 1. The research framework used for extracting the dataset and classification prob- lem.

Datasets

The primary contribution is the FakeClaim dataset, which includes:

  • 1,499 claims sourced from 30 different languages.
  • A distribution across major platforms, with Facebook and YouTube being the most prominent sources of debunked claims.
  • A specific YouTube classification subset consisting of 756 unique videos (301 fake, 455 "likely real") and 166,645 comments.
Table 2. Distribution of claim debunked from different social media platforms(* some fact-checked article debunks multiple claims from different platforms)

The "real" class was generated by searching for keywords related to the conflict and selecting videos that had not been flagged by fact-checkers.

Key Insights and Ideas

  • The Power of the Crowd: One of the most significant findings is that user comments are highly predictive. When users argue or provide counter-evidence in the comments, they create a "signal" that machine learning models can use to flag potential misinformation.
  • Context is King: Models performed significantly better when the "Claim" (the expert debunking text) was concatenated with the video title and comments. This suggests that providing models with a "gold standard" reference drastically improves veracity prediction.
  • Platform Diversity: Misinformation isn't siloed. The dataset shows how the same fake claim often migrates across Facebook, YouTube, and TikTok, necessitating a platform-agnostic approach to data collection.

Key Results

The Universal Sentence Encoder (USE) emerged as the clear winner. When the model was fed a combination of the Video Title + User Comments + Fact-Checked Claims, it achieved a Macro F1 score of 87%.

Interestingly, even without the expert claims—using only the video title and user comments—the USE model still managed an F1 score of 80%. This highlights the utility of user engagement as a standalone feature for early detection. In contrast, RoBERTa struggled in this specific task, yielding significantly lower scores (F1 ~72%).

Table 3. Results with pre-trained word embedding models for classification of Fake claims. Model performance is based on the F1 score, best models are in bold.

Limitations and Future Work

The authors acknowledge several hurdles:

  • The "Real" Class Problem: While the "fake" videos were verified by journalists, the "real" videos were gathered via keyword search and were not manually fact-checked. This introduces a potential bias where some "real" videos might actually be undetected fakes.
  • Standardization: There is no global standard for how fact-checking organizations publish their data, making the extraction process highly manual and difficult to scale.
  • Expansion: Future work aims to incorporate more diverse data from platforms like TikTok and to refine the labeling of "correct" information to create a more balanced and verified ground truth.

The dataset and models are available for the research community on GitHub.

Summary updated

Abstract

We contribute the first publicly available dataset of factual claims from different platforms and fake YouTube videos on the 2023 Israel-Hamas war for automatic fake YouTube video classification. The FakeClaim data is collected from 60 fact-checking organizations in 30 languages and enriched with metadata from the fact-checking organizations curated by trained journalists specialized in fact-checking. Further, we classify fake videos within the subset of YouTube videos using textual information and user comments. We used a pre-trained model to classify each video with different feature combinations. Our best-performing fine-tuned language model, Universal Sentence Encoder (USE), achieves a Macro F1 of 87\%, which shows that the trained model can be helpful for debunking fake videos using the comments from the user discussion. The dataset is available on Github\footnote{https://github.com/Gautamshahi/FakeClaim}
BibTeX
@misc{shahi2024fakeclaim,
  author = {Gautam Kishore Shahi and Amit Kumar Jaiswal and Thomas Mandl},
  title = {{FakeClaim: A Multiple Platform-driven Dataset for Identification of Fake News on 2023 Israel-Hamas War}},
  year = {2024},
  eprint = {2401.16625},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2401.16625},
}
Figure