TL;DR
Researchers have released FakeClaim, the first publicly available dataset specifically targeting misinformation surrounding the 2023 Israel-Hamas war. By combining YouTube video metadata and user comments with ground-truth data from 60 fact-checking organizations, they achieved an 87% Macro F1 score in identifying fake content, demonstrating that user engagement is a powerful signal for automated debunking.
What Problem Does This Paper Address?
During geopolitical conflicts, the volume of misinformation—ranging from mislabeled old footage to AI-generated "deepfakes"—skyrockets. While fact-checking organizations work tirelessly to debunk these claims, the sheer scale of social media makes manual verification a losing battle.
The authors identify a critical gap: most existing fake news datasets are limited to a single platform (like X/Twitter) or lack the "connective tissue" between a specific claim, the social media post propagating it, and the expert evidence used to debunk it. This paper introduces a multi-platform dataset and a methodology to bridge these elements, specifically focusing on the challenging medium of video on YouTube.
Methods
The researchers employed a multi-stage pipeline to build and test their dataset:
- Data Collection (AMUSED Framework): They used the AMUSED framework to scrape 1,370 fact-checked articles from 60 organizations globally.
- Claim Extraction: They extracted 1,499 specific claims across platforms like Facebook, X, YouTube, TikTok, Instagram, and Telegram.
- Feature Engineering: For the YouTube subset, they collected video titles, metadata, and over 166,000 user comments. They also calculated sentiment polarity for comments using
TextBlobto capture the "emotional temperature" of the discussion. - Classification: They benchmarked three pre-trained embedding models:
- GN-Swivel-20D: A 20-dimensional model trained on Google News.
- Universal Sentence Encoder (USE): A high-dimensional model designed for sentence-level semantics.
- RoBERTa: A robustly optimized BERT approach.
The research framework is visualized in the following figure:
Datasets
The primary contribution is the FakeClaim dataset, which includes:
- 1,499 claims sourced from 30 different languages.
- A distribution across major platforms, with Facebook and YouTube being the most prominent sources of debunked claims.
- A specific YouTube classification subset consisting of 756 unique videos (301 fake, 455 "likely real") and 166,645 comments.
The "real" class was generated by searching for keywords related to the conflict and selecting videos that had not been flagged by fact-checkers.
Key Insights and Ideas
- The Power of the Crowd: One of the most significant findings is that user comments are highly predictive. When users argue or provide counter-evidence in the comments, they create a "signal" that machine learning models can use to flag potential misinformation.
- Context is King: Models performed significantly better when the "Claim" (the expert debunking text) was concatenated with the video title and comments. This suggests that providing models with a "gold standard" reference drastically improves veracity prediction.
- Platform Diversity: Misinformation isn't siloed. The dataset shows how the same fake claim often migrates across Facebook, YouTube, and TikTok, necessitating a platform-agnostic approach to data collection.
Key Results
The Universal Sentence Encoder (USE) emerged as the clear winner. When the model was fed a combination of the Video Title + User Comments + Fact-Checked Claims, it achieved a Macro F1 score of 87%.
Interestingly, even without the expert claims—using only the video title and user comments—the USE model still managed an F1 score of 80%. This highlights the utility of user engagement as a standalone feature for early detection. In contrast, RoBERTa struggled in this specific task, yielding significantly lower scores (F1 ~72%).
Limitations and Future Work
The authors acknowledge several hurdles:
- The "Real" Class Problem: While the "fake" videos were verified by journalists, the "real" videos were gathered via keyword search and were not manually fact-checked. This introduces a potential bias where some "real" videos might actually be undetected fakes.
- Standardization: There is no global standard for how fact-checking organizations publish their data, making the extraction process highly manual and difficult to scale.
- Expansion: Future work aims to incorporate more diverse data from platforms like TikTok and to refine the labeling of "correct" information to create a more balanced and verified ground truth.
The dataset and models are available for the research community on GitHub.
Summary updated