Abstract
Health misinformation poses unique, life-threatening risks, yet existing datasets often lack the depth required for robust detection. We introduce FakeHealth, a multi-dimensional repository that bridges this gap by integrating expert-reviewed news articles with extensive social media engagement data and user network information. By providing rich context beyond simple text, FakeHealth enables the development of accurate and explainable models for identifying health-related misinformation.
Introduction
While misinformation in politics and sports is frequently studied, fake health news carries immediate physical dangers. A patient might forgo life-saving treatment like chemotherapy in favor of unproven remedies like ginger based on viral, but false, claims. Detecting such misinformation is challenging due to the specialist knowledge required and the "half-true" nature of many claims, which often distort risks or confuse correlation with causation. To address these challenges, researchers need datasets that capture not just the content of an article, but also its social spread and expert-provided ground truth.
Methodology
The authors developed a comprehensive pipeline to collect data centered on HealthNewsReview.org, where medical experts evaluate health news based on ten specific criteria, such as cost discussion and novelty.
The pipeline consists of four primary stages:
- Expert Reviews: Crawling 13 years of detailed evaluations.
- Content Scraping: Retrieving original news stories and press releases.
- Social Engagement: Utilizing Twitter’s API to capture tweets, replies, and retweets.
- User Network Mapping: Mapping profiles and follower lists to understand the demographics and behaviors of users spreading the news.
The FakeHealth Dataset
The repository is divided into two distinct datasets:
- HealthStory: Contains 1,690 stories from mainstream outlets like Reuters and CNN.
- HealthRelease: Includes 606 news releases from universities and research institutes.
In total, the dataset encompasses over 500,000 tweets and detailed profiles for 27,000 users, offering a significant upgrade in social context compared to previous benchmarks like LIAR or BuzzFeedNews.
Exploratory Analysis and Insights
Analysis of the FakeHealth data revealed several critical signals for misinformation detection:
- Source Credibility: There is a marked disparity in reliability among publishers. For example, the Associated Press maintained a high average rating (~3.87/5), whereas WebMD scored significantly lower (~2.82/5).
- The "Bot" Signature: Users spreading fake health news showed a higher likelihood of being automated accounts (bots), often characterized by sudden bursts of activity rather than organic growth.
- Topic Consistency: Real and fake news often overlap in topics (e.g., cancer, Alzheimer’s), suggesting that simple topic-based classification is insufficient.
- Sentiment and Crowdsourcing: Social media replies to real news are generally more positive, while fake news often attracts neutral or negative skepticism from the community.
Experimental Results
The authors evaluated several baseline models to demonstrate the dataset's utility. The Social Article Fusion (SAF) model, which incorporates reply sequences and user embeddings, achieved the best performance on the HealthRelease dataset with an AUC of 0.809. For mainstream news (HealthStory), a CNN focusing on linguistic markers in the article text performed best (AUC 0.814). Crucially, the models showed potential for explainability, predicting which specific expert criteria a news piece failed to meet.
Conclusion and Future Work
FakeHealth provides a robust foundation for future research into health misinformation. While limited by Twitter's API rate caps and currently underutilizing visual data, the repository opens doors for multi-modal analysis and the creation of "Health Knowledge Graphs." Such tools could eventually automate the verification of health claims against established medical facts, providing users with clear explanations of why a story may be suspicious.
Summary updated