An Empirical Assessment of the Qualitative Aspects of Misinformation in Health News

Chaoyuan Zuo, Qi Zhang, Ritwik BanerjeeNAACL 2021

TL;DR

This paper tackles the critical issue of health news misinformation by moving beyond simple "true/false" labels. The authors introduce a novel dataset and evaluation framework to assess the qualitative aspects of health news accuracy, revealing that even reputable sources often fall short on crucial criteria. Their findings highlight the complexity of medical misinformation and the need for more sophisticated detection methods.

What Problem Does This Paper Address?

In an era where individuals increasingly turn to online health news for information, the proliferation of low-quality content poses a significant public health risk. While fact-checking efforts have focused on identifying outright false claims, medical news presents a unique challenge. Misinformation in this domain often arises not from outright falsehoods, but from a lack of nuance, incomplete understanding of research processes, or the sensationalization of findings. This can lead to "disease-mongering" or misrepresenting the benefits and risks of interventions. Existing work has largely overlooked these qualitative aspects, focusing instead on binary true/false classifications. This paper aims to bridge this gap by empirically assessing the qualitative accuracy of health news articles based on criteria deemed important by medical experts and journalists.

Methods

The authors frame their problem as a series of supervised classification tasks, evaluating articles against specific review criteria. They compare two main approaches:

  • Feature-based models: These models utilize traditional Natural Language Processing (NLP) techniques, including TF-IDF for feature encoding. They employ Support Vector Machines (SVM) and Gradient Boosting (GB) as learning algorithms. To address class imbalance, they use ADASYN (Adaptive Synthetic Sampling) and employ recursive feature elimination to reduce dimensionality.
  • Transformer-based models: Leveraging state-of-the-art deep learning architectures, they experiment with BERT, XLNet, RoBERTa, ALBERT, DistilBERT, and Longformer. For these models, they use random undersampling to mitigate class imbalance.

Datasets

The core of this research is a newly curated dataset derived from Health News Review (Schwitzer, 2006). This dataset comprises:

  • 1,119 health news articles: This includes 740 news stories and 379 public relations (PR) news releases.
  • Systematic Reviews: Each article is paired with detailed reviews based on ten domain-specific criteria developed by medical experts and health journalists. These criteria assess aspects like the discussion of costs, benefits, harms, evidence quality, disease-mongering, source independence, and novelty.

The authors note that approximately 30% of the original online articles were no longer accessible, and some were behind paywalls. For their empirical experiments, they exclude criteria that require highly specialized medical knowledge (criteria 5 and 9) and those that differ between news stories and PR releases (criteria 6 and 10), focusing on the remaining eight criteria.

Key Insights and Ideas

  • Beyond True/False: The paper strongly advocates for moving beyond simple binary classifications for health news. The qualitative aspects of reporting are crucial for accurate public understanding.
  • Complexity of Medical Misinformation: Misinformation in health news is often subtle, arising from context, nuance, or sensationalism rather than outright fabrication.
  • Expert-Defined Criteria are Key: The use of criteria developed by domain experts (journalists and medical professionals) provides a robust framework for evaluating health news quality.
  • Reputation Isn't Everything: Even articles from well-known news organizations frequently fall short of these qualitative benchmarks, underscoring the pervasive nature of the problem.
  • Challenging Task: Accurately assessing these qualitative aspects is a difficult task, even for human readers without specialized training.

Key Results

The experiments revealed that:

  • Class Imbalance is Prevalent: For most of the evaluated criteria, less than half of the news articles were deemed satisfactory.
  • Gradient Boosting Shines: Gradient Boosting models, particularly when combined with oversampling and feature selection, generally outperformed other models on several criteria (1, 3, 4, and 8).
  • Transformer Models Struggle (Initially): Feature-based models, especially Gradient Boosting, outperformed transformer-based models on the first four criteria. The authors attribute this, in part, to the dataset size after undersampling and the "inverted pyramid" structure of news articles, where essential information is in the lead.
  • Longformer's Limitations: Even Longformer, designed for longer sequences, did not achieve significantly better performance, suggesting that the core information might be concentrated early in articles.
  • Criteria Specificity Matters: The first four criteria, which are more specific and potentially answerable by token-level searching (e.g., cost of intervention), showed better performance compared to more nuanced criteria.

Limitations and Future Work

  • Dataset Size: The authors acknowledge that the dataset size, even after combining sources, might be a limiting factor for transformer-based models, especially after undersampling.
  • Exclusion of Criteria: The exclusion of criteria 5 and 9 (requiring specialized medical knowledge) and criteria 6 and 10 (differing for news stories/PR) means the current evaluation doesn't cover all aspects of the original review framework.
  • Future Directions: The authors hope their work will inspire the collaborative creation of larger-scale datasets for qualitative health news analysis, potentially extending to other domains. They emphasize the need for continued research into more sophisticated methods for detecting nuanced misinformation.

Summary updated

BibTeX
@inproceedings{zuo2021empirical,
  author = {Chaoyuan Zuo and Qi Zhang and Ritwik Banerjee},
  title = {{An Empirical Assessment of the Qualitative Aspects of Misinformation in Health News}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2021},
}
Figure