Querying Across Genres for Medical Claims in News

Chaoyuan Zuo, Narayan Acharya, Ritwik BanerjeeEMNLP 2020 · DOI

Introduction

This research addresses the "cross-genre" medical information retrieval task: automatically identifying the specific scientific research paper that supports a medical claim found in a news headline. News outlets often simplify complex findings into catchy headlines like "Tea drinkers live longer," frequently omitting direct links to the source study. This leaves readers and fact-checkers with the difficult task of searching through massive databases like PubMed to verify claims.

The challenge is primarily linguistic and structural. News articles use "lay terms" (e.g., "heart disease"), while research abstracts use "specialist language" (e.g., "atherosclerotic cardiovascular disease"). Furthermore, the sheer scale of scientific literature makes finding the exact supporting paper a "needle in a haystack" problem.

Methods

The authors propose a two-stage pipeline to bridge the gap between layperson "news-speak" and technical "science-speak":

  1. Candidate Selection: To reduce the search space from hundreds of thousands of abstracts to a manageable subset, the authors employed BM25+, a classic token-based ranking function. Extensive preprocessing—including lowercasing, stemming, and expanding medical abbreviations—was found to be critical for this stage.
  2. Transformer-based Re-ranking: The top 500 candidates were then re-ranked using domain-specific Transformer models, primarily Bio+Clinical BERT (BC-BERT). To address data imbalance (1 positive vs. 499 negative samples), the authors used NLP data augmentation to generate additional positive training examples.

Datasets

The study introduced a novel dataset consisting of 5,034 news headlines collected over 18 months from various medical news RSS feeds. These were paired with 4,566 "ground truth" research abstracts cited in the articles. To simulate a realistic search environment, over 217,000 "spurious" abstracts from PubMed Central were added as negative samples.

Distribution of news sources in the dataset.

Key Results and Insights

The initial filtering stage using BM25+ achieved a Recall@500 of 0.830, successfully capturing the correct paper in the top 500 list 83% of the time. In the subsequent re-ranking stage, the Bio+Clinical BERT model significantly outperformed general-purpose models.

Performance comparison of various re-ranking models on the test set.

As shown in the table above, the BC-BERT model (using 20 augmented positive samples) achieved a Recall@1 of 0.626 and a Mean Reciprocal Rank (MRR) of 0.695. Notably, the hybrid approach allowed the Transformer model to retrieve the correct abstract in 33% of cases where the initial BM25+ filter had failed, highlighting the value of semantic understanding over simple keyword matching.

Discussion and Limitations

A key insight from the study is the "Middle Region" hypothesis: scientific findings are typically concentrated in the "Background" or "Conclusion" sections of an abstract. Encoding only the first and last three sentences actually improved performance. However, the system still struggles with generic claims (e.g., "Research could help design better flu vaccines") that lack specific "hooks" for retrieval. Furthermore, the model's success depends on the headline being a faithful representation of the study; "clickbait" headlines remain a significant hurdle for automated verification.

Summary updated

BibTeX
@inproceedings{zuo2020querying,
  author = {Chaoyuan Zuo and Narayan Acharya and Ritwik Banerjee},
  title = {{Querying Across Genres for Medical Claims in News}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2020},
  doi = {10.18653/v1/2020.emnlp-main.139},
}
Figure