Measuring the relative importance of full text sections for information retrieval from scientific literature.

Lana Yeganova, Won Gyu Kim, Donald Comeau, W John Wilbur, Zhiyong LuNAACL 2021

TL;DR

This paper tackles the challenge of effectively using the full text of scientific articles for information retrieval, beyond just the abstract. They introduce a method to weigh different sections of a paper based on their predicted relevance, showing that this nuanced approach significantly improves search results compared to traditional methods.

What Problem Does This Paper Address?

The explosion of full-text scientific literature presents a massive opportunity for improving information retrieval. While abstracts offer a concise summary, they can sometimes miss crucial details or context. Integrating full text could boost recall and help users find relevant papers that might otherwise be missed. However, simply concatenating abstracts with full text often degrades precision, as full texts can be lengthy, discuss multiple subtopics, or mention information in a less direct way (e.g., conjecture). The core problem is how to intelligently leverage the diverse sections within a full-text document to accurately predict a document's relevance to a user's query, rather than treating all parts of the text equally.

Methods

The authors propose a novel approach to weigh the importance of different sections within a scientific article for information retrieval. Their method involves:

  • BM25 Scoring: They start by using the standard BM25 algorithm to score the relevance of query tokens within individual paragraphs of a document.
  • Section-Level Scoring: For each section type (e.g., Abstract, Introduction, Results, Discussion), they identify the maximum BM25 score of a query token within any paragraph of that section. This is done to avoid bias from section length.
  • Probabilistic Transformation (PAV): Crucially, they recognize that a raw BM25 score doesn't have the same meaning across different sections. A high BM25 score in the "Results" section might indicate higher importance than the same score in the "Methods" section. To address this, they use the Pool Adjacent Violators (PAV) algorithm to transform these section-specific BM25 scores into probabilities of a document being relevant. This probabilistic transformation allows for direct comparison of scores across sections.
  • Log Odds Scoring: They then combine these section-specific probabilities using a log-odds framework. This framework is derived from Bayes' Theorem and aims to provide a single, comparable score for a query token across all sections of a document.
  • Combining Section Scores: Finally, they explore two strategies for combining these log-odds scores from different sections to create an overall document score:
  • Sum LogOdds: The sum of log-odds scores from all relevant sections.
  • Max LogOdds: The maximum log-odds score from any section.

Datasets

The authors evaluate their methods on two datasets:

  • TREC 2007 Genomics Dataset: A standard benchmark for information retrieval, consisting of 36 queries and 162,259 full-text articles. Human relevance judgments are provided at the paragraph level.
  • PubMed User Click Dataset: A newly introduced dataset constructed from PubMed queries and user click data. The assumption is that clicked documents are relevant. This dataset was split into training and testing sets. A subset of this dataset, referred to as "Set_FT," specifically includes queries where all retrieved documents have full-text available in PubMed Central, allowing for a direct measurement of full-text benefits.

Key Insights and Ideas

  • Section Matters: The most significant insight is that raw BM25 scores are not directly comparable across different sections of a scientific paper. The context of where a term appears (e.g., Results vs. Methods) drastically changes its predictive power for relevance.
  • Probabilistic Weighting is Key: Transforming BM25 scores into probabilities using PAV is essential for creating a unified and comparable measure of a token's importance across diverse sections.
  • Log Odds as a Unified Score: The log-odds framework effectively combines these section-specific probabilities into a single, meaningful score that reflects a token's overall contribution to document relevance.
  • Sum vs. Max: For combining section scores, the "Sum LogOdds" approach generally showed a slight advantage over "Max LogOdds," suggesting that the cumulative evidence from multiple relevant sections is more informative than the single strongest section.

Key Results

  • Significant Improvement: The proposed LogOdds scoring methods (both Sum and Max variants) significantly outperformed raw BM25 scoring on both the PubMed Click Dataset and the TREC Genomics Dataset.
  • PubMed Click Dataset: On this dataset, Sum LogOdds using full text demonstrated a statistically significant improvement over abstract-only scoring.
  • TREC Genomics Dataset: Here, full-text LogOdds performance was comparable to abstract-only scoring, but still significantly better than raw BM25.
  • Statistical Significance: Pairwise statistical tests confirmed that the differences in performance between the LogOdds methods and raw BM25 methods were statistically significant, with a few exceptions.

Limitations and Future Work

  • Single Token Analysis: The current work focuses on scoring individual query tokens. The authors acknowledge that extending this analysis to full queries, which involve multiple tokens and their interactions, is a crucial next step.
  • Independence Assumption: The log-odds combination relies on a naive Bayes assumption of independence between section scores, which may not always hold true in practice.
  • Dataset Specifics: While the PubMed Click dataset offers valuable insights into user behavior, its semi-automatic construction might have limitations. The TREC Genomics dataset, while standard, has a fixed set of queries.

The authors plan to extend their single-token analysis to full queries, which will be a significant step towards building more effective full-text retrieval systems for scientific literature.

Summary updated

BibTeX
@inproceedings{yeganova2021measuring,
  author = {Lana Yeganova and Won Gyu Kim and Donald Comeau and W John Wilbur and Zhiyong Lu},
  title = {{Measuring the relative importance of full text sections for information retrieval from scientific literature.}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2021},
}
Figure