A Fine-grained citation graph for biomedical academic papers: the finding-citation graph

Yuan Liang, Massimo Poesio, Roonak RezvaniACL 2024

TL;DR

This paper introduces the "finding-citation graph" (FCG), a novel way to represent scientific literature that goes beyond traditional citation networks by explicitly modeling research findings. They developed a pipeline to construct this graph from biomedical literature, enabling deeper analysis of how scientific discoveries evolve and are cited.

What Problem Does This Paper Address?

The sheer volume of biomedical literature makes it challenging for researchers to keep up with new discoveries and assess research output effectively. While citation networks help link papers, they don't capture the specific findings being cited. Existing academic databases often lack the granular detail needed to understand what is being cited and why. This gap hinders a deeper understanding of scientific progress at the level of individual research outcomes.

Methods

The core of this work is a pipeline to construct the Finding-Citation Graph (FCG). This pipeline consists of three main modules:

  • Finding Identification: This module identifies sentences within a paper that describe research findings. It treats this as a sentence classification task, leveraging a fine-tuned RoBERTa model trained on a large dataset of PubMed abstracts. The model identifies sentences in "result" or "conclusion" sections as potential finding sentences.
  • Citation Sentence Extraction: This module identifies sentences in the full text of an article that contain citations. It uses a regular expression-based approach to link citation sentences to their cited papers.
  • Graph Construction: This module combines the outputs from the previous two to build the FCG. The graph represents papers and findings as nodes, with edges indicating relationships like a paper containing a finding, or a citation sentence linking to a finding in another paper.

Datasets

  • Europe PMC: The primary dataset used for constructing the finding-citation graph. The pipeline processes articles in XML format from Europe PMC.
  • PubMed Abstracts: Used for training the finding identification module. The authors curated a dataset of approximately 206,000 PubMed abstracts.
  • PubMed 200K RCT: Mentioned as a dataset used in prior work for sentence classification in medical abstracts.
  • CITEWORTH dataset: Mentioned in the context of cite-worthiness detection, used for evaluating models.

Key Insights and Ideas

  • Findings are key: The paper argues that research findings, not just entire papers, are the fundamental units of scientific progress and should be explicitly modeled in citation analysis.
  • Beyond the abstract: A significant portion of findings (nearly 44%) are not mentioned in the abstract, highlighting the importance of analyzing full-text articles for a comprehensive understanding.
  • Citation context matters: The paper emphasizes that understanding why a paper is cited (i.e., the specific finding or information being referenced) is crucial for evaluating research impact and evolution.
  • Heterogeneous graph: The FCG is a heterogeneous graph, allowing for richer representations and analyses than traditional homogeneous citation networks.

Key Results

  • Finding Identification Accuracy: The finding identification module achieved 91% accuracy for identifying finding sentences and 80% accuracy for identifying multiple findings per paper.
  • Citation Sentence Extraction Accuracy: The citation sentence extraction module achieved 90% F1-score on paper-level evaluation and 97.7% accuracy on tuple-level evaluation.
  • Graph Size: The constructed finding-citation graph from Europe PMC contains 14.25 million nodes (4.25 million paper nodes and 10 million finding nodes) and 76 million edges (66 million representing citation relationships).
  • Similarity Score Distribution: Analysis of similarity scores between citation sentences and cited findings revealed that about half of citation sentences contain less information about the cited findings, often serving more for literature review or background.

Limitations and Future Work

  • Citation Span: The current method extracts sentences with citation markers but doesn't consider the specific span of text within the sentence that refers to the citation, which could lead to errors.
  • Closed Dataset: The graph is built using Europe PMC, meaning citations and articles not present in this dataset are excluded, leading to an incomplete graph.
  • Future Work: The authors plan to use the FCG to assess research findings from a citation perspective, pinpoint future research directions at the finding level, and improve graph optimization strategies. They also explored using LLMs like Gemma as a substitute for fine-tuned models, finding comparable performance but opting for RoBERTa due to resource considerations.

Summary updated

BibTeX
@inproceedings{liang2024fine,
  author = {Yuan Liang and Massimo Poesio and Roonak Rezvani},
  title = {{A Fine-grained citation graph for biomedical academic papers: the finding-citation graph}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2024},
}
Figure