CiteSum: Citation Text-guided Scientific Extreme Summarization and Low-resource Domain Adaptation

Yuning Mao, Ming Zhong, Jiawei Han

Introduction

In the rapidly expanding landscape of scientific literature, researchers face a significant challenge in quickly identifying the core contributions of new papers. While abstracts provide summaries, they are often lengthy and contain excessive background information. "Extreme summarization," or the generation of single-sentence "TLDRs," has emerged as a solution. However, the development of such models is hindered by a "scalability crisis": high-quality TLDRs typically require expensive expert human annotation. For instance, the previously dominant SciTLDR dataset contains only about 2,000 training examples.

This work introduces CiteSum, a massive, automatically curated dataset for scientific extreme summarization. By leveraging the natural summaries found within citation sentences, the authors generated nearly 100,000 training pairs, providing a 30-fold increase in data scale over previous benchmarks.

Methods: Citations as Natural Summaries

The fundamental insight of this research is that when a scientist cites a paper, particularly within a "Related Work" section, they often provide a concise, one-sentence summary of that paper's primary contribution. To harness this "crowdsourced" data, the authors developed a multi-stage pipeline:

  1. Source Selection: Data was extracted from the Semantic Scholar Open Research Corpus (S2ORC).
  2. Section Filtering: The pipeline specifically targeted citation sentences in "Related Work" sections to ensure the text described actual contributions rather than peripheral details.
  3. Quality Control: Filters were applied to select sentences citing only a single paper. ROUGE similarity metrics ensured significant overlap between the citation text and the cited paper's abstract.
  4. The CITES Model: A BART-large model was fine-tuned on CiteSum. To address the stylistic difference between citations (e.g., "Author et al. proposed...") and TLDRs, a "prompting" technique was used during decoding to force the model to begin summaries with "This paper...", aligning the output with standard TLDR formats.

Datasets

The primary contribution is the CiteSum dataset, which contains 92,946 citation-summary pairs. As shown in the comparison below, CiteSum significantly exceeds the scale of previous scientific summarization datasets like SciTLDR while maintaining a focus on high-quality, contribution-oriented summaries.

Comparison of CiteSum with existing summarization datasets.

Key Results

The resulting model, CITES, demonstrates exceptional performance across multiple benchmarks:

  • State-of-the-Art Performance: On the SciTLDR benchmark, CITES achieved SOTA results. Notably, in a zero-shot setting, it outperformed many fully-supervised models. With only 128 human-annotated examples (few-shot), it established a new performance ceiling.
  • Cross-Domain Transferability: Despite being trained on scientific text, CITES showed remarkable "zero-shot" transfer to the news domain, improving ROUGE-1 scores on the XSum dataset by +7.2 points over the base BART model.
  • Human Validation: Blind human evaluations confirmed the quality of the automated dataset, with 80% of CiteSum-derived summaries rated as "mostly accurate" or "accurate."
Performance of CITES on the SciTLDR benchmark compared to supervised and zero-shot baselines.

Limitations and Future Work

While CiteSum offers a massive leap in data availability, it is currently skewed toward Computer Science (representing approximately 80% of the papers). This bias arises because "Related Work" sections are a staple of CS literature but may be structured differently in other disciplines. Additionally, approximately 10% of the automatically extracted summaries were rated as "misleading" by human annotators, indicating a need for more refined filtering algorithms. Future research will explore incorporating domain-specific taxonomies and terminology to further enhance the factual precision of these ultra-short summaries.

Summary updated

BibTeX
@inproceedings{mao2022citesum,
  author = {Yuning Mao and Ming Zhong and Jiawei Han},
  title = {{CiteSum: Citation Text-guided Scientific Extreme Summarization and Low-resource Domain Adaptation}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2022},
  doi = {10.18653/v1/2022.emnlp-main.750},
}
Figure