A Summarization System for Scientific Documents

Shai Erera, Michal Shmueli-Scheuer, Guy Feigenblat, Ora Peled Nakash, Odellia Boni, Haggai Roitman, Doron Cohen, Bar Weiner, Yosi Mass, Or Rivlin, Guy Lev, Achiya Jerbi, Jonathan Herzig, Yufang Hou, Charles Jochim, Martin Gleize, Francesca Bonin, David Konopnicki

Abstract

Shai Erera et al. present the IBM Science Summarizer, a system designed to help researchers navigate the overwhelming flood of new scientific publications by generating structured, section-by-section summaries of Computer Science papers. By treating a paper as a collection of distinct parts rather than a flat text block, the system provides more precise and useful overviews that outperformed traditional "flat" summarization in human expert trials.

Introduction

The "information overload" in academia is a well-known crisis. With thousands of papers published weekly on platforms like arXiv and the ACL Anthology, researchers struggle to stay current. While search engines help find papers and abstracts provide a brief overview, there is a lack of tools that help users quickly "gist" the actual content of a paper without reading the whole PDF. Existing text summarizers often fail because they are designed for short news articles, whereas scientific papers are long, highly structured, and use specialized language that varies significantly between sections like "Introduction" and "Methodology."

System Architecture and Methods

The IBM Science Summarizer employs a multi-stage pipeline to process and summarize scientific documents at scale, as illustrated in the architecture diagram below.

The IBM Science Summarizer pipeline: (a) offline ingestion and entity extraction, and (b) online query execution and summarization.
  1. Ingestion & Parsing: Using Science-Parse, the system converts PDFs into structured JSON, identifying titles, authors, and—crucially—distinct sections.
  2. Entity Extraction (TDM): The system identifies Tasks, Datasets, and Metrics (TDM). This is framed as a textual entailment task, where the paper content is treated as "text" and potential TDM triples as a "hypothesis" to see if the paper supports that specific research configuration.
  3. Section-Based Summarization: Instead of one summary for the whole paper, the system generates a standalone summary for each section (e.g., "Introduction" or "Experiments").
  4. Extractive Algorithm: An unsupervised approach based on the Cross Entropy (CE) method selects the best sentences by optimizing for five criteria: query saliency, entity coverage, diversity, text coverage, and sentence length.

Evaluation and Results

The system was evaluated on a corpus of 270,000 papers from arXiv and the ACL Anthology. A key part of the study involved a human evaluation where 12 NLP researchers assessed summaries of 24 papers they had personally co-authored. The researchers compared their section-based approach against a section-agnostic (flat text) approach.

Quantitative comparison of section-agnostic vs. section-based summarization across precision, coverage, and overall quality.

As shown in the evaluation results, the section-based approach consistently outperformed the flat baseline:

  • Precision: Section-based summaries were preferred by authors 63% of the time.
  • Coverage: Section-based summaries provided better coverage of the paper's actual contributions in 68% of cases.
  • Overall Quality: On a 1–5 scale, the section-based approach achieved a significantly higher mean score (3.32) compared to the flat approach (2.86).
  • Entity Extraction: The TDM extraction module achieved a Micro-F1 score of 66.0, validating the textual entailment approach for identifying research configurations.

Conclusion

The IBM Science Summarizer demonstrates that structure matters in scientific document summarization. By providing summaries per section, the tool aligns better with the high-frequency "monitoring" behavior of researchers who need to quickly determine a paper's relevance. Future work aims to expand entity extraction to include "Methods" and to scale the evaluation across other scientific domains beyond Computer Science.

Summary updated

BibTeX
@inproceedings{erera2019summarization,
  author = {Shai Erera and Michal Shmueli-Scheuer and Guy Feigenblat and Ora Peled Nakash and Odellia Boni and Haggai Roitman and Doron Cohen and Bar Weiner and Yosi Mass and Or Rivlin and Guy Lev and Achiya Jerbi and Jonathan Herzig and Yufang Hou and Charles Jochim and Martin Gleize and Francesca Bonin and David Konopnicki},
  title = {{A Summarization System for Scientific Documents}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2019},
  url = {https://openreview.net/forum?id=0qv2xF-VaLxV},
}
Figure