A Summarization System for Scientific Documents

Shai Erera, Michal Shmueli-Scheuer, Guy Feigenblat, Ora Peled Nakash, Odellia Boni, Haggai Roitman, Doron Cohen, Bar Weiner, Yosi Mass, Or Rivlin, Guy Lev, Achiya Jerbi, Jonathan Herzig, Yufang Hou, Charles Jochim, Martin Gleize, Francesca Bonin, David Konopnicki

Abstract

We present a novel system providing summaries for Computer Science publications. Through a qualitative user study, we identified the most valuable scenarios for discovery, exploration and understanding of scientific documents. Based on these findings, we built a system that retrieves and summarizes scientific documents for a given information need, either in form of a free-text query or by choosing categorized values such as scientific tasks, datasets and more. Our system ingested 270,000 papers, and its summarization module aims to generate concise yet detailed summaries. We validated our approach with human experts.

Introduction

The exponential growth of scientific literature, particularly in Artificial Intelligence and Machine Learning, has created a significant "information overload." While traditional search engines excel at document retrieval, researchers often struggle during the evaluation phase. Standard abstracts frequently lack the technical detail necessary to determine a paper's specific relevance to a researcher's needs. To address this, we developed the IBM Science Summarizer, a system designed to generate query-focused, section-based summaries that highlight critical technical components like tasks, datasets, and results.

System Architecture and Methodology

The IBM Science Summarizer utilizes a multi-stage pipeline to transform raw PDFs into structured, searchable, and summarizable content.

Overview of the system architecture, showing (a) the ingestion pipeline and (b) the query and summarization process.

Ingestion and Parsing

The pipeline begins with PDF ingestion using Science-Parse, which converts documents into structured JSON. This process identifies sections, tables, and figures, merging sub-sections into approximately 6–7 main functional sections (e.g., Introduction, Methods, Results). The system ingested a corpus of 270,000 papers from the arXiv "Computer Science" subset and the ACL Anthology.

Entity Extraction

A core feature of the system is the extraction of Task, Dataset, and Metric (TDM) triples. We employ a hybrid approach:

  1. Dictionary-based lookups: Leveraging data from paperswithcode.com.
  2. Textual Entailment: A machine learning model frames entity extraction as a hypothesis validation task, allowing the system to reason about whether a specific entity is discussed without requiring massive labeled datasets.

Section-Based Summarization

Unlike traditional "flat" summarizers that treat a paper as a single block of text, our system summarizes each functional section independently. This prevents high-level introductory text from overshadowing technical details in the methodology or results. The system uses an unsupervised extractive algorithm optimized via the Cross Entropy (CE) method, guided by predictors such as query saliency, entity coverage, and diversity.

Evaluation and Results

The TDM extraction module achieved a micro-F1 score of 66.0, successfully indexing 872 tasks and 345 datasets across the corpus. The most critical finding was the superiority of the section-based approach over section-agnostic baselines.

Comparison of section-agnostic vs. section-based summarization performance across different tasks.

In a blind evaluation by 12 original paper authors, the section-based summaries were preferred across all metrics. As shown in the results table, the section-based approach achieved a significantly higher average quality score (3.32/5) compared to the flat-text baseline. Furthermore, for 68% of the papers, authors found that section-based summaries provided superior coverage of the paper's actual contributions.

Conclusion and Future Work

The IBM Science Summarizer demonstrates that treating scientific papers as structured, heterogeneous documents leads to more informative and relevant summaries. Future work will focus on expanding entity extraction to include specific methodologies and algorithms, as well as implementing variable-length summaries that adapt to the relative importance of different sections. We aim to scale the system further and provide it as a service to the broader scientific community.

Summary updated

Abstract

We present a novel system providing summaries for Computer Science publications. Through a qualitative user study, we identified the most valuable scenarios for discovery, exploration and understanding of scientific documents. Based on these findings, we built a system that retrieves and summarizes scientific documents for a given information need, either in form of a free-text query or by choosing categorized values such as scientific tasks, datasets and more. Our system ingested 270,000 papers, and its summarization module aims to generate concise yet detailed summaries. We validated our approach with human experts.
BibTeX
@inproceedings{erera2019summarization,
  author = {Shai Erera and Michal Shmueli-Scheuer and Guy Feigenblat and Ora Peled Nakash and Odellia Boni and Haggai Roitman and Doron Cohen and Bar Weiner and Yosi Mass and Or Rivlin and Guy Lev and Achiya Jerbi and Jonathan Herzig and Yufang Hou and Charles Jochim and Martin Gleize and Francesca Bonin and David Konopnicki},
  title = {{A Summarization System for Scientific Documents}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2019},
  url = {https://openreview.net/forum?id=0qv2xF-VaLxV},
}
Figure