Abstract
Shai Erera et al. present the IBM Science Summarizer, a system designed to help researchers navigate the overwhelming flood of new scientific publications by generating structured, section-by-section summaries of Computer Science papers. By treating a paper as a collection of distinct parts rather than a flat text block, the system provides more precise and useful overviews that outperformed traditional "flat" summarization in human expert trials.
Introduction
The "information overload" in academia is a well-known crisis. With thousands of papers published weekly on platforms like arXiv and the ACL Anthology, researchers struggle to stay current. While search engines help find papers and abstracts provide a brief overview, there is a lack of tools that help users quickly "gist" the actual content of a paper without reading the whole PDF. Existing text summarizers often fail because they are designed for short news articles, whereas scientific papers are long, highly structured, and use specialized language that varies significantly between sections like "Introduction" and "Methodology."
System Architecture and Methods
The IBM Science Summarizer employs a multi-stage pipeline to process and summarize scientific documents at scale, as illustrated in the architecture diagram below.
- Ingestion & Parsing: Using Science-Parse, the system converts PDFs into structured JSON, identifying titles, authors, and—crucially—distinct sections.
- Entity Extraction (TDM): The system identifies Tasks, Datasets, and Metrics (TDM). This is framed as a textual entailment task, where the paper content is treated as "text" and potential TDM triples as a "hypothesis" to see if the paper supports that specific research configuration.
- Section-Based Summarization: Instead of one summary for the whole paper, the system generates a standalone summary for each section (e.g., "Introduction" or "Experiments").
- Extractive Algorithm: An unsupervised approach based on the Cross Entropy (CE) method selects the best sentences by optimizing for five criteria: query saliency, entity coverage, diversity, text coverage, and sentence length.
Evaluation and Results
The system was evaluated on a corpus of 270,000 papers from arXiv and the ACL Anthology. A key part of the study involved a human evaluation where 12 NLP researchers assessed summaries of 24 papers they had personally co-authored. The researchers compared their section-based approach against a section-agnostic (flat text) approach.
As shown in the evaluation results, the section-based approach consistently outperformed the flat baseline:
- Precision: Section-based summaries were preferred by authors 63% of the time.
- Coverage: Section-based summaries provided better coverage of the paper's actual contributions in 68% of cases.
- Overall Quality: On a 1–5 scale, the section-based approach achieved a significantly higher mean score (3.32) compared to the flat approach (2.86).
- Entity Extraction: The TDM extraction module achieved a Micro-F1 score of 66.0, validating the textual entailment approach for identifying research configurations.
Conclusion
The IBM Science Summarizer demonstrates that structure matters in scientific document summarization. By providing summaries per section, the tool aligns better with the high-frequency "monitoring" behavior of researchers who need to quickly determine a paper's relevance. Future work aims to expand entity extraction to include "Methods" and to scale the evaluation across other scientific domains beyond Computer Science.
Summary updated