Abstract: The CL-SciSumm 2016 Shared Task established a benchmark for scientific document summarization by leveraging "citances"—the sentences in citing papers that describe a target work. The task challenges researchers to link these community-generated perspectives back to the original text and generate summaries that reflect how a paper is actually used and perceived by its peers. This overview details the three core sub-tasks: citation linkage, facet classification, and summarization. Results indicate that while progress has been made, accurately mapping citations to source text remains a significant challenge.
Introduction
Summarizing scientific literature is a complex endeavor because a paper’s impact is often defined by its reception and interpretation within the academic community. Traditional summarization methods typically rely on the author's abstract, which may overlook the specific contributions that later prove most influential. The CL-SciSumm task addresses this by focusing on citances: sentences in citing papers that reference a specific work. These citances serve as a "community-created summary" providing meta-commentary and interpretation. However, citances often lack explicit context regarding which part of the original paper they discuss. The goal of CL-SciSumm is to automate the linkage of citations to specific text spans and classify the research "facet" (e.g., method, results) being addressed.
Task Definition and Methodology
The 2016 shared task was organized into three distinct sub-tasks:
- Task 1A (Linkage): Identifying the specific span of text (up to five sentences) in the Reference Paper (RP) referred to by a given citance.
- Task 1B (Facet Classification): Classifying the identified span into discourse facets such as Aim, Hypothesis, Method, Results, or Implication.
- Task 2 (Summarization): Generating a structured summary of the RP (maximum 250 words) based on the cited text spans.
Participating teams utilized a variety of NLP techniques, ranging from lexical features (TF-IDF, WordNet) to machine learning models like SVM-rank, Decision Trees, and Neural Networks. Graph-based methods (e.g., Manifold Ranking) and Kernel Methods were also employed to capture structural similarities between citances and the original text.
Dataset and Evaluation
The task utilized the CL-SciSumm corpus, comprising 30 annotated sets of citing and reference papers from the Computational Linguistics domain. This was an expansion of the 2014 pilot, increasing the number of citing papers per reference paper to up to 60 to provide a broader community perspective. Evaluation was conducted against three types of gold standards: the original author abstract, a community summary collated from citances, and a human-written summary.
Results and Analysis
The performance of the participating systems highlighted the difficulty of the task. In Task 1A (Linkage), system sys16 (PolyU) achieved the best performance using SVM-rank with lexical and structural features. In Task 1B (Facets), system sys8 (Mao et al.) led the field using an SVM classifier paired with a topical lexicon.
For the summarization task (Task 2), performance varied depending on the gold standard used. System sys8 performed best against abstracts and community summaries. Conversely, system sys3 (Conroy and Davis) excelled when evaluated against human-written summaries, as shown in the comparative performance across different benchmarks.
Overall, F1 scores for linkage and facet classification remained relatively low, suggesting that mapping citations to source text is a persistent challenge. Most systems performed better when evaluated against the author's abstract than against community or human-annotated summaries.
Conclusion and Future Work
The CL-SciSumm 2016 task underscored the importance of context and granularity in scientific summarization. While citances provide a valuable interpretative layer, they represent community perception rather than verified claims. Future work will focus on improving data quality, exploring alternative evaluation metrics beyond ROUGE, and introducing new sub-tasks to further advance the state-of-the-art in scientific document processing.
Summary updated