PolyU at CL-SciSumm 2016

Ziqiang Cao, Wenjie Li, Dapeng WuBIRNDL@JCDL 2016 · OpenReview

Executive Summary

The PolyU team’s entry for the CL-SciSumm 2016 challenge provides a robust framework for automatic scientific paper summarization. By combining learning-to-rank algorithms (SVM Rank) for citation matching, decision trees for handling imbalanced facet classification, and an enhanced Manifold Ranking model for summary generation, they demonstrate how to effectively bridge the gap between a paper’s content and how the broader research community cites it.

Problem Formulation

Summarizing a scientific paper involves more than just condensing its text; it requires understanding its impact and context within the research community. The CL-SciSumm (Computational Linguistics Scientific Document Summarization) Shared Task addresses this by analyzing "citances"—the specific sentences in citing papers that refer to a reference paper.

The authors address three specific challenges:

  1. Task 1A (Linkage): Identifying which specific sentences in the original paper a citation refers to.
  2. Task 1B (Facet Classification): Categorizing the "facet" of the cited text (e.g., Method, Results, or Hypothesis).
  3. Task 2 (Summarization): Generating a 250-word summary of the original paper that incorporates these community citations.

Methodology

Task 1: Citation Linkage and Facet Classification

For Task 1A, the team treated linkage as a ranking problem using SVM Rank. Instead of analyzing single sentences in isolation, they grouped adjacent sentences into "chunks" (1 to 4 sentences long) to better capture citations referring to multi-sentence ideas.

Frequency of sentence counts in citation references, supporting the 'Adjacency Property'.

For Task 1B, the team encountered extreme class imbalance. As shown in the dataset distribution, the "Method" facet accounts for the vast majority of instances, while facets like "Hypothesis" are rare.

Distribution of citation facets in the CL-SciSumm 2016 dataset, illustrating significant class imbalance.

While standard models like SVMs often struggle with such imbalance, the team utilized a Decision Tree Classifier with $\chi^2$-test feature selection. This allowed the system to better capture the specific vocabulary and patterns associated with less frequent facets.

Task 2: Enhanced Manifold Ranking

For the final summarization task, the authors adapted the Manifold Ranking algorithm. They treated citing papers as "queries" and the reference paper's sections as "documents." A key innovation was the introduction of a parameter ($\gamma$) to balance the importance of community citations against the original paper's content.

Key Insights and Results

  • The Adjacency Property: The model confirmed that most citations refer to one or two adjacent sentences. Treating the reference paper as a collection of $n$-sentence chunks significantly simplified the ranking task.
  • Positional Importance: Sentences at the beginning of a paper or section are much more likely to be cited. Features like SENT_POSITION and SECTION_POSITION were critical in the ranking model.
  • Balancing the Community Voice: In the summarization task, a $\gamma$ value of 0.5 was found to be optimal, suggesting that effective scientific summaries should give equal weight to the author's original text and the community's interpretation.

Performance Summary

The system's performance across the three tasks was as follows:

  • Task 1A: F1-score of 10.0 (Accuracy: 11.8, Recall: 8.7).
  • Task 1B: Micro-accuracy of 61.9 and Macro-accuracy of 21.4.
  • Task 2: ROUGE-1 score of 49.5 and ROUGE-2 score of 15.4.

Limitations and Future Work

The authors noted that their Decision Tree for facet classification was prone to over-fitting, sometimes resulting in leaf nodes supported by only a single case. Future research will focus on developing more robust features or data augmentation techniques to help models distinguish between subtle facets without losing generalization ability.

Summary updated

BibTeX
@inproceedings{cao2016polyu,
  author = {Ziqiang Cao and Wenjie Li and Dapeng Wu},
  title = {{PolyU at CL-SciSumm 2016}},
  booktitle = {BIRNDL@JCDL},
  year = {2016},
  url = {https://openreview.net/forum?id=Hmq8CuU8kjo},
}
Figure