Extractive Summarisation Based on Keyword Profile and Language Model

Han Xu, Eric Martin, Ashesh MahidadiaNAACL 2015

TL;DR

This paper introduces a novel statistical framework for automatically extracting key sentences that summarize the main contributions of scientific papers. By leveraging the "citation summary" (sentences from other papers that cite the target paper) and a keyword profiling approach, the system achieves state-of-the-art performance in scientific paper summarization, outperforming existing methods.

What Problem Does This Paper Address?

Scientific research builds upon a vast and interconnected network of existing work. For researchers exploring new areas, understanding the core contributions of papers outside their immediate expertise is crucial but challenging. Traditional paper abstracts can sometimes deviate from how the community perceives a paper's contributions over time. This paper addresses the need for tools that can automatically identify and extract the most informative sentences summarizing a paper's key contributions, using a unique data source: the sentences from papers that cite the target paper.

Methods

The proposed approach is a two-stage process:

  1. Keyword Profiling: The first stage automatically identifies "salient keywords" from a paper's citation summary. These keywords are those that are statistically over-represented and relatively exclusive to the target paper's citations, indicating they characterize its main contributions. This is achieved using a Hypergeometric test to measure the statistical association between words and the paper's citation summary, effectively capturing words' salience.
  2. Keyword Profile Language Model (KPLM) and Sentence Selection: The identified keywords are used to build a discriminative unigram language model (KPLM) for the paper. This KPLM assigns higher probabilities to words that are more salient in characterizing the paper's contributions. The task of summarization is then framed as an Information Retrieval (IR) problem, where citing sentences are treated as documents and the KPLM acts as the query. Sentences are selected based on their negative cross-entropy with the KPLM, meaning sentences that best conform to the paper's keyword profile are chosen.
  3. Top Sentence Re-ranking (TSR): To improve diversity in the generated summaries, a re-ranking heuristic is employed. This process iteratively selects sentences that add the most "extra information" (measured by cross-entropy) to the summary, ensuring a broader coverage of key concepts.

Datasets

The experiments and evaluations were conducted on a corpus of 25 highly cited papers from the ACL Anthology Network (AAN), covering five domains: Dependency Parsing (DP), Phrase Based Machine Translation (PBMT), Text Summarisation (SUM), Question Answering (QA), and Textual Entailment (TE). For each paper, the dataset includes:

  • A citation summary file containing all citing sentences.
  • A manually constructed "key fact" file, which serves as a gold standard of the paper's main contributions, hand-picked by human annotators.

This corpus is described as a "gold standard" for research paper summarization and has been widely used in previous system evaluations.

Key Insights and Ideas

  • Citation Summaries as a Rich Source: The paper argues that citation summaries, being a form of "crowd-sourced review," contain valuable information about a paper's contributions that can be more representative than abstracts alone.
  • Statistical Salience for Keywords: Using statistical measures like the Hypergeometric test to identify keywords based on their over-representation and exclusivity in citations is a robust way to pinpoint a paper's core contributions.
  • KPLM as a Discriminative Model: The Keyword Profile Language Model (KPLM) is not a traditional language model but a discriminative one that directly encodes the salience of words, providing a more focused representation for summarization.
  • IR Framework for Sentence Selection: Framing summarization as an IR problem, where sentences are ranked based on their similarity to the KPLM, offers a natural and effective way to select informative sentences.
  • Diversity through Re-ranking: The top sentence re-ranking mechanism is crucial for generating summaries that cover a broader range of key concepts, preventing redundancy.

Key Results

The proposed KPLM-based summarization system, especially when combined with the Top Sentence Re-ranking (TSR) method, achieved state-of-the-art performance.

  • Pyramid Scores: The system achieved comparable results to C-LexRank (a strong baseline) even without re-ranking. With TSR, it outperformed C-LexRank. For example, for paper P05-1012, the pyramid score increased from 0.23 (without re-ranking) to 0.73 (with re-ranking).
  • Median Score: The KPLM+TSR system achieved a higher median pyramid score (0.86) compared to the best performing variants of previous systems (0.80).
  • Robustness to Length Limits: The system demonstrated graceful performance decay even when the summary length limit was significantly tightened, maintaining a mean pyramid score close to 0.6 even with a limit of 1 sentence.
  • Simplicity and Interpretability: The authors highlight that their approach is substantially simpler than C-LexRank and yields more interpretable results.

Limitations and Future Work

  • Statistical Significance: While the system performed well, a one-tailed Wilcoxon signed-rank test did not show statistically superior results at the 0.05 significance level compared to C-LexRank on the small sample of 25 papers. A larger effect size would be needed for statistical significance.
  • N-gram Extension: The current keyword profiling focuses on unigrams. The authors suggest extending this to higher-order n-grams to capture "key phrases."
  • Multi-document Summarization: The authors express interest in applying their method to multi-document summarization, particularly for generating technical surveys of scientific paradigms.
  • Other Genres and Tasks: Further experiments are needed to test the method's effectiveness on more generic summarization tasks and different text genres.

Summary updated

BibTeX
@inproceedings{xu2015extractive,
  author = {Han Xu and Eric Martin and Ashesh Mahidadia},
  title = {{Extractive Summarisation Based on Keyword Profile and Language Model}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2015},
}
Figure