Enhancing Single-Document Summarization by Combining RankNet and Third-Party Sources

Krysta Svore, Lucy Vanderwende, Christopher BurgesEMNLP 2007

TL;DR

This paper introduces NetSum, a novel neural network-based approach for single-document summarization that leverages external knowledge from news query logs and Wikipedia to improve sentence selection. By training a RankNet model on these enhanced features, NetSum significantly outperforms traditional baselines in generating both summary blocks and individual highlight sentences.

What Problem Does This Paper Address?

Automatic summarization, the task of condensing a document into a shorter version, has been a long-standing challenge in Natural Language Processing. While extractive summarization (selecting existing sentences) is more straightforward than abstractive summarization (generating new sentences), accurately identifying the most important sentences remains difficult. Traditional methods often rely on sentence position or simple keyword matching. Even sophisticated systems evaluated in past Document Understanding Conferences (DUC) struggled to significantly outperform a simple baseline of selecting the first few sentences. This paper addresses the challenge of improving single-document summarization by exploring how to better identify salient sentences, especially in the context of news articles where concise summaries (like "story highlights") are increasingly valuable.

Methods

The core of NetSum is a neural network ranking algorithm called RankNet. Instead of directly classifying sentences as important or not, RankNet learns to rank pairs of sentences. The system is trained on pairs of sentences (Si, Sj) where Si is considered "better" than Sj based on their ROUGE scores (a standard metric for evaluating summary quality).

What's novel about NetSum is its use of third-party data sources to create richer sentence features. Beyond standard features like sentence position and basic term frequency (SumBasic), NetSum incorporates:

  • News Query Logs: Features derived from frequently searched terms on news websites. The hypothesis is that sentences containing these popular query terms are more likely to be relevant to what readers are looking for.
  • Wikipedia Entities: Features based on entities (like people, organizations, or places) that appear in a sentence and are also linked to Wikipedia pages. The idea is that sentences mentioning prominent entities are likely to be important.

These enhanced features are fed into the RankNet model, which is trained to predict the ranking of sentences that best match human-generated "story highlights." The paper explores two main tasks:

  1. Block Matching: Extracting three sentences that, when combined, best match a block of three human-generated highlights.
  2. Ordered Highlight Matching: Extracting three sentences that individually best match each of the three human-generated highlights, preserving the order.

Datasets

The primary dataset used in this paper is a collection of 1365 news documents from CNN.com. Each document includes a title, timestamp, 3-4 human-generated "story highlights," and the full article text. This dataset was hand-collected over consecutive days in February 2007. The human-generated highlights serve as the "gold standard" for evaluation.

Key Insights and Ideas

  • External Knowledge is Powerful: The most significant insight is that incorporating information from external sources like news query logs and Wikipedia can substantially improve sentence selection for summarization. This moves beyond relying solely on the internal structure and content of the document itself.
  • RankNet for Sentence Ranking: Using a pair-wise learning approach like RankNet is effective for ordering sentences, which is crucial for identifying the most important ones. This is more nuanced than simple classification.
  • Highlights as a Proxy for Importance: Using human-generated "story highlights" as the target for summarization is a practical and relevant task, especially for news articles. It provides a clear objective for the system.
  • The "First Sentence" Baseline is Strong but Beatable: The paper confirms that the first sentence of a news article is often a good summary, but NetSum demonstrates that more sophisticated methods can indeed surpass this baseline significantly.

Key Results

NetSum achieved statistically significant improvements over the baseline of selecting the first N sentences.

  • Summarization Block Task: NetSum(b) produced a block of three sentences that scored equal to or better than the baseline in 73.26% of documents for ROUGE-1. On average, NetSum(b) achieved a higher ROUGE-1 score.
  • Ordered Highlight Task:
    • For the first highlight (often similar to the article's lead), NetSum(1) outperformed the baseline in 93.26% of documents for ROUGE-1.
  • For the second and third highlights, NetSum(2) and NetSum(3) also showed significant improvements, with NetSum(2) outperforming the baseline in 77.73% of documents for ROUGE-1.
  • Feature Ablation: Removing the news-based and Wikipedia-based features led to a noticeable drop in performance, confirming their importance.

Limitations and Future Work

  • ROUGE Scores are Relatively Low: The authors acknowledge that ROUGE scores are not exceptionally high because human-generated highlights rarely match a single article sentence perfectly. This suggests a need for more advanced summarization techniques.
  • Future Work:
    • Abstractive Summarization: The current system is extractive. Future work could explore incorporating sentence compression or generation to create more condensed, abstractive summaries.
  • Feature Selection: Further investigation into optimal feature selection, potentially tailored for each highlight (e.g., different features for the first highlight vs. the third), could yield further improvements.
  • Input Difficulty: The paper suggests exploring methods to identify "hard" vs. "easy" inputs to better evaluate system performance across varying document complexities.

Summary updated

BibTeX
@inproceedings{svore2007enhancing,
  author = {Krysta Svore and Lucy Vanderwende and Christopher Burges},
  title = {{Enhancing Single-Document Summarization by Combining RankNet and Third-Party Sources}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2007},
}
Figure