TL;DR
This paper tackles the challenge of generating concise, informative single-sentence summaries from multiple related sentences (Multi-Sentence Compression or MSC). They propose a novel approach that reranks candidate summaries generated by a word graph method, using keyphrases extracted from the source sentences to boost informativeness. This method significantly improves the quality of generated summaries, making them more representative of the original content.
What Problem Does This Paper Address?
Generating a single, coherent sentence that captures the essence of several related sentences is a complex task known as Multi-Sentence Compression (MSC). While existing methods often rely on syntactic parsers or word graph approaches, they frequently struggle to retain all the important information from the source sentences, leading to summaries that miss key facts. The core problem is how to ensure that the generated compressed sentence is not only grammatically correct but also maximally informative, covering the main topics present in the input.
Methods
The authors build upon Filippova's (2010) word graph-based MSC approach. This method constructs a directed graph where nodes represent words (and their Part-of-Speech tags) and edges represent their order in the original sentences. Candidate compressions are then generated by finding paths through this graph.
The key innovation in this paper is the N-best reranking step using keyphrase extraction. After generating an initial set of candidate compressions (the "N-best list"), these candidates are re-evaluated. The reranking process involves:
- Keyphrase Extraction: An unsupervised method (based on Wan and Xiao, 2008) is used to identify keyphrases from the set of related input sentences. These keyphrases are considered to represent the main topics.
- Candidate Reranking: Each candidate compression is scored based on the number and relevance of the extracted keyphrases it contains. Compressions that include more relevant keyphrases are ranked higher.
- Path Scoring: The final score for a compression path is calculated by normalizing the sum of edge weights (representing cohesion and word frequency) by the path length and the sum of keyphrase scores it contains.
To improve the original word graph approach, they also made a minor modification to include punctuation marks in the graph construction.
Datasets
The authors introduce a newly constructed French evaluation dataset. This dataset consists of:
- 40 sets of related sentences.
- Reference compressions manually created by human annotators for each set.
The sentences were extracted from news articles on Google News over three months, focusing on clusters of at least 20 articles related to a single event. Duplicate sentences were removed, and leading sentences were used as a baseline. The dataset contains 618 sentences with an average of 15 sentences per cluster. The reference compressions achieved a compression rate of 60%.
Key Insights and Ideas
- Keyphrases as Informativeness Proxies: The central idea is that keyphrases act as semantic metadata, effectively capturing the core topics of a document. By prioritizing compressions that contain these keyphrases, the system can ensure that the generated summary is more informative.
- Complementary Strengths: The paper highlights that while word graph methods are good at generating grammatical sentences, they can miss important information. Keyphrase extraction, on the other hand, directly targets informativeness. Combining them through reranking leverages the strengths of both.
- Unsupervised Keyphrase Extraction is Sufficient: The use of an unsupervised method for keyphrase extraction is crucial, as it avoids the need for large, domain-specific annotated datasets, making the approach more generalizable.
- Reranking vs. Generation: Instead of trying to generate the perfect sentence from scratch, the approach focuses on reranking a set of plausible candidates, which is often a more tractable problem.
Key Results
- Manual Evaluation: The proposed KeyRank method showed a significant improvement in informativity compared to the baseline (Filippova's approach). Specifically, 97.5% of KeyRank compressions were related to the main theme, and 62.5% conveyed the gist, an absolute increase of 19.2% over the baseline. Grammaticality scores slightly decreased, which the authors attribute to potentially longer compressions being favored.
- Automatic Evaluation: KeyRank significantly outperformed the baseline on ROUGE metrics (ROUGE-1, ROUGE-2, ROUGE-SU4), indicating better informativeness. BLEU scores showed a slight, non-significant increase.
- Correlation with Human Ratings: The study found that ROUGE scores correlate moderately to strongly with informativity ratings, while BLEU correlates better with grammaticality. However, overall, automatic metrics showed only a medium correlation with human judgments, suggesting they cannot fully replace manual evaluation.
Limitations and Future Work
- Grammaticality Trade-off: The reranking process, while improving informativeness, led to a slight decrease in grammaticality. The authors suggest that the longer compressions favored by the reranking might be a contributing factor.
- Automatic Metric Limitations: The paper acknowledges that automatic evaluation metrics, while useful indicators, have limitations and do not perfectly align with human judgments.
- Future Work: The authors plan to explore ways to enhance the grammaticality of generated compressions, potentially by experimenting with higher-order POS language models for reranking.
Summary updated