TL;DR
This paper introduces GRANUSUM, the first unsupervised framework for generating text summaries at multiple levels of detail (granularity). By focusing on "events" as core semantic units and ranking their salience, GRANUSUM can produce summaries tailored to different user preferences without requiring manual annotations, while also achieving state-of-the-art results in standard summarization tasks.
What Problem Does This Paper Address?
Traditional text summarization systems often produce a single summary for a given document. However, users have diverse needs; some might want a quick overview (coarse granularity), while others desire more in-depth information (fine granularity). This paper addresses the challenge of creating summarization systems that can adapt to these varying user preferences by generating summaries with customizable semantic coverage. Existing approaches often rely on supervised learning or use simple length control, which doesn't truly capture semantic depth. The authors highlight that automatically generating summaries with controllable semantic coverage, without manual supervision, is an under-explored area.
Methods
GRANUSUM is built around two core components: an Event-aware Summarizer and an Event Selector.
- Event Extraction: The system first identifies "events" within the source document. These are defined as verb-centric phrases extracted using syntactic patterns.
- Event-based Summarizer Pre-training: A sequence-to-sequence model is pre-trained to reconstruct masked sentences using extracted events and the remaining text as input. This teaches the model to generate text related to specific events.
- Event Selector: This component ranks the extracted events based on their salience. It uses the pre-trained summarizer to assess how much a summary would change if an event were removed, indicating the event's importance. To manage computational complexity, it also employs a pruning step based on sentence relevance and redundancy scores.
- Multi-Granularity Summary Generation: During inference, the system feeds a varying number of the most salient events (selected by the Event Selector) as "hints" to the pre-trained Event-aware Summarizer. This allows the summarizer to generate summaries with different levels of semantic detail, effectively controlling the granularity.
Datasets
- GranuDUC: A new benchmark dataset created by re-annotating DUC2004. It contains document clusters with multiple summaries at three distinct granularity levels (coarse, medium, fine), focusing on semantic coverage rather than just sentence count.
- Multi-News: A large-scale multi-document news summarization dataset used for bucket-based evaluation and unsupervised summarization experiments.
- DUC2004: Used for testing only, as a basis for GranuDUC.
- arXiv: A collection of long scientific papers used for unsupervised summarization experiments.
Key Insights and Ideas
- Events as Granularity Control: The core idea is that "events" are fundamental semantic units that can effectively control the granularity of a summary. By selecting and ranking events, the system can precisely dictate how much information is included.
- Unsupervised Approach is Key: The paper emphasizes the importance of an unsupervised approach, as creating large-scale datasets for multi-granularity summarization is challenging and expensive. This makes GRANUSUM more practical and scalable.
- Salience Ranking for Quality: The Event Selector's ability to rank events by salience is crucial. It helps filter out generic or redundant events, leading to more focused and informative summaries, especially at finer granularities.
- Bridging Granularity and Abstractiveness: While focusing on granularity, the framework also shows strong performance in standard unsupervised abstractive summarization, suggesting that event-based control can improve overall summary quality.
Key Results
- Superiority on GranuDUC: GRANUSUM significantly outperforms strong baselines like PEGASUS and LED-LC on the GranuDUC benchmark across all granularity levels, showing improvements of up to 4.53 in R-1 score at the fine granularity level.
- Effective Bucket-based Evaluation: In bucket-based evaluations on Multi-News, GRANUSUM consistently improves R-1 scores, particularly in high semantic coverage buckets, demonstrating its ability to generate summaries with appropriate detail.
- State-of-the-Art in Unsupervised Abstractive Summarization: Surprisingly, GRANUSUM achieves state-of-the-art results on standard unsupervised abstractive summarization tasks across multiple datasets, even outperforming models specifically designed for these tasks.
- Human Evaluation: Human evaluations confirm that GRANUSUM generates more relevant and faithful summaries compared to baselines, especially at finer granularities.
Limitations and Future Work
- Event Definition: The current event definition is based on syntactic patterns. Future work could explore richer event representations (e.g., including entities, relationships) for more flexible granularity control.
- Dataset Limitations: GranuDUC, while valuable, is relatively small and intended for testing. The authors call for more diverse and larger multi-granularity summarization datasets.
- Abstractiveness vs. Event Hints: Extracting events as hints might slightly reduce the abstractiveness of summaries. Future work could explore rephrasing events to enhance abstractiveness.
- Scope of Input: The current work focuses on document clusters and three granularity levels. Extending GRANUSUM to handle much longer texts (e.g., entire books) and more granular levels is a promising direction.
Summary updated