Unsupervised Multi-Granularity Summarization

Ming Zhong, Yang Liu, Suyu Ge, Yuning Mao, Yizhu Jiao, Xingxing Zhang, Xu Yi‐chong, Chenguang Zhu, Michael Zeng, Jiawei Han

TL;DR

This paper introduces GRANUSUM, the first unsupervised framework for generating text summaries at multiple levels of detail (granularity). By focusing on "events" as core semantic units and ranking their salience, GRANUSUM can produce summaries tailored to different user preferences without requiring manual annotations, while also achieving state-of-the-art results in standard summarization tasks.

What Problem Does This Paper Address?

Traditional text summarization systems often produce a single summary for a given document. However, users have diverse needs; some might want a quick overview (coarse granularity), while others desire more in-depth information (fine granularity). This paper addresses the challenge of creating summarization systems that can adapt to these varying user preferences by generating summaries with customizable semantic coverage. Existing approaches often rely on supervised learning or use simple length control, which doesn't truly capture semantic depth. The authors highlight that automatically generating summaries with controllable semantic coverage, without manual supervision, is an under-explored area.

Methods

GRANUSUM is built around two core components: an Event-aware Summarizer and an Event Selector.

  1. Event Extraction: The system first identifies "events" within the source document. These are defined as verb-centric phrases extracted using syntactic patterns.
  2. Event-based Summarizer Pre-training: A sequence-to-sequence model is pre-trained to reconstruct masked sentences using extracted events and the remaining text as input. This teaches the model to generate text related to specific events.
  3. Event Selector: This component ranks the extracted events based on their salience. It uses the pre-trained summarizer to assess how much a summary would change if an event were removed, indicating the event's importance. To manage computational complexity, it also employs a pruning step based on sentence relevance and redundancy scores.
  4. Multi-Granularity Summary Generation: During inference, the system feeds a varying number of the most salient events (selected by the Event Selector) as "hints" to the pre-trained Event-aware Summarizer. This allows the summarizer to generate summaries with different levels of semantic detail, effectively controlling the granularity.

Datasets

  • GranuDUC: A new benchmark dataset created by re-annotating DUC2004. It contains document clusters with multiple summaries at three distinct granularity levels (coarse, medium, fine), focusing on semantic coverage rather than just sentence count.
  • Multi-News: A large-scale multi-document news summarization dataset used for bucket-based evaluation and unsupervised summarization experiments.
  • DUC2004: Used for testing only, as a basis for GranuDUC.
  • arXiv: A collection of long scientific papers used for unsupervised summarization experiments.

Key Insights and Ideas

  • Events as Granularity Control: The core idea is that "events" are fundamental semantic units that can effectively control the granularity of a summary. By selecting and ranking events, the system can precisely dictate how much information is included.
  • Unsupervised Approach is Key: The paper emphasizes the importance of an unsupervised approach, as creating large-scale datasets for multi-granularity summarization is challenging and expensive. This makes GRANUSUM more practical and scalable.
  • Salience Ranking for Quality: The Event Selector's ability to rank events by salience is crucial. It helps filter out generic or redundant events, leading to more focused and informative summaries, especially at finer granularities.
  • Bridging Granularity and Abstractiveness: While focusing on granularity, the framework also shows strong performance in standard unsupervised abstractive summarization, suggesting that event-based control can improve overall summary quality.

Key Results

  • Superiority on GranuDUC: GRANUSUM significantly outperforms strong baselines like PEGASUS and LED-LC on the GranuDUC benchmark across all granularity levels, showing improvements of up to 4.53 in R-1 score at the fine granularity level.
  • Effective Bucket-based Evaluation: In bucket-based evaluations on Multi-News, GRANUSUM consistently improves R-1 scores, particularly in high semantic coverage buckets, demonstrating its ability to generate summaries with appropriate detail.
  • State-of-the-Art in Unsupervised Abstractive Summarization: Surprisingly, GRANUSUM achieves state-of-the-art results on standard unsupervised abstractive summarization tasks across multiple datasets, even outperforming models specifically designed for these tasks.
  • Human Evaluation: Human evaluations confirm that GRANUSUM generates more relevant and faithful summaries compared to baselines, especially at finer granularities.

Limitations and Future Work

  • Event Definition: The current event definition is based on syntactic patterns. Future work could explore richer event representations (e.g., including entities, relationships) for more flexible granularity control.
  • Dataset Limitations: GranuDUC, while valuable, is relatively small and intended for testing. The authors call for more diverse and larger multi-granularity summarization datasets.
  • Abstractiveness vs. Event Hints: Extracting events as hints might slightly reduce the abstractiveness of summaries. Future work could explore rephrasing events to enhance abstractiveness.
  • Scope of Input: The current work focuses on document clusters and three granularity levels. Extending GRANUSUM to handle much longer texts (e.g., entire books) and more granular levels is a promising direction.

Summary updated

Abstract

Text summarization is a user-preference based task, i.e., for one document, users often have different priorities for summary. As a key aspect of customization in summarization, granularity is used to measure the semantic coverage between the summary and source document. However, developing systems that can generate summaries with customizable semantic coverage is still an under-explored topic. In this paper, we propose the first unsupervised multi-granularity summarization framework, GranuSum. We take events as the basic semantic units of the source documents and propose to rank these events by their salience. We also develop a model to summarize input documents with given events as anchors and hints. By inputting different numbers of events, GranuSum is capable of producing multi-granular summaries in an unsupervised manner. Meanwhile, we annotate a new benchmark GranuDUC that contains multiple summaries at different granularities for each document cluster. Experimental results confirm the substantial superiority of GranuSum on multi-granularity summarization over strong baselines. Further, by exploiting the event information, GranuSum also exhibits state-of-the-art performance under the conventional unsupervised abstractive setting. Dataset for this paper can be found at: https://github.com/maszhongming/GranuDUC
BibTeX
@inproceedings{zhong2022unsupervised,
  author = {Ming Zhong and Yang Liu and Suyu Ge and Yuning Mao and Yizhu Jiao and Xingxing Zhang and Xu Yi‐chong and Chenguang Zhu and Michael Zeng and Jiawei Han},
  title = {{Unsupervised Multi-Granularity Summarization}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2022},
  doi = {10.18653/v1/2022.findings-emnlp.366},
}
Figure