Two-phase Multi-document Event Summarization on Core Event Graphs

Zengjian Chen, Jin Xu, Meng Liao, Tong Xue, Kun HeJAIR 2022 · OpenReview

TL;DR

This paper introduces a novel two-phase framework for Multi-document Event Summarization (MES), a challenging task that aims to generate concise, query-level event descriptions from multiple news articles. The proposed method first constructs an event semantic graph to identify the core event and then uses an event-aware pointer generator to produce a fluent and informative summary, outperforming existing baselines.

What Problem Does This Paper Address?

The internet bombards us with a constant stream of news articles, making it difficult to grasp the essence of trending topics or breaking events. Existing summarization techniques often struggle with the redundancy and sparsity inherent in multiple articles, and they may not effectively capture the core event that spans across different sentences or even documents. The Multi-document Event Summarization (MES) task aims to distill this information into a short, precise event description, typically under 14 Chinese characters, which is crucial for news systems and search engines. This paper tackles the challenge of identifying the most dominant event from a cluster of articles and then summarizing it concisely and fluently, a task that has not been sufficiently addressed by prior work.

Methods

The authors propose a two-phase framework to address the MES task:

  1. Dominant Event Identification: This phase aims to extract the most important event information. It involves:
  2. Event Semantic Graph Construction: Using off-the-shelf tools for word segmentation and named entity recognition, they extract topic keywords and article keywords. Then, they construct a graph representing the core event by identifying key sentences, performing dependency parsing, and reserving specific relation edges (SBV, VOB, IOB) that connect to topic keywords.
  3. Representative Event Sentence Selection: To ensure fluency, they select a representative sentence that best captures the core event. This is done by calculating a similarity score between candidate sentences and the event semantic graph, considering both keyword similarity and semantic link similarity.
  4. Neural Event Sequence Generation: This phase summarizes the extracted information into a concise event sequence.
  5. Event-Aware Pointer Generator: They adapt a standard sequence-to-sequence model with an attention mechanism. The key innovation is the "event-aware pointer generator," which integrates the event semantic graph information. This allows the model to attend not only to the representative sentence but also to relevant event elements from the graph, even if they are not present in the selected sentence. This context-aware attention mechanism helps to produce more informative and accurate summaries.

Datasets

To facilitate research in MES, the authors introduce two new large-scale datasets:

  • TMES (Topic Multi-document Event Summarization): This dataset was created through manual editing. Articles were clustered, and then less than 10 articles per cluster were selected by professional editors based on representativeness and diversity. Summaries were then annotated by two editors, with a high inter-annotator consistency of 87%. The dataset covers event topics from August 2018 to August 2020.
  • SMES (Search Multi-document Event Summarization): This dataset was constructed from search engine results. Event summaries were crawled from Chinese social platforms and news apps, and these summaries were used as queries to retrieve related news articles. This resulted in 53,787 summary-article pairs, covering major events from August 2019 to August 2021.

These datasets are significantly larger than previous multi-document summarization datasets and are designed to be suitable for deep neural model training.

Key Insights and Ideas

  • The importance of a two-phase approach: Separating event identification from summarization allows for a more focused and effective extraction of core event information before generating the summary.
  • Leveraging graph structures for event understanding: Constructing an event semantic graph provides a structured representation of event elements and their relationships, which is crucial for identifying the dominant event and its key components.
  • Bridging the gap between extractive and abstractive methods: The event-aware pointer generator effectively combines the strengths of both approaches, using extractive techniques to identify salient information and abstractive techniques to generate fluent summaries.
  • Context-aware attention is key for informativeness: By incorporating event semantic graph information into the attention mechanism, the model can go beyond the representative sentence to capture crucial event details scattered across multiple articles, leading to more informative summaries.

Key Results

The proposed two-phase framework, Event-Pg, significantly outperforms various baselines on both automatic (ROUGE scores) and manual evaluation metrics.

  • Automatic Evaluation: Event-Pg achieved the best ROUGE-1, ROUGE-2, and ROUGE-L scores on both TMES and SMES datasets, demonstrating superior performance in capturing event information.
  • Manual Evaluation: Event-Pg also showed the highest accuracy in core event identification and informativeness. While its readability was comparable to other sequence-to-sequence models, it excelled in accuracy and informativeness, especially on the more challenging TMES dataset.
  • Event Identification Analysis: The paper shows that their graph-based event identification approach outperforms simpler methods like lead-title and Text-rank, and even event-extraction-based methods, by effectively leveraging semantic graph information.
  • Event Extraction Study: Using the ACE2005 benchmark, Event-Pg demonstrated a higher Event Element Extraction Score (EEES), indicating its ability to retain more golden event mentions in its generated summaries.

Limitations and Future Work

  • Pretrained Models: The authors acknowledge that pretrained abstractive models like BART might outperform pointer-based models in generation phase. They plan to explore incorporating pretrained summarization models with event graphs in future work to better capture unique event elements.
  • Dataset Characteristics: Performance differences between TMES and SMES highlight the impact of dataset characteristics. TMES, with its manually edited summaries and focus on diversity, presents a greater challenge for summarization from a single representative sentence.
  • Readability on SMES: While Event-Pg performs well, it is slightly inferior to some sequence-to-sequence models in terms of fluency on the SMES dataset. Future work could focus on further improving the readability of generated summaries.

Summary updated

Abstract

Succinct event description based on multiple documents is critical to news systems as well as search engines. Different from existing summarization or event tasks, Multi-document Event Summarization (MES) aims at the query-level event sequence generation, which has extra constraints on event expression and conciseness. Identifying and summarizing the key event from a set of related articles is a challenging task that has not been sufficiently studied, mainly because online articles exhibit characteristics of redundancy and sparsity, and a perfect event summarization needs high level information fusion among diverse sentences and articles. To address these challenges, we propose a two-phase framework for the MES task, that first performs event semantic graph construction and dominant event detection via graph-sequence matching, then summarizes the extracted key event by an event-aware pointer generator. For experiments in the new task, we construct two large-scale real-world datasets for training and assessment. Extensive evaluations show that the proposed framework significantly outperforms the related baseline methods, with the most dominant event of the articles effectively identified and correctly summarized.
BibTeX
@article{chen2022two,
  author = {Zengjian Chen and Jin Xu and Meng Liao and Tong Xue and Kun He},
  title = {{Two-phase Multi-document Event Summarization on Core Event Graphs}},
  journal = {J. Artif. Intell. Res.},
  year = {2022},
  url = {https://openreview.net/forum?id=db4jxJP2QQw},
}
Figure