Event-Keyed Summarization

William Gantt, Alexander Martin, Pavlo Kuchmiichuk, Aaron Steven White

Introduction

In the field of Information Extraction (IE), researchers have become highly proficient at transforming unstructured text into structured data, such as event tables detailing types, victims, and weapons. However, these tables often lack the narrative nuance required by human readers. Conversely, traditional document summarization provides a general overview but frequently fails to address specific user interests within a multi-event document.

To bridge this gap, this research introduces Event-Keyed Summarization (EKS). Unlike standard summarization, EKS requires a model to generate a targeted narrative summary based on both a full document and a specific, structured event template. This allows users to "key" a summary to a particular event of interest, such as one specific bombing described in a report covering multiple incidents.

Defining the Task

The authors define EKS as a mapping task: given a document $D$ and a query event $\langle E, R \rangle$ (consisting of an event type and its associated roles/fillers), the model must produce a concise, faithful summary $S$.

Overview of the Event-Keyed Summarization (EKS) task.

As illustrated in Figure 1, the model synthesizes information from the raw document and the structured event table to create a readable summary that captures the essential details of the specific event.

The MUCSUM Dataset

A primary contribution of this work is the release of MUCSUM, a dataset built upon the classic MUC-4 corpus of 1990s-era political violence reports. The dataset was constructed using a "generate-then-edit" pipeline: ChatGPT initially generated summaries based on gold-standard event templates, which were then manually edited by humans to ensure accuracy, context, and complete coverage of relevant roles. The final dataset contains over 1,500 event-summary pairs, with summaries averaging 44–51 words in length.

Methodology

The researchers evaluated two primary modeling strategies:

  1. Fine-tuning: Standard encoder-decoder models (BART, PEGASUS, and T5) were fine-tuned on MUCSUM. The input consisted of a linearized event template concatenated with the original document text.
  2. Zero-Shot Prompting: Frontier Large Language Models (ChatGPT and GPT-4) were tested using specialized prompts to determine their inherent ability to perform EKS without task-specific training.

Evaluation utilized lexical metrics (ROUGE, BERTScore), event-argument recovery metrics (CEAF-REE), and Natural Language Inference (NLI) to verify document-to-summary faithfulness.

Experimental Results

The experiments revealed that fine-tuned models significantly outperform traditional baselines and zero-shot LLMs in lexical overlap and stylistic alignment.

Performance comparison of various models on the MUCSUM dataset.

As shown in Table 1 (Figure 4), T5-large and BART-large emerged as the top performers. T5 achieved a ROUGE-L score of 53.4, a substantial improvement over the 3-sentence lead baseline (33.6). While GPT-4 demonstrated high faithfulness (NLI entailment), it lagged in ROUGE scores, likely due to the specific stylistic constraints of the MUCSUM dataset that fine-tuned models were able to learn.

Key Insights

  • Synthesis is Essential: Ablation studies confirmed that models perform poorly if provided with only the document (lacking focus) or only the template (lacking context). The EKS task successfully forces the synthesis of both inputs.
  • Structure as Control: The event template serves as a powerful control signal, preventing models from generating generic summaries and forcing specificity.
  • Metric Correlation: Interestingly, for this specific task, traditional metrics like ROUGE and BERTScore correlated better with human judgments than newer NLI-based metrics.

Future Directions

While the results are promising, the authors note that MUCSUM is currently limited to the domain of political violence. Future work should explore EKS in more diverse domains and investigate how models handle "noisy" inputs from automated event extraction systems rather than gold-standard templates. Moving toward open-domain event structures will be critical for the broader application of EKS technology.

Summary updated

Abstract

We introduce event-keyed summarization (EKS), a novel task that marries traditional summarization and document-level event extraction, with the goal of generating a contextualized summary for a specific event, given a document and an extracted event structure. We introduce a dataset for this task, MUCSUM, consisting of summaries of all events in the classic MUC-4 dataset, along with a set of baselines that comprises both pretrained LM standards in the summarization literature, as well as larger frontier models. We show that ablations that reduce EKS to traditional summarization or structure-to-text yield inferior summaries of target events and that MUCSUM is a robust benchmark for this task. Lastly, we conduct a human evaluation of both reference and model summaries, and provide some detailed analysis of the results.
BibTeX
@inproceedings{gantt2024event,
  author = {William Gantt and Alexander Martin and Pavlo Kuchmiichuk and Aaron Steven White},
  title = {{Event-Keyed Summarization}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2024},
  doi = {10.18653/v1/2024.findings-emnlp.431},
}
Figure