Introduction
In the field of Information Extraction (IE), researchers have become highly proficient at transforming unstructured text into structured data, such as event tables detailing types, victims, and weapons. However, these tables often lack the narrative nuance required by human readers. Conversely, traditional document summarization provides a general overview but frequently fails to address specific user interests within a multi-event document.
To bridge this gap, this research introduces Event-Keyed Summarization (EKS). Unlike standard summarization, EKS requires a model to generate a targeted narrative summary based on both a full document and a specific, structured event template. This allows users to "key" a summary to a particular event of interest, such as one specific bombing described in a report covering multiple incidents.
Defining the Task
The authors define EKS as a mapping task: given a document $D$ and a query event $\langle E, R \rangle$ (consisting of an event type and its associated roles/fillers), the model must produce a concise, faithful summary $S$.
As illustrated in Figure 1, the model synthesizes information from the raw document and the structured event table to create a readable summary that captures the essential details of the specific event.
The MUCSUM Dataset
A primary contribution of this work is the release of MUCSUM, a dataset built upon the classic MUC-4 corpus of 1990s-era political violence reports. The dataset was constructed using a "generate-then-edit" pipeline: ChatGPT initially generated summaries based on gold-standard event templates, which were then manually edited by humans to ensure accuracy, context, and complete coverage of relevant roles. The final dataset contains over 1,500 event-summary pairs, with summaries averaging 44–51 words in length.
Methodology
The researchers evaluated two primary modeling strategies:
- Fine-tuning: Standard encoder-decoder models (BART, PEGASUS, and T5) were fine-tuned on MUCSUM. The input consisted of a linearized event template concatenated with the original document text.
- Zero-Shot Prompting: Frontier Large Language Models (ChatGPT and GPT-4) were tested using specialized prompts to determine their inherent ability to perform EKS without task-specific training.
Evaluation utilized lexical metrics (ROUGE, BERTScore), event-argument recovery metrics (CEAF-REE), and Natural Language Inference (NLI) to verify document-to-summary faithfulness.
Experimental Results
The experiments revealed that fine-tuned models significantly outperform traditional baselines and zero-shot LLMs in lexical overlap and stylistic alignment.
As shown in Table 1 (Figure 4), T5-large and BART-large emerged as the top performers. T5 achieved a ROUGE-L score of 53.4, a substantial improvement over the 3-sentence lead baseline (33.6). While GPT-4 demonstrated high faithfulness (NLI entailment), it lagged in ROUGE scores, likely due to the specific stylistic constraints of the MUCSUM dataset that fine-tuned models were able to learn.
Key Insights
- Synthesis is Essential: Ablation studies confirmed that models perform poorly if provided with only the document (lacking focus) or only the template (lacking context). The EKS task successfully forces the synthesis of both inputs.
- Structure as Control: The event template serves as a powerful control signal, preventing models from generating generic summaries and forcing specificity.
- Metric Correlation: Interestingly, for this specific task, traditional metrics like ROUGE and BERTScore correlated better with human judgments than newer NLI-based metrics.
Future Directions
While the results are promising, the authors note that MUCSUM is currently limited to the domain of political violence. Future work should explore EKS in more diverse domains and investigate how models handle "noisy" inputs from automated event extraction systems rather than gold-standard templates. Moving toward open-domain event structures will be critical for the broader application of EKS technology.
Summary updated