Planning with Learned Entity Prompts for Abstractive Summarization

Shashi Narayan, Yao Zhao, Joshua Maynez, Gonçalo Simões, Vitaly Nikolaev, Ryan T. McDonald

Abstract

Modern neural summarization models often struggle with factual consistency and hallucinations. We present FROST (FReeze entity-level infOrmation in ab-stractive SummarizaTion with planning), a framework that introduces an intermediate "entity chain" planning step. By training models to generate ordered sequences of entities before the final summary, and employing a novel "drop-prompt" mechanism to filter hallucinations, FROST achieves state-of-the-art performance on XSum and SAMSum benchmarks while significantly improving entity specificity.

Introduction

While large-scale neural summarizers like PEGASUS and BART produce highly fluent text, they frequently suffer from a lack of grounding. These end-to-end models often "hallucinate" facts or entities not present in the source document because they lack an explicit content planning stage. Historically, summarization involved a pipeline of representation, planning, and synthesis. This research aims to reintegrate that planning phase into neural architectures to ensure summaries are both concise and factually anchored.

Methodology

The core innovation of FROST is the Entity Chain: a structured sequence of all named entities, dates, and numerical values intended for the summary. The framework operates through three primary stages:

  1. Target Augmentation: During training, target summaries are modified to include a prepended entity chain (e.g., [ENTITYCHAIN] Frozen | Disney [SUMMARY] Frozen is a Disney musical...).
  2. Architecture: Built upon the PEGASUS Transformer encoder-decoder, the model is trained to predict the entity chain first, then use that chain as a prompt to generate the prose summary.
  3. Pre-training and Fine-tuning: The planning objective is integrated into both the initial pre-training (predicting chains for masked sentences) and task-specific fine-tuning.
The FROST framework showing the pre-training (left) and fine-tuning (right) objectives centered on entity chain generation.

To further mitigate hallucinations at inference time, the authors introduce a Drop-Prompt mechanism. If the model predicts an entity in the chain that is absent from the source text, that entity is removed from the plan before the final summary generation begins.

Key Insights

  • Global Planning: Experimental results indicate that planning the entity chain for the entire summary at once (Summary-Level) is more effective than sentence-by-sentence planning, as it improves global coherence and reduces redundancy.
  • Controllability: The entity chain serves as a controllable prompt. Users can manipulate the chain—by reordering or swapping entities—to influence the focus of the generated summary without retraining.
  • Entity Specificity: FROST models demonstrate a superior ability to utilize specific entities from the source text rather than falling back on generic terms or incorrect associations.

Experimental Results

FROST was evaluated across four major benchmarks: CNN/DailyMail, XSum, SAMSum, and BillSum. The model achieved new state-of-the-art (SOTA) ROUGE scores on XSum (47.80 R1) and SAMSum (52.39 R1).

Performance comparison showing the incremental gains in ROUGE-2 and Entity F1 (ENTF1) when moving from PEGASUS to FROST with fine-tuning (F) and full pre-training plus fine-tuning (P+F).

Beyond ROUGE scores, the faithfulness of the model improved dramatically. On XSum, the drop-prompt mechanism increased entity precision (ENTPREC) from 0.806 to 0.943. Human evaluations confirmed these findings, with raters preferring FROST summaries for both factual accuracy and overall quality over standard PEGASUS baselines.

Conclusion and Future Work

FROST demonstrates that explicit entity planning is a powerful tool for improving the reliability of abstractive summarization. While the current implementation uses "soft" constraints (the chain acts as a prompt), future research may explore "hard" constraints to strictly enforce the inclusion of planned entities. Additionally, the reliance on external Named Entity Recognition (NER) tools suggests that improvements in NER technology will directly translate to even more robust summarization performance.

Summary updated

Abstract

We introduce a simple but flexible mechanism to learn an intermediate plan to ground the generation of abstractive summaries. Specifically, we prepend (or prompt) target summaries with entity chains—ordered sequences of entities mentioned in the summary. Transformer-based sequence-to-sequence models are then trained to generate the entity chain and then continue generating the summary conditioned on the entity chain and the input. We experimented with both pretraining and finetuning with this content planning objective. When evaluated on CNN/DailyMail, XSum, SAMSum, and BillSum, we demonstrate empirically that the grounded generation with the planning objective improves entity specificity and planning in summaries for all datasets, and achieves state-of-the-art performance on XSum and SAMSum in terms of rouge. Moreover, we demonstrate empirically that planning with entity chains provides a mechanism to control hallucinations in abstractive summaries. By prompting the decoder with a modified content plan that drops hallucinated entities, we outperform state-of-the-art approaches for faithfulness when evaluated automatically and by humans.
BibTeX
@article{narayan2021planning,
  author = {Shashi Narayan and Yao Zhao and Joshua Maynez and Gonçalo Simões and Vitaly Nikolaev and Ryan T. McDonald},
  title = {{Planning with Learned Entity Prompts for Abstractive Summarization}},
  journal = {Trans. Assoc. Comput. Linguistics},
  year = {2021},
  doi = {10.1162/tacl\_a\_00438},
}
Figure