Abstract
Modern neural summarization models often struggle with factual consistency and hallucinations. We present FROST (FReeze entity-level infOrmation in ab-stractive SummarizaTion with planning), a framework that introduces an intermediate "entity chain" planning step. By training models to generate ordered sequences of entities before the final summary, and employing a novel "drop-prompt" mechanism to filter hallucinations, FROST achieves state-of-the-art performance on XSum and SAMSum benchmarks while significantly improving entity specificity.
Introduction
While large-scale neural summarizers like PEGASUS and BART produce highly fluent text, they frequently suffer from a lack of grounding. These end-to-end models often "hallucinate" facts or entities not present in the source document because they lack an explicit content planning stage. Historically, summarization involved a pipeline of representation, planning, and synthesis. This research aims to reintegrate that planning phase into neural architectures to ensure summaries are both concise and factually anchored.
Methodology
The core innovation of FROST is the Entity Chain: a structured sequence of all named entities, dates, and numerical values intended for the summary. The framework operates through three primary stages:
- Target Augmentation: During training, target summaries are modified to include a prepended entity chain (e.g.,
[ENTITYCHAIN] Frozen | Disney [SUMMARY] Frozen is a Disney musical...). - Architecture: Built upon the PEGASUS Transformer encoder-decoder, the model is trained to predict the entity chain first, then use that chain as a prompt to generate the prose summary.
- Pre-training and Fine-tuning: The planning objective is integrated into both the initial pre-training (predicting chains for masked sentences) and task-specific fine-tuning.
To further mitigate hallucinations at inference time, the authors introduce a Drop-Prompt mechanism. If the model predicts an entity in the chain that is absent from the source text, that entity is removed from the plan before the final summary generation begins.
Key Insights
- Global Planning: Experimental results indicate that planning the entity chain for the entire summary at once (Summary-Level) is more effective than sentence-by-sentence planning, as it improves global coherence and reduces redundancy.
- Controllability: The entity chain serves as a controllable prompt. Users can manipulate the chain—by reordering or swapping entities—to influence the focus of the generated summary without retraining.
- Entity Specificity: FROST models demonstrate a superior ability to utilize specific entities from the source text rather than falling back on generic terms or incorrect associations.
Experimental Results
FROST was evaluated across four major benchmarks: CNN/DailyMail, XSum, SAMSum, and BillSum. The model achieved new state-of-the-art (SOTA) ROUGE scores on XSum (47.80 R1) and SAMSum (52.39 R1).
Beyond ROUGE scores, the faithfulness of the model improved dramatically. On XSum, the drop-prompt mechanism increased entity precision (ENTPREC) from 0.806 to 0.943. Human evaluations confirmed these findings, with raters preferring FROST summaries for both factual accuracy and overall quality over standard PEGASUS baselines.
Conclusion and Future Work
FROST demonstrates that explicit entity planning is a powerful tool for improving the reliability of abstractive summarization. While the current implementation uses "soft" constraints (the chain acts as a prompt), future research may explore "hard" constraints to strictly enforce the inclusion of planned entities. Additionally, the reliance on external Named Entity Recognition (NER) tools suggests that improvements in NER technology will directly translate to even more robust summarization performance.
Summary updated