TL;DR
This paper introduces "FROST," a novel approach to abstractive summarization that uses "entity chains" (ordered lists of key entities) as an intermediate planning step. By training models to first generate an entity chain and then the summary, FROST significantly improves summary quality, entity specificity, and crucially, reduces hallucinations, achieving state-of-the-art results on several benchmarks.
What Problem Does This Paper Address?
Modern abstractive summarization models, while powerful, often struggle with "hallucinations" – generating factual inaccuracies or information not present in the source document. This is partly because they lack a clear intermediate planning step that traditional summarization methods employed. The authors aim to bridge this gap by introducing a flexible mechanism to explicitly plan the summary's content, focusing on key entities, before generating the final text. This planning step is intended to ground the generation process and improve faithfulness.
Methods
The core innovation is the entity chain, an ordered sequence of entities (named entities, dates, numbers) extracted from the target summary. The authors propose a new training objective for sequence-to-sequence models (like Transformers) where the model first generates this entity chain and then, conditioned on both the input document and the generated entity chain, produces the summary. This is achieved by concatenating the entity chain and summary as the target sequence during training.
They explore two main ways to integrate this:
- Finetuning: Taking pre-trained summarization models (like PEGASUS) and finetuning them with the entity chain objective.
- Pretraining: Modifying pretraining objectives to incorporate entity chain generation from the start.
A key application of this method is controlling hallucinations. By modifying the predicted entity chain at inference time (e.g., removing entities not found in the source document), they can prompt the decoder to generate more faithful summaries. This "drop-prompt" mechanism is a novel way to leverage the learned planning step for controlled generation.
Datasets
The authors evaluate their approach on four standard abstractive summarization datasets:
- CNN/DailyMail: Highlight generation.
- XSum: Extreme summarization (very short summaries).
- SAMSum: Dialogue summarization.
- BillSum: US Congressional bills summarization.
For pretraining, they use large web corpora:
- C4: Common Crawl dataset.
- HugeNews: A collection of news articles.
Key Insights and Ideas
- Entity Chains as a Planning Mechanism: The core idea that an ordered list of entities can serve as an effective intermediate "plan" for abstractive summarization is a significant contribution. It provides a structured way to guide the generation process.
- Improved Entity Specificity and Planning: Explicitly planning with entity chains leads to summaries that are more specific about entities and better organized in terms of entity order.
- Hallucination Control: The "drop-prompt" mechanism, where predicted entity chains are filtered to remove unsupported entities before generating the summary, is a powerful and intuitive way to combat factual inaccuracies.
- Pretraining vs. Finetuning: While finetuning existing models with the entity chain objective yields improvements, pretraining models from scratch or continuing pretraining with this objective shows even greater benefits, especially when starting from a summarization-pretrained model.
- Faithfulness is Not Just Entailment: The authors observe that standard entailment scores are not always reliable indicators of faithfulness, especially for extractive summaries, highlighting the need for more nuanced evaluation.
Key Results
- State-of-the-Art Performance: FROST achieves state-of-the-art ROUGE scores on XSum and SAMSum datasets.
- Improved Entity Specificity: The models consistently show higher entity specificity (ENTF1 scores) across all tested datasets.
- Significant Hallucination Reduction: The drop-prompt mechanism, when applied to FROST models, significantly outperforms state-of-the-art approaches in faithfulness, as measured by both automatic metrics (ENTPREC) and human evaluations. For example, ENTPREC scores reach 0.943 on extractive test sets.
- Competitive Performance: While not always outperforming specialized models on CNN/DailyMail in terms of ROUGE, FROST excels in entity planning and specificity.
Limitations and Future Work
- Oracle Prompts: The paper uses oracle entity chains (derived from reference summaries) for some experiments to establish upper bounds, highlighting the potential of perfect planning.
- Extractive vs. Abstractive: The effectiveness of data filtering for faithfulness is more pronounced on datasets with extractive entity chains. For highly abstractive summaries where novel entities are expected, the benefits of filtering are less clear.
- External Systems: Unlike some controlled summarization methods, FROST does not rely on external systems for augmentation, making it more self-contained. However, generating entity chains at inference time could still benefit from an external chain generator for more complex control.
- Future Work: The authors suggest exploring FROST for synthetic data generation and further investigating its use in controlling topical and style diversity in summaries. They also note that while their method doesn't strictly enforce the summary to be only about the entity chain, future work could focus on stronger constraints.
Summary updated