TL;DR
This paper explores how to make abstractive document summarization models generate factually consistent summaries by incorporating external knowledge. They propose a method using prefix-tuning combined with extracted factual relations as prompts, demonstrating improved factual accuracy and ROUGE scores.
What Problem Does This Paper Address?
Abstractive summarization, while producing more human-like summaries, often struggles with factual inconsistencies, where the generated summary contradicts the original text. Existing methods to inject knowledge into these models typically require complex structural modifications. This paper investigates a simpler approach: leveraging prompts to guide the model towards factual accuracy without altering its core architecture.
Methods
The core of the proposed method involves two key components:
- Open Information Extraction (OpenIE): This technique is used to extract factual knowledge in the form of relation triples (subject, predicate, object) from the source documents. These extracted relations serve as explicit factual prompts.
- Prefix-Tuning: This is a prompt-based learning technique where trainable continuous "prefix" vectors are prepended to the input text. In this work, these continuous prefixes are trained to steer the GPT-2 model towards generating summaries that are factually consistent with the discrete factual prompts (the extracted relations). The idea is that the model learns to attend to and incorporate the information provided by these prefixes.
The process involves:
- Extracting factual relations from the source text using OpenIE.
- Prepending these relations as discrete prompts to the original text.
- Training a set of continuous prefix prompts for GPT-2 on this modified input.
- Evaluating the generated summaries for factual consistency and overlap with the original text.
Datasets
The experiments were conducted on two standard datasets for abstractive summarization:
- CNN/Daily Mail: A large dataset of news articles and their bullet-point summaries.
- XSum: A dataset known for its highly abstractive summaries, sourced from BBC articles.
For training, a subset of 50,000 data instances was used for both datasets, with further reduction if the combined length of the source text and added relation exceeded GPT-2's input length limit.
Key Insights and Ideas
- Prompts as Knowledge Injectors: The paper highlights that simple prompting, by prepending extracted factual knowledge, can effectively guide abstractive summarization models towards factual consistency.
- Prefix-Tuning Synergy: Combining trainable continuous prefixes with discrete factual prompts offers a flexible way to enhance summarization without complex architectural changes. The continuous prefixes learn to interpret and utilize the discrete knowledge.
- CoCo Metric Limitations: While CoCo is proposed as a factual consistency metric, the study reveals its limitations. It can sometimes favor summaries that are closer to the original text's wording (even if less abstract) over those that are factually consistent with the intended meaning derived from added relations. This is particularly evident when comparing with human-annotated reference summaries.
- Abstractness Matters: The effectiveness of the approach, particularly in terms of ROUGE scores, is more pronounced on datasets like CNN/Daily Mail where summaries are less abstract. For highly abstract datasets like XSum, the gains are less dramatic, suggesting that the model's inherent abstractive capabilities play a significant role.
Key Results
- Improved ROUGE Scores: Summaries generated with added relations (G') consistently achieved higher ROUGE scores compared to those generated without added relations (G), indicating better overlap with target summaries. For CNN/Daily Mail, ROUGE-1 scores increased significantly, reaching up to 32.89 with a prefix length of 100.
- Factual Consistency Gains (with caveats): The study aimed to improve factual consistency. While case studies showed instances where the generated summaries (G') were factually correct and aligned with the added relations, the CoCo metric results were mixed. In many cases, G' had lower CoCo scores than G, suggesting that CoCo might not be the ideal metric for evaluating factual consistency in this context, especially when comparing against human references.
- Prefix Length Impact: Longer prefix lengths (e.g., 100) generally led to better performance in terms of ROUGE scores, suggesting that more expressive prefixes can better steer the model.
Limitations and Future Work
- CoCo Metric Suitability: The authors acknowledge that CoCo's effectiveness as a factual consistency metric is questionable in their experiments, as it sometimes favored summaries less aligned with the intended factual knowledge. They suggest further discussion on this issue.
- Relation Extraction Quality: The quality of the extracted relations directly impacts the summary. Errors or incompleteness in OpenIE can lead to incorrect summaries.
- Abstractness vs. Factuality: The trade-off between abstractness and factual consistency, especially on datasets like XSum, remains an area for further exploration.
- Future Directions: The authors suggest exploring:
- The optimal format and content of knowledge to be added.
- The ideal length of prefixes to train.
- The application of this approach to other NLP tasks like Question Answering and Information Extraction.
Summary updated