Introduction
Neural abstractive summarization models have achieved high levels of fluency but remain prone to hallucinations—generating content not supported by the source text. This issue is often rooted in the training data itself. Large-scale datasets like XSUM are inherently "noisy," containing reference summaries with external information that models learn to mimic.
While filtering out noisy samples can improve factuality, it often results in a significant loss of informativeness. As shown in the table below, training only on filtered "clean" data leads to a sharp decline in ROUGE scores and entity recall, creating a trade-off between being factual and being useful.
Methodology: Contrastive Parameter Ensembling (CaPE)
To address this, the authors introduce Contrastive Parameter Ensembling (CaPE). Instead of discarding noisy data, CaPE uses it to define what the model should not do. The method involves five steps:
- Base Training: A base model ($\theta_B$) is trained on the full, noisy dataset.
- Data Partitioning: The training set is split into "clean" and "noisy" subsets using automated metrics like Entity Overlap and Dependency Arc Entailment (DAE).
- Expert/Anti-Expert Fine-tuning: The base model is briefly fine-tuned on the clean subset to create an Expert ($\theta_E$) and on the noisy subset to create an Anti-Expert ($\theta_{\bar{E}}$).
- Parameter Ensembling: The final model weights are calculated by steering the base model toward the expert and away from the anti-expert: $$\theta_{CaPE} = \theta_B + \alpha(\theta_E - \theta_{\bar{E}})$$
This approach allows the model to retain the broad knowledge of the full dataset while explicitly penalizing hallucinatory behaviors learned from noise. Crucially, because the ensembling happens in the parameter space, there is zero inference overhead; the resulting model is identical in architecture and speed to the original.
Key Results and Insights
The researchers evaluated CaPE on the XSUM and CNN/DM datasets, finding that it successfully navigates the factuality-informativeness trade-off.
- Factuality Gains: CaPE improved summary-level entailment accuracy (DAE) by up to 16.69% on XSUM and 15.38% on CNN/DM.
- Preserving Informativeness: Unlike simple data filtering, CaPE maintains ROUGE scores comparable to the base model. The mixing coefficient $\alpha$ serves as a "control knob," allowing users to tune the balance between factual strictness and summary detail.
- Human Evaluation: Human judges preferred CaPE-generated summaries for their factual consistency more often than the base model (19% vs 14% on XSUM).
- Efficiency: The additional fine-tuning steps are computationally inexpensive, adding less than 14% to the total training time.
Conclusion
CaPE offers a practical, production-ready solution for reducing hallucinations in summarization. By leveraging "anti-experts" to define negative behaviors in parameter space, it avoids the pitfalls of data filtering and the latency costs of traditional ensembling. Future work may extend this contrastive approach to other domains, such as medical or legal summarization, where factual precision is paramount.
Summary updated