Improving Faithfulness of Large Language Models in Summarization via Sliding Generation and Self-Consistency
Introduction
Large Language Models (LLMs) are highly proficient at generating fluent summaries, but they frequently struggle with factual accuracy, often producing "hallucinations"—information not present in the source text. Furthermore, LLMs exhibit a pronounced position bias, often prioritizing information at the beginning or end of a document while "losing" critical details in the middle. This is particularly problematic for long-form content like scientific papers. SliSum is introduced as a "plug-and-play" framework designed to address these issues without requiring additional training or external data.
The SliSum Framework
SliSum enhances faithfulness through a three-phase process that ensures every part of a document is processed under optimal conditions:
- Sliding Generation: Instead of processing a long document in one pass, SliSum breaks it into overlapping sliding windows. This exploits the "front-row advantage," where LLMs are most factually accurate when processing information at the start of their context window. Every segment of the text eventually appears at the beginning of a window.
- Events Filtering: Because windows overlap, the same events are summarized multiple times. SliSum uses lexical clustering (DBSCAN) to group similar sentences. Sentences that appear only once are flagged as potential hallucinations or noise and filtered out.
- Contradiction Detection & Aggregation: Within clusters of similar sentences, the LLM identifies semantic contradictions. A majority voting mechanism selects the most consistent version, which is then synthesized into a final, coherent global summary.
Experimental Evaluation
The researchers evaluated SliSum across four diverse benchmarks: short news (CNN/DM, XSum) and long scientific papers (PubMed, arXiv). These were tested using state-of-the-art models including LLaMA-2-13B, Claude-2, and GPT-3.5-Turbo.
Key Results and Insights
SliSum demonstrated significant improvements in factual consistency metrics like FactCC and SummaC across all datasets and models.
- Faithfulness Boost: On the CNN/DM dataset, Claude-2 with SliSum saw a 13.9% relative gain in FactCC scores. For long documents, the gains were even more dramatic; LLaMA-2's FactCC scores on the arXiv dataset jumped from 47.31 to 66.39.
- The Power of Aggregation: Ablation studies confirm that the combination of sliding windows and the aggregation mechanism is critical. As shown in the performance data, sliding windows with aggregation consistently outperform parallel windows or configurations without aggregation.
- Linear Scaling: Standard LLM processing complexity grows quadratically with document length. In contrast, SliSum’s windowed approach scales linearly ($O(L)$), making it theoretically more efficient for extremely long documents.
Discussion and Limitations
While SliSum effectively reduces hallucinations, it does come with increased computational overhead. Generating multiple local summaries and performing aggregation can take roughly twice as long and cost more in API tokens than a single-pass summary. Additionally, the system relies on the LLM's internal ability to recognize semantic contradictions during the aggregation phase. Future research may focus on more sophisticated clustering methods to handle highly abstractive summaries where wording varies significantly despite identical meanings.
Conclusion
SliSum provides a robust method for ensuring LLM summaries remain faithful to their source material. By restructuring the input process and enforcing self-consistency, it mitigates common pitfalls like position bias and hallucinations, particularly in the challenging domain of long-form summarization.
Summary updated