Improving Faithfulness of Large Language Models in Summarization via Sliding Generation and Self-Consistency

Taiji Li, Zhi Li, Yin Zhang

Improving Faithfulness of Large Language Models in Summarization via Sliding Generation and Self-Consistency

Introduction

Large Language Models (LLMs) are highly proficient at generating fluent summaries, but they frequently struggle with factual accuracy, often producing "hallucinations"—information not present in the source text. Furthermore, LLMs exhibit a pronounced position bias, often prioritizing information at the beginning or end of a document while "losing" critical details in the middle. This is particularly problematic for long-form content like scientific papers. SliSum is introduced as a "plug-and-play" framework designed to address these issues without requiring additional training or external data.

The SliSum Framework

SliSum enhances faithfulness through a three-phase process that ensures every part of a document is processed under optimal conditions:

  1. Sliding Generation: Instead of processing a long document in one pass, SliSum breaks it into overlapping sliding windows. This exploits the "front-row advantage," where LLMs are most factually accurate when processing information at the start of their context window. Every segment of the text eventually appears at the beginning of a window.
  2. Events Filtering: Because windows overlap, the same events are summarized multiple times. SliSum uses lexical clustering (DBSCAN) to group similar sentences. Sentences that appear only once are flagged as potential hallucinations or noise and filtered out.
  3. Contradiction Detection & Aggregation: Within clusters of similar sentences, the LLM identifies semantic contradictions. A majority voting mechanism selects the most consistent version, which is then synthesized into a final, coherent global summary.
Overview of the SliSum framework, illustrating the sliding window process, local summary generation, clustering, and final aggregation.

Experimental Evaluation

The researchers evaluated SliSum across four diverse benchmarks: short news (CNN/DM, XSum) and long scientific papers (PubMed, arXiv). These were tested using state-of-the-art models including LLaMA-2-13B, Claude-2, and GPT-3.5-Turbo.

Key Results and Insights

SliSum demonstrated significant improvements in factual consistency metrics like FactCC and SummaC across all datasets and models.

  • Faithfulness Boost: On the CNN/DM dataset, Claude-2 with SliSum saw a 13.9% relative gain in FactCC scores. For long documents, the gains were even more dramatic; LLaMA-2's FactCC scores on the arXiv dataset jumped from 47.31 to 66.39.
  • The Power of Aggregation: Ablation studies confirm that the combination of sliding windows and the aggregation mechanism is critical. As shown in the performance data, sliding windows with aggregation consistently outperform parallel windows or configurations without aggregation.
  • Linear Scaling: Standard LLM processing complexity grows quadratically with document length. In contrast, SliSum’s windowed approach scales linearly ($O(L)$), making it theoretically more efficient for extremely long documents.
Comparison of FactCC scores across different configurations (Parallel vs. Sliding windows, with/without aggregation) on four datasets, showing the superior performance of the full SliSum approach.

Discussion and Limitations

While SliSum effectively reduces hallucinations, it does come with increased computational overhead. Generating multiple local summaries and performing aggregation can take roughly twice as long and cost more in API tokens than a single-pass summary. Additionally, the system relies on the LLM's internal ability to recognize semantic contradictions during the aggregation phase. Future research may focus on more sophisticated clustering methods to handle highly abstractive summaries where wording varies significantly despite identical meanings.

Conclusion

SliSum provides a robust method for ensuring LLM summaries remain faithful to their source material. By restructuring the input process and enforcing self-consistency, it mitigates common pitfalls like position bias and hallucinations, particularly in the challenging domain of long-form summarization.

Summary updated

BibTeX
@inproceedings{li2024improving,
  author = {Taiji Li and Zhi Li and Yin Zhang},
  title = {{Improving Faithfulness of Large Language Models in Summarization via Sliding Generation and Self-Consistency}},
  booktitle = {International Conference on Computational Linguistics},
  year = {2024},
}
Figure