Automatic Summarization of Long Documents (Student Abstract)

Naman Chhibbar, Jugal KalitaAAAI 2025 · DOI

TL;DR

This paper tackles the challenge of summarizing extremely long documents, a task that current transformer-based Large Language Models (LLMs) struggle with due to their limited input context windows. The authors propose novel, architecture-agnostic methods that allow LLMs to efficiently process texts far exceeding their native context size, achieving competitive summary quality with significant improvements in semantic evaluation metrics like BERTScore.

What Problem Does This Paper Address?

The internet is awash with text, making it increasingly difficult to extract key information efficiently. Automatic text summarization is a vital tool for this, but a significant hurdle exists for current state-of-the-art transformer models: their limited context window. This limitation, stemming from the quadratic complexity of the attention mechanism, prevents them from effectively processing documents that are much longer than their predefined input size. This means valuable information in extensive texts, like research papers, legal documents, or books, can be missed or poorly summarized. This paper aims to overcome this input size limitation without altering the core LLM architecture, enabling better utilization of their full potential for long document summarization.

Methods

The authors introduce three novel algorithms designed to enable LLMs to handle long documents:

  • Central Truncation: This method, adapted from prior work on long text classification, involves keeping only the beginning and end portions of the document. A new hyperparameter allows control over the fraction of text retained from the start. This is a time-efficient approach.
  • Document Skimming: Inspired by speed-reading techniques, this method first segments the document and then uniformly samples these segments. This ensures exposure to all parts of the document while maintaining efficiency. The authors also explore removing redundant segments, either before or after sampling, to further improve summary quality and prevent repetition. Pre-sampling removal is more computationally intensive but ensures better context utilization, while post-sampling removal is more efficient but might not fully leverage the context. Over-sampling is used to mitigate this.
  • Summarization with Keyword Extraction: This approach uses Latent Dirichlet Allocation (LDA) to extract topic keywords from the text. These keywords are then used to score segments via cosine similarity with a sentence transformer. Segments with the highest scores are selected for summarization, aiming to intelligently guide the selection of important content.

Crucially, these methods are designed to be integrated into existing summarization pipelines without requiring any architectural modifications to the LLM itself.

Datasets

The experiments were conducted on two large-scale datasets:

  • GovReport: This dataset contains documents with maximum word counts of 73,815.
  • BigPatent: This dataset features documents with maximum word counts of 71,027.

These datasets are notable for their substantial length, making them suitable for evaluating methods designed to handle long documents.

Key Insights and Ideas

  • Architecture-Agnostic Solutions: The most significant insight is that effective long document summarization can be achieved without modifying the underlying LLM architecture. This makes the proposed methods highly practical and broadly applicable to various existing LLMs.
  • Simple Methods Can Be Powerful: The "Central Truncation" method, despite its simplicity, demonstrates good results, highlighting that even straightforward approaches can be effective for this challenging task.
  • BERTScore as a Superior Metric for Semantic Similarity: The authors emphasize the importance of BERTScore over traditional ROUGE scores for evaluating summaries of long documents. They hypothesize that while ROUGE scores might be lower, the higher BERTScore achieved by their methods indicates better capture of semantic meaning and coherence, leading to superior summaries.
  • Potential for Larger Models: The fact that their methods achieve competitive results even when used with LLMs having significantly smaller context sizes than the baselines suggests that these algorithms have even greater potential when applied to models with larger native context windows.

Key Results

The paper reports experimental results on the GovReport and BigPatent datasets, evaluating summaries using ROUGE-1 (R-1), ROUGE-2 (R-2), ROUGE-L (R-L), and BERTScore.

  • "Document Skimming" with post-sampling removal and "Summarization with Keyword Extraction" achieved the best overall results, particularly in terms of BERTScore.
  • The proposed methods showed a significant increase in BERTScore compared to baselines like Unlimiformer.
  • While ROUGE-2 scores were sometimes lower than baselines, ROUGE-1 and ROUGE-L scores were competitive.
  • For example, on the GovReport dataset, "Summarization /w Extraction /w GPT-3.5 Turbo (4,096)" achieved a BERTScore of 86.20, significantly higher than some baselines. Similarly, "Central truncation /w LongT5 (4,096)" achieved 82.19 on the BigPatent dataset.

Limitations and Future Work

  • Computational Expense: The "Summarization with Keyword Extraction" method, while achieving strong results, is noted as being computationally expensive.
  • Segmenter Uniformity: The authors acknowledge that the choice and uniformity of segment length are crucial for pipeline performance. They encourage future work to explore different segmenters and focus on ensuring segment length uniformity while preserving coherence.
  • Broader Applicability: While experiments focused on summarization, the authors hypothesize that their methods could be applied to other NLP tasks requiring the processing of long texts. Future work could explore this broader applicability.

Summary updated

BibTeX
@inproceedings{chhibbar2025automatic,
  author = {Naman Chhibbar and Jugal Kalita},
  title = {{Automatic Summarization of Long Documents (Student Abstract)}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2025},
  doi = {10.1609/aaai.v39i28.35243},
}
Figure