Semantic Self-Segmentation for Abstractive Summarization of Long Documents in Low-Resource Regimes

Gianluca Moro, Luca RagazziAAAI 2022 · DOI

Introduction

State-of-the-art Transformer models, such as BART and LED, have revolutionized natural language processing but face significant hurdles when applied to specialized, long-form content like legal documents. These challenges are twofold: hardware memory scarcity and data scarcity. The quadratic $O(n^2)$ complexity of the attention mechanism makes processing documents with tens of thousands of words impossible on standard consumer-grade hardware. Furthermore, fine-tuning these models typically requires massive labeled datasets, which are rarely available in niche domains.

The Semantic Self-Segmentation (Se3) approach addresses these issues by enabling high-quality abstractive summarization on mid-range GPUs with minimal training data. By intelligently partitioning long texts into semantically coherent chunks, Se3 avoids the destructive nature of simple truncation and acts as a powerful data augmentation strategy.

The Se3 Framework

The Se3 pipeline consists of three primary phases designed to optimize both memory usage and information retention:

  1. Metric Learning for Embeddings: To ensure segmentation is based on thematic meaning rather than simple word overlap, a domain-specific model (e.g., LEGAL-BERT) is fine-tuned using triplet and contrastive losses. This forces sentences from the same document section to be closer in vector space.
  2. Semantic Segmentation: Using these fine-tuned embeddings, the document is grouped into semantically coherent chunks. These chunks are sized (e.g., 1024 tokens) to fit within the memory constraints of the target Transformer model.
  3. Target Assignment: This phase serves as a data augmentation step. During training, sentences from the gold-standard summary are assigned to the input chunk they most closely resemble, typically determined by ROUGE-1 precision. This process transforms a single long document into multiple high-correlation source-target pairs.
The Se3 pipeline: From semantic segmentation of the document to target assignment and final concatenation.

At inference time, the model summarizes each chunk independently, and the resulting fragments are concatenated to produce the final generated summary.

Experimental Evaluation

The researchers evaluated Se3 on two legal benchmarks: AustLII (Australian Legal Case Reports, avg. ~7,300 words) and BillSum (US Congressional Bills, avg. ~1,600 words). These datasets represent the extremes of document length and provide a rigorous test for both memory efficiency and performance in data-starved environments.

Key Results

The Se3 approach demonstrated substantial improvements over standard baselines:

  • Performance Gains: On the long-form AustLII dataset, Se3 improved the ROUGE-1 score from 33.51 to 59.04 compared to a standard BART baseline that uses truncation.
  • Low-Resource Efficiency: Remarkably, with as few as 10 labeled training instances on the BillSum dataset, Se3 achieved results comparable to models trained on significantly larger datasets.
  • Optimal Chunking: The study identified a "sweet spot" at a chunk size of 1024 tokens, balancing context understanding with hardware efficiency.
Performance (ROUGE) and GPU memory usage across different chunk sizes for the BillSum dataset using the Se3-LED model.

Discussion and Future Work

The core innovation of Se3 lies in treating segmentation not just as a hardware workaround, but as a method for data augmentation. By splitting a document into $N$ chunks, the available training signal is multiplied. Furthermore, the use of ROUGE-1 precision for target assignment ensures that summary sentences are strictly grounded in their corresponding chunks, reducing the likelihood of model hallucinations.

Despite its success, the modular nature of Se3 introduces challenges such as potential redundancy and the loss of cross-chunk context. Future research may explore memory-based neural layers or knowledge graphs to maintain global context across the entire document while preserving the memory efficiency of the segmentation approach.

Summary updated

BibTeX
@inproceedings{moro2022semantic,
  author = {Gianluca Moro and Luca Ragazzi},
  title = {{Semantic Self-Segmentation for Abstractive Summarization of Long Documents in Low-Resource Regimes}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2022},
  doi = {10.1609/aaai.v36i10.21357},
}
Figure