Two-step Text Summarization for Long-form Biographical Narrative Genre

Avi BleiweissACL 2023

TL;DR

This paper introduces a novel two-step summarization approach for long-form biographical narratives, focusing on improving factual consistency and relevance. By first extracting salient sentences and then abstractively summarizing them, the method achieves competitive performance against baselines on a custom biographical dataset, addressing the challenges of generating faithful summaries from lengthy texts.

What Problem Does This Paper Address?

Generating accurate and coherent summaries for long-form texts, particularly biographical narratives, is a significant challenge. These texts, often spanning hundreds of thousands of words, require readers to immerse themselves for extended periods. While automatic summarization can aid readers in deciding whether a book is worth their time and assist critics, existing methods struggle with factual inconsistencies and hallucinations – where generated summaries contain information not present in the source text. This paper tackles the problem of producing high-quality, factually consistent abstractive summaries from lengthy biographical novels, a genre that presents unique challenges due to its progressive plot structure and the need to capture nuanced character development.

Methods

The authors propose a two-step summarization framework:

  1. Extractive Summary Generation (e-summary): This step focuses on identifying and extracting the most important sentences from each chapter. They employ LexRank, a graph-based algorithm that ranks sentences based on their centrality within a sentence similarity graph. This graph is constructed using sentence embeddings generated by Sentence-BERT (SBERT), a model known for producing semantically meaningful embeddings. The output of this stage is a collection of top-ranked sentences, aiming to retain factual accuracy by directly using source text.
  2. Abstractive Summary Generation (a-summary): In the second step, an abstractive summary is generated from the extracted sentences of the first stage. This is achieved using a Transformer-based sequence-to-sequence model, specifically a distilled version of BART (DistilBART-CNN-12-6). This approach aims to rephrase the extracted content in a more concise and fluent manner, while leveraging the factual grounding provided by the extractive step to mitigate hallucinations.

Datasets

The authors created a custom biographical literary dataset for their experiments. This dataset consists of four biographies:

  • Narrative of the Life of Frederick Douglass
  • Life on the Mississippi by Mark Twain
  • Personal Memoirs of U. S. Grant by Ulysses S. Grant
  • Memoirs of Napoleon Bonaparte by Louis Antoine Fauvelet de Bourrienne

These texts were sourced from Project Gutenberg and comprise 256 chapters, totaling over four million words. The dataset is made publicly available.

Key Insights and Ideas

  • Two-step approach for faithfulness: The core idea is that by first extracting salient sentences (which are inherently factual) and then abstractively rephrasing them, the model can achieve better factual consistency than a direct end-to-end abstractive summarization.
  • SBERT for semantic embeddings: Utilizing SBERT for generating sentence embeddings is crucial for accurately capturing semantic similarity, which in turn improves the effectiveness of LexRank for sentence importance ranking.
  • Chapter-level summarization: The approach breaks down the summarization task by chapter, making it more manageable for long-form narratives compared to treating the entire book as a single document.
  • Information Alignment for Evaluation: The paper adopts and extends the concept of information alignment (Deng et al., 2021) to evaluate summary consistency and relevance, providing a more robust evaluation framework.

Key Results

  • Improved ROUGE scores: The proposed two-step method achieved competitive ROUGE-2 and ROUGE-L scores compared to external baselines. For ROUGE-2, the method exceeded the state-of-the-art Gidiotis and Tsoumakas (2020) by 0.11 F1.
  • Factual Consistency: The paper highlights that their method produces summaries with compelling factual consistency, with scores around 0.77-0.81 on their test set, significantly outperforming previous consistency metrics.
  • Compression Ratio: The first stage of summarization achieves a compression ratio of roughly 40:1 (source chapter text to e-summary), and the second stage further compresses the text by an order of magnitude.
  • Single-tier vs. Two-step: A single-tier summarization approach (direct abstractive summarization) resulted in significantly lower ROUGE scores (declining quadruply for ROUGE-2 and triply for ROUGE-L) compared to their two-step method.

Limitations and Future Work

  • Pretraining Domain: The summarization model was pretrained exclusively on news datasets, while experiments were conducted on biographical narratives. This domain mismatch might limit generalizability and propagate inductive biases.
  • Language Specificity: The study focused solely on English summarization, and findings may not directly apply to long multi-lingual documents.
  • Generalization to Other Genres: The current work is limited to biographical narratives. Investigating summarization quality for other domains like scientific articles, patent documents, or meeting discourses is suggested for future work.
  • Human Evaluation Challenges: The paper acknowledges that human evaluation of long-form content is challenging, time-consuming, and can yield irreproducible results, suggesting the need for more standardized and efficient human evaluation protocols.
  • Finetuning Performance: Finetuning the BART model on the biographical dataset resulted in lower scores compared to the pretrained model, likely due to the smaller size and genre difference of the finetuning dataset.

Summary updated

BibTeX
@inproceedings{bleiweiss2023two,
  author = {Avi Bleiweiss},
  title = {{Two-step Text Summarization for Long-form Biographical Narrative Genre}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2023},
}
Figure