From Sparse to Dense: GPT-4 Summarization with Chain of Density Prompting

Griffin Adams, Alexander R. Fabbri, Faisal Ladhak, Eric Lehman, Noémie Elhadad

Abstract

Standard Large Language Model (LLM) summaries often prioritize brevity over detail, resulting in "sparse" outputs that omit salient information. This paper introduces Chain of Density (CoD) prompting, an iterative technique designed to increase the information density of GPT-4 summaries without increasing their length. Through a series of five rewrites, the model identifies and integrates missing entities while maintaining a fixed word count. Our findings indicate that humans prefer denser summaries than those typically produced by zero-shot prompting, identifying a "sweet spot" of density that aligns with high-quality human writing.

Introduction: The Informativeness-Readability Trade-off

In automatic summarization, researchers face a persistent challenge: balancing informativeness (including all relevant details) with readability (ensuring the text is not overly dense). While LLMs like GPT-4 are proficient at condensing text, their default summaries tend to be overly sparse. They often focus on a few central themes but neglect specific salient entities—such as people, locations, and events—to maintain a concise, accessible style. Conversely, summaries that are too dense risk becoming "word salad," sacrificing coherence and increasing the likelihood of factual errors. This study explores the optimal level of information density and proposes a method to steer LLMs toward that limit.

Methodology: The Chain of Density (CoD) Framework

The authors developed the Chain of Density (CoD) strategy, an iterative prompting framework that forces the model to pack more information into a fixed-length summary. The researchers utilized the CNN/DailyMail dataset for evaluation, creating a benchmark of 500 annotated and 5,000 unannotated summaries.

The Chain of Density (CoD) prompting framework and resulting summary iterations.

The process follows five distinct steps:

  1. Initial Generation: GPT-4 creates an entity-sparse summary of approximately 80 words.
  2. Entity Identification: The model identifies 1–3 salient entities from the source text that were missing from the previous draft.
  3. Iterative Rewriting: The model rewrites the summary to include these new entities alongside all previously included ones.
  4. Length Constraint: The summary must remain roughly the same length. This forces the model to employ advanced writing techniques such as abstraction, fusion (merging ideas), and compression.
  5. Repetition: This cycle repeats five times, yielding five summaries of increasing density for each article.

Key Insights: Stylistic Evolution and Density Metrics

The study utilizes "entities per token" as a primary metric for information density. Vanilla GPT-4 summaries typically exhibit a density of 0.122, which is significantly lower than the 0.151 average found in human-written summaries.

As the CoD process progresses, the summaries undergo significant stylistic shifts. They become increasingly abstractive, moving away from direct extraction to using the model's own synthesized language. Furthermore, the summaries show a reduction in lead bias—the common LLM tendency to over-represent the beginning of an article—as the model reaches deeper into the source text to find new salient entities in later iterations.

Results: Identifying the Human "Sweet Spot"

The research identifies a clear "breaking point" where density begins to degrade quality. While informativeness peaks at Step 4, metrics for coherence and overall quality begin to decline after the first few iterations as the text becomes overly cramped.

Information density (Entities/Tokens) across CoD steps compared to human and vanilla GPT-4 baselines.

Human evaluation revealed a distinct preference for increased density:

  • The Sweet Spot: Most humans (61%) preferred summaries from Step 3 or higher. Step 2 was the most common individual "first place" choice (30.8%).
  • Alignment with Human Writing: The preferred density levels (Steps 2 and 3) closely mirror the density of professional human-written summaries.
  • Entity Growth: By the fifth iteration, summaries contained an average of 12.1 entities—nearly double the 6.4 entities found in Step 1—within the same word count.

Conclusion and Future Directions

Chain of Density prompting demonstrates that LLMs can be steered to produce more informative summaries that better align with human preferences. However, the study notes limitations, including the subjectivity of "quality" (indicated by a low Fleiss’ kappa of 0.112) and the reliance on GPT-4's high-level reasoning. Future work may focus on density distillation, using CoD-generated data to fine-tune smaller, open-source models to produce high-density summaries more efficiently.

Summary updated

Abstract

Selecting the “right” amount of information to include in a summary is a difficult task. A good summary should be detailed and entity-centric without being overly dense and hard to follow. To better understand this tradeoff, we solicit increasingly dense GPT-4 summaries with what we refer to as a “Chain of Density” (CoD) prompt. Specifically, GPT-4 generates an initial entity-sparse summary before iteratively incorporating missing salient entities without increasing the length. Summaries generated by CoD are more abstractive, exhibit more fusion, and have less of a lead bias than GPT-4 summaries generated by a vanilla prompt. We conduct a human preference study on 100 CNN DailyMail articles and find that humans prefer GPT-4 summaries that are more dense than those generated by a vanilla prompt and almost as dense as human written summaries. Qualitative analysis supports the notion that there exists a tradeoff between informativeness and readability. 500 annotated CoD summaries, as well as an extra 5,000 unannotated summaries, are freely available on HuggingFace().
BibTeX
@misc{adams2023from,
  author = {Griffin Adams and Alexander R. Fabbri and Faisal Ladhak and Eric Lehman and Noémie Elhadad},
  title = {{From Sparse to Dense: GPT-4 Summarization with Chain of Density Prompting}},
  year = {2023},
  eprint = {2309.04269},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2309.04269},
}
Figure