From Sparse to Dense: GPT-4 Summarization with Chain of Density Prompting

Griffin Adams, Alex Fabbri, Faisal Ladhak, Eric Lehman, Noémie Elhadad

Abstract

Selecting the “right” amount of information to include in a summary is a difficult task. A good summary should be detailed and entity-centric without being overly dense and hard to follow. To better understand this tradeoff, we solicit increasingly dense GPT-4 summaries with what we refer to as a “Chain of Density” (CoD) prompt. Specifically, GPT-4 generates an initial entity-sparse summary before iteratively incorporating missing salient entities without increasing the length. Summaries generated by CoD are more abstractive, exhibit more fusion, and have less of a lead bias than GPT-4 summaries generated by a vanilla prompt. We conduct a human preference study on 100 CNN DailyMail articles and find that humans prefer GPT-4 summaries that are more dense than those generated by a vanilla prompt and almost as dense as human written summaries. Qualitative analysis supports the notion that there exists a tradeoff between informativeness and readability. 500 annotated CoD summaries, as well as an extra 5,000 unannotated summaries, are freely available on HuggingFace().

Introduction: The Informativeness-Readability Tradeoff

In automatic summarization, a fundamental tension exists between informativeness and readability. A summary that is too sparse may omit critical details, while one that is excessively dense can become a "word salad"—a collection of facts that is difficult for a human to parse.

While Large Language Models (LLMs) like GPT-4 are highly capable summarizers, they often default to a "vanilla" style that prioritizes safety and readability over information density. These models typically exhibit a "lead bias," focusing on the beginning of an article and missing salient entities buried deeper in the text. This research explores the limits of information packing: how much detail can be integrated into a fixed word budget before quality degrades, and what level of density do humans actually prefer?

Methodology: The Chain of Density (CoD) Prompt

To investigate this, the authors developed the Chain of Density (CoD) prompting technique. Unlike standard prompts that request a single summary, CoD initiates an iterative five-step process designed to force the model to increase information density while maintaining a constant length (approximately 80 words).

The process follows these steps:

  1. Initial Draft: GPT-4 generates an entity-sparse summary.
  2. Identify & Integrate: The model identifies 1-3 "missing entities" from the source text that are relevant and novel.
  3. Rewrite: The model rewrites the summary to include these new entities without increasing the total word count.

To achieve this, the model must employ sophisticated linguistic maneuvers such as compression (removing filler), abstraction (paraphrasing), and fusion (combining multiple ideas into a single sentence).

Example of five iterative CoD summaries showing increasing entity density for a fixed length.

Results: Identifying the "Goldilocks Zone"

The study utilized 100 randomly sampled articles from the CNN/DailyMail benchmark for a human preference study. The results revealed a clear "Goldilocks zone" for information density.

Standard "Vanilla" GPT-4 summaries were found to be significantly less dense (~0.122 entities per token) than human-written reference summaries (~0.151 entities per token). Through the CoD process, GPT-4 was able to bridge this gap. Human annotators showed a strong preference for the middle iterations (Steps 2, 3, and 4), with Step 3 emerging as the peak. At Step 3, the entity density reached ~0.148 entities per token—nearly identical to human-written summaries.

Entity density across CoD steps compared to human and vanilla GPT-4 summaries.

However, as density increased to Step 5, preference declined. The summaries became too dense, leading to "awkward fusions" where sentences became grammatically strained or logically cluttered to save space.

Analysis: Beyond Density

The CoD prompt does more than just add entities; it fundamentally changes the model's summarization behavior. As density increases, the summaries become more abstractive, moving away from simple extraction (copy-pasting) toward complex rewriting. They also exhibit less lead bias, pulling facts from throughout the article rather than just the opening sentences, and show higher levels of fusion, combining multiple distinct facts into single, cohesive sentences.

Conclusion and Future Work

This research demonstrates that humans prefer summaries with a higher information density than what LLMs typically produce by default. The CoD technique successfully pushes GPT-4 to match human-level density.

Future work includes exploring domain-specific "ideal densities" for legal or medical texts and using the CoD dataset (500 annotated and 5,000 unannotated summaries released on HuggingFace) for model distillation. By training smaller, open-source models on CoD outputs, researchers hope to achieve high-density summarization without the need for iterative prompting.

Summary updated

Abstract

Selecting the “right” amount of information to include in a summary is a difficult task. A good summary should be detailed and entity-centric without being overly dense and hard to follow. To better understand this tradeoff, we solicit increasingly dense GPT-4 summaries with what we refer to as a “Chain of Density” (CoD) prompt. Specifically, GPT-4 generates an initial entity-sparse summary before iteratively incorporating missing salient entities without increasing the length. Summaries generated by CoD are more abstractive, exhibit more fusion, and have less of a lead bias than GPT-4 summaries generated by a vanilla prompt. We conduct a human preference study on 100 CNN DailyMail articles and find that humans prefer GPT-4 summaries that are more dense than those generated by a vanilla prompt and almost as dense as human written summaries. Qualitative analysis supports the notion that there exists a tradeoff between informativeness and readability. 500 annotated CoD summaries, as well as an extra 5,000 unannotated summaries, are freely available on HuggingFace().
BibTeX
@misc{adams2023from,
  author = {Griffin Adams and Alex Fabbri and Faisal Ladhak and Eric Lehman and Noémie Elhadad},
  title = {{From Sparse to Dense: GPT-4 Summarization with Chain of Density Prompting}},
  year = {2023},
  eprint = {2309.04269},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2309.04269},
}
Figure