A Simple Theoretical Model of Importance for Summarization

Maxime PeyrardACL 2019 · OpenReview

Abstract: Maxime Peyrard. Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics.

2019.

Introduction

Automatic text summarization has long relied on empirical heuristics and "black-box" models. While systems often perform well on specific benchmarks, the fundamental concept of "importance"—the criteria by which information is selected—has lacked a rigorous mathematical definition. Most existing methods use proxies like sentence centrality or word frequency, which, while intuitive, are susceptible to adversarial exploitation and lack a generalizable framework. This paper addresses this gap by proposing a formal information-theoretic definition of Importance that unifies the concepts of Relevance, Redundancy, and Informativeness.

Theoretical Framework

The proposed model is rooted in Information Theory, utilizing Shannon’s entropy and cross-entropy to define the pillars of summarization:

  1. Redundancy (Entropy): A summary should minimize redundancy by maximizing entropy, ensuring it covers a diverse set of semantic units.
  2. Relevance (Cross-Entropy): A summary is relevant if it minimizes the cross-entropy with the source document ($D$), effectively reducing the "surprise" relative to the source.
  3. Informativeness (Cross-Entropy with Background): Informativeness is defined by the cross-entropy between the summary and the user's background knowledge ($K$). A summary is informative if it provides information not already present in $K$.

The centerpiece of this framework is the Importance Distribution ($\mathbb{P}_{D/K}$), which identifies semantic units that are frequent in the source but rare in the background knowledge.

Visualization of the document distribution (a), background knowledge (b), and the resulting importance distribution (c).

Using the Kullback Minimum Description Principle, the author derives a scoring function $\theta_I$. Mathematically, Importance emerges as a linear combination: $$\theta_I \approx -Redundancy + \alpha \cdot Relevance + \beta \cdot Informativeness$$

Experimental Setup

The framework was validated on the Text Analysis Conference (TAC) 2008 and 2009 datasets. These benchmarks include:

  • Generic Summarization: Standard multi-document summarization tasks.
  • Update Summarization: Tasks where a system must summarize new documents given that a user has already read an initial set.

The model's performance was evaluated by comparing its scores against human-generated Pyramid scores, which quantify the amount of important information captured in a summary.

Results

Despite using simple n-grams as semantic units, the theoretical model significantly outperformed established baselines in terms of correlation with human judgments.

Correlation of the proposed importance score $\theta_I$ with human scores compared to baselines.

As shown in the results, the proposed score $\theta_I$ achieved a Kendall’s $\tau$ correlation of .294 for generic tasks and .211 for update tasks, surpassing popular methods like LexRank and ICSI. Furthermore, the model proved more effective at distinguishing human-written summaries from machine-generated ones than standard divergence metrics like Jensen-Shannon (JS).

Discussion and Insights

A key insight of this work is the natural unification of summarization objectives. The mathematical derivation shows that Importance is not an arbitrary collection of features but a single quantity that decomposes into intuitive components.

Additionally, the model handles "update summarization" naturally by treating previously seen documents as the background knowledge $K$. The concept of "Summarizability" is also introduced: a "spiky" importance distribution indicates a clear summary target, whereas a flat distribution suggests the task is inherently more difficult due to many equally valid information pieces.

Conclusion and Future Work

This research moves summarization toward a more principled foundation. While the current experiments use words as semantic units, future work will explore richer semantic representations like distributional embeddings. Another promising avenue is the personalization of summaries by modeling the vast and subjective background knowledge of specific users.

Summary updated

Abstract

Maxime Peyrard. Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics. 2019.
BibTeX
@inproceedings{peyrard2019simple,
  author = {Maxime Peyrard},
  title = {{A Simple Theoretical Model of Importance for Summarization}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2019},
  url = {https://openreview.net/forum?id=p2HYEyvzTPg},
}
Figure