Modeling Information Salience for Text Summarization and Evaluation

This direction is merged into another direction: content selection bottleneck. Its last version stays readable and citable here.

Version 1 of 116 papers22 conceptsfrozen 2026-10-04

In the early days of natural language processing, Stephen L. Taylor and colleagues at Northwestern University identified a fundamental hurdle that still defines the field today: the "mathematical criteria" problem. In their 1977 paper, they noted that a system cannot summarize effectively if it cannot distinguish a document's primary theme from its secondary details.

Decades later, even as Large Language Models (LLMs) have mastered fluency, this "salience" problem remains. If a model treats every entity, number, and fact as equally important, the resulting summary becomes a cluttered list of incidental details rather than a coherent synthesis. Modern research has shifted toward explicitly modeling information salience—the "weight" or "centrality" of specific content—to guide both how summaries are generated and how they are judged.

The Problem of Flat Information

Most standard NLP models view text as a sequence of tokens with relatively uniform importance. However, human readers apply a heavy filter. We recognize that in a news story about a corporate merger, the names of the companies and the price are "load-bearing" facts, while the weather on the day of the announcement is peripheral.

When models lack this filter, they fall into the "equality trap." As Dhruv Mehra and Daniel Preotiuc-Pietro demonstrate in their work on ENTSUMv2, abstractive models often struggle to distinguish between an entity's central role and its minor, incidental mentions. Without a mechanism to weight these roles, models produce summaries that are fluent but fail to capture the "point" of the source text.

Building a Cheat Sheet for Importance

To solve this, researchers are moving away from treating text as a flat sequence and toward structured representations that highlight importance. One approach is the use of Document-Level Semantic Graphs.

In the DSGSUM framework, Haoyuan Li and Kun Lu provide the model with a graph of key entities and their relationships. By integrating this graph with a BERT encoder, the model gains a "cheat sheet" of which subjects are central to the narrative. This allows the generator to maintain the "thread" of a complex document, ensuring that the most connected (and thus salient) entities are prioritized in the output.

Similarly, in the domain of data-to-text, Heng Gong and Wei Bi introduced DUV (Data Understanding and Verification). This system uses a ranking module to "understand" the relative magnitude of numbers, rewarding the model for including high-importance data points while penalizing the inclusion of trivial records.

Weighting the Load-Bearing Facts

The challenge of salience is perhaps most acute in evaluation. Traditional metrics like ROUGE simply count overlapping words. If a summary misses a crucial name but includes five minor adjectives, ROUGE might still give it a high score.

To fix this, researchers are developing "reference-free" metrics that weight facts by their importance. Thomas Scialom and his team developed QuestEval, which evaluates a summary by asking questions about the source document. Crucially, QuestEval uses a question weighter to identify which questions target the most salient information. If a summary can answer a question about a core fact, it receives a much higher score than if it answers a question about a peripheral detail.

This shift toward importance-weighting is also seen in the work of Xinnuo Xu and Ondřej Dušek. Their Corr-F metric assigns weights to "information units" based on how often they appear in human-authored references. By prioritizing these "load-bearing" facts, the metric aligns much more closely with human judgment than simple word-matching ever could.

Why Salience Works

The intuition behind these diverse approaches—from 1970s logical graphs to modern neural question-weighters—is that summarization is a lossy process by design.

As Alexander Patrick Mathews argues in the SupMMD framework, purely statistical representativeness (how well a sentence covers the words in a document) is insufficient. High-quality summarization requires a "supervised importance function" that mimics human selection criteria—such as the tendency to prioritize lead sentences in news or specific numerical shifts in financial reports (Chung-Chi Chen et al., NumEval).

By explicitly modeling salience, we move from models that merely shorten text to models that understand what makes text worth reading.

Papers in this version

History & citationVersion 1 · 2026-10-04

Merged into another direction · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_modeling_information_salience_for_text_s_3f7e7ce9,
  title = {Modeling Information Salience for Text Summarization and Evaluation},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/modeling-information-salience-for-text-summarization-and-evaluation/rdr_3f7e7ce97e80493a9712c14361ff1922},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_modeling_information_salience_for_text_s_3f7e7ce9_v1,
  title = {Modeling Information Salience for Text Summarization and Evaluation},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/modeling-information-salience-for-text-summarization-and-evaluation/rdr_3f7e7ce97e80493a9712c14361ff1922/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}