In single-document text processing, redundancy is typically treated as a flaw. If an author repeats the same phrase or fact multiple times within an article, it is considered poor style or noise that a summarizer must filter out. However, when we shift our perspective from an isolated document to a cluster of independent documents covering the same topic, the nature of redundancy fundamentally changes. It transforms from a textual defect into a powerful statistical signal of importance, verification, and consensus.
Why does this work? Imagine five journalists independently writing about a breaking news event. Each journalist brings their own stylistic choices, peripheral details, and structural preferences. However, because they are reporting on the same core event, the most critical facts—the "who, what, and where"—will inevitably appear across all five articles. The independent repetition of information across distinct sources serves as an implicit voting mechanism. By tracking what information is repeated across a cluster, algorithms can mathematically separate core, authoritative facts from the idiosyncratic noise of individual writers.
The Power of Cross-Document Consensus
Early work in multi-document summarization recognized this statistical reality. While single-document summarizers often struggle to beat simple baselines (such as selecting the first few sentences of an article), Nenkova demonstrated in Automatic Text Summarization of Newswire: Lessons Learned from the Document Understanding Conference that multi-document systems can consistently outperform these baselines by exploiting cross-document redundancy. This redundancy provides a clear signal of importance that is entirely absent when analyzing an isolated text.
Systems like Columbia University's Newsblaster, detailed by McKeown et al. in Text Summarization: News and Beyond, operationalized this by using cross-document repetition as the primary driver for content selection. Similarly, Barzilay et al. in Information Fusion in the Context of Multi-Document Summarization used this recurrence to identify the core facts that should be "fused" together into a non-redundant, authoritative summary. By treating cross-document overlap as a proxy for salience, these systems effectively let the collective consensus of multiple authors dictate what information matters most.
Differentiating Inter-Document and Intra-Document Links
To mathematically capture this intuition, researchers frequently model document collections as graphs, where sentences or text fragments are nodes, and edges represent semantic similarity. However, a naive graph approach treats all similarity edges equally, which introduces a major flaw: a single, highly repetitive author could artificially inflate the importance of their own idiosyncratic details.
To solve this, a series of papers introduced a critical structural distinction: separating intra-document links (connections within the same document) from inter-document links (connections across different documents).
Naive Graph Approach: Differentiated Graph Approach:
[Doc 1: Sent A] <=======> [Doc 1: Sent B] [Doc 1: Sent A] --------> [Doc 1: Sent B] (Low Weight)
^ ^ ^
| | | (High Weight Inter-Doc Link)
v v v
[Doc 2: Sent C] <=======> [Doc 3: Sent D] [Doc 2: Sent C] <=======> [Doc 3: Sent D] (High Weight)
In Improved Affinity Graph Based Multi-Document Summarization, Wan et al. demonstrated that heavily weighting inter-document links over intra-document links drastically improves the accuracy of sentence importance rankings. Later, Wan took this to its logical conclusion in Using only cross-document relationships for both generic and topic-focused multi-document summarizations, showing that ignoring internal document structure entirely and relying exclusively on cross-document relationships yields optimal performance.
The intuition here is clear: if Sentence A in Document 1 is highly similar to Sentence B in Document 1, it tells us very little about global importance. But if Sentence A in Document 1 is highly similar to Sentence C in Document 2 and Sentence D in Document 3, it represents a cross-author consensus. Prioritizing these cross-document links acts as an automated validation mechanism, filtering out local redundancies while elevating globally verified facts—a principle also leveraged in manifold-ranking frameworks by Wan et al. in Manifold-Ranking Based Topic-Focused Multi-Document Summarization.
Scaling the Granularity of Redundancy
The mechanism of cross-document signaling operates across various levels of textual granularity, from simple surface tokens to deep semantic units:
- Words and N-grams: At the simplest level, word frequency matches complex semantic models because essential information is naturally repeated (You et al.). Peyrard et al. showed in Optimizing an Approximation of ROUGE - a Problem-Reduction Approach to Extractive Multi-Document Summarization that in the absence of ground-truth summaries, high-frequency n-grams across a collection serve as excellent proxies for both salience and redundancy estimation.
- Entities and Arguments: In open-domain event extraction, Liu et al. in Open Domain Event Extraction Using Neural Latent Variable Models calculated a "redundancy ratio" of entity mentions across news clusters to separate core event participants (like primary agents or victims) from peripheral actors.
- Propositions: Moving beyond sentences, Ernst et al. introduced ProCluster in Proposition-Level Clustering for Multi-Document Summarization. By breaking text down into atomic propositions and clustering them, the authors showed that high-density clusters of identical propositions provide an explicit, explainable signal of what facts represent the core consensus.
Applications in Verification and Context Expansion
Beyond standard multi-document summarization, this consensus-driven framework solves unique challenges in high-velocity, search, and specialized domains:
Real-Time Verification
In fast-moving news streams, redundancy acts as a trust signal. Rücklé et al. demonstrated in Real-Time News Summarization with Adaptation to Media Attention that during periods of intense media attention, requiring information to appear across multiple independent sources before publishing an update prevents the spread of unconfirmed or low-confidence rumors. This concept is also used in visual analytics, where Krstajic et al. in Visual analysis of news streams with article threads mapped the density of redundant incoming reports to visual attributes to highlight high-impact global events.
Document Expansion
The power of cross-document signaling can even be backported to improve single-document summarization. In Single Document Summarization with Document Expansion, Wan et al. retrieved topically related "neighbor" documents for a single target text. By running a graph-ranking algorithm across this expanded set, the system used the cross-document consensus of the neighbors to identify which sentences in the target document were truly the most important, bypassing the limitations of internal document structure.
Search and Community Consensus
In information retrieval, de Rijke et al. in The University of Amsterdam at WebCLEF 2007: Using Centrality to Rank Web Snippets applied similarity-based centrality to rank web snippets, operating on the principle that consensus across diverse search results indicates high relevance.
A similar phenomenon occurs in scientific literature. While a paper's abstract only provides the author's self-assessment, Mohammad et al. showed in Using Citations to Generate surveys of Scientific Paradigms that analyzing the redundancy across citation texts—how the rest of the community describes a paper—reveals the true, consensus-backed contributions of a scientific paradigm.
Key Foundational Work
Text Summarization: News and Beyond (2005) — contributed 3 concept elements to this direction.
Information Fusion in the Context of Multi-Document Summarization (1999) — contributed 2 concept elements to this direction.
Automatic Text Summarization of Newswire: Lessons Learned from the Document Understanding Conference (2005) — contributed 2 concept elements to this direction.