Filtering Semantic Redundancy to Ensure Information Novelty

20 concepts16 papers1 proposal

When building information systems—whether search engines, text summarizers, or question-answering tools—the default instinct is to maximize relevance. We want pieces of text that match the user's intent or the source topic as closely as possible. However, optimizing purely for relevance introduces a critical failure mode: redundancy. If five sentences express the exact same core fact using slightly different phrasing, a purely relevance-driven system might return all five, wasting the reader's time and cognitive capacity.

To solve this, modern text processing frameworks treat information selection as a joint optimization problem: maximizing relevance while simultaneously minimizing redundancy. The core intuition is that a piece of information should only be selected if it provides genuine novelty relative to what the system (or the user) has already processed.

The Mechanics of Redundancy Penalization

How do we mathematically suppress repetitive information? The most common approach is the redundancy penalty. Instead of evaluating a candidate text snippet in isolation, the system scores it dynamically based on its similarity to a growing set of already-accepted text.

In information retrieval, this is often implemented as an iterative re-ranking algorithm. For instance, in Redundancy Removal to Selectively Diversify Information Retrieval Results, Xiaolin Wang et al. calculate a redundancy score using a unigram model that measures word overlap between a candidate document and higher-ranked results. By subtracting this redundancy score from the initial relevance score, the system actively suppresses documents that repeat known facts, thereby diversifying the final results.

Similarly, in web snippet retrieval, Maarten de Rijke et al. demonstrate a novelty-weighted filtering mechanism in The University of Amsterdam at WebCLEF 2007: Using Centrality to Rank Web Snippets. Their system penalizes candidate snippets based on their similarity to a set of pre-existing data points, ensuring that the final output prioritizes novel content over redundant facts. This heuristic is also highly effective in real-time streaming contexts, such as filtering the Twitter firehose. As shown by Qingwei Meng et al. in BJUT at TREC 2017: Real-Time Summarization Track, enforcing a strict vocabulary overlap threshold allows a system to discard incoming updates that are too similar to previously delivered content, preventing user fatigue.

This filtering logic can even be applied in reverse to clean up inputs or queries. In The University of Sheffield TREC 2002 Q&A System, Mark A. Greenwood et al. filter out candidate answer strings that exhibit high lexical overlap with the question itself, operating on the intuition that a valid answer must provide new information rather than merely restating the query terms. In e-commerce question generation, Zhiling Zhang et al. use a similar filtering mechanism in Diverse and Specific Clarification Question Generation with Keywords to remove predicted keywords already present in the source text, forcing the generation phase to focus entirely on missing details.

Moving Beyond Surface Words: Semantic and Structural Overlap

While literal word overlap (lexical matching) is fast and computationally cheap, it struggles when redundancy is semantic rather than literal. As noted by Ruihua Song et al. in THU TREC 2002: Novelty Track Experiments, a new sentence might repeat an identical concept using entirely different phrasing or synonyms. To capture this, researchers have moved toward semantic and structural representations.

One approach is to project text into continuous vector spaces or leverage external knowledge bases. In The University of Amsterdam (ILPS.UvA) at TREC 2015 Temporal Summarization Track, Cristina Garbacea et al. utilize vector similarity measures to compare candidate sentences against a history of emitted content, filtering out candidates that exceed a semantic similarity threshold. Taking this a step further, Chunping Li et al. introduce "wikification" in WikiSummarizer - A Wikipedia-based Summarization System. By mapping sentences to Wikipedia concepts and calculating their semantic relatedness via link structures, their system identifies redundancy between sentences that share zero common words but discuss the same underlying ideas.

Alternatively, systems can evaluate redundancy using structural semantic units rather than raw n-grams. In UofL at TAC 2008 Update Summarization and Question Answering, Shafiq Joty et al. demonstrate that measuring the overlap of "Basic Elements" (BE)—which represent structural, dependency-based semantic units—is significantly more effective at minimizing redundancy in update summarization than standard n-gram similarity metrics. By comparing these discrete semantic units against a background corpus of already-read documents, the system prunes repetitive information with high precision.

Redundancy as Logical Subsumption: Textual Entailment

The most rigorous framing of semantic redundancy is textual entailment: a piece of information is redundant if its meaning is already logically implied by a previously processed text.

As articulated in Overview of WiQA 2006 by Maarten de Rijke et al., determining whether a new snippet provides genuine value requires evaluating whether its content is already subsumed by the existing text. In The University of Amsterdam at WiQA 2006, the authors apply this to Wikipedia article authoring, penalizing candidate snippets that exceed semantic overlap thresholds to maintain the conciseness of the knowledge resource.

This connection between semantic inference and redundancy filtering was formalized through the Recognizing Textual Entailment (RTE) challenges. In The Sixth PASCAL Recognizing Textual Entailment Challenge, Luisa Bentivogli et al. demonstrate how textual entailment serves as an effective mechanism for novelty detection in news aggregation and update summarization. By identifying when a new sentence is logically implied by prior text, the system can confidently filter it out. The authors further validate this in The Seventh PASCAL Recognizing Textual Entailment Challenge, showing that semantic inference is a mature and highly effective component for improving the quality of automated summary generation by reliably isolating novel information from redundant noise.

Evaluating Redundancy Inwardly

While redundancy filtering is typically used to select external content, it can also be turned inward to evaluate the quality of generated text without relying on human-written references.

In A Training-free and Reference-free Summarization Evaluation Metric via Centrality-weighted Relevance and Self-referenced Redundancy, Piji Li et al. introduce a self-referenced approach to identify repetitive content within a machine-generated summary. Instead of comparing the summary to an external gold standard, the metric calculates the semantic similarity between every token or sentence and all other components within the same summary. This allows the evaluation framework to penalize internal information recycling automatically.

On the generation side, controlling this behavior during dataset creation is equally vital. In TeSum: Human-Generated Abstractive Summarization Corpus for Telugu, Ashok Urlana et al. employ n-gram novelty filtering via abstractivity scores during corpus construction. By enforcing a specific range of unique words relative to the source text, they prevent models from learning purely extractive behaviors or recycling redundant metadata, ensuring that the resulting models learn to synthesize truly novel, non-redundant summaries.

Key Foundational Work

The University of Amsterdam at WiQA 2006 (2006) — contributed 3 concept elements to this direction.
Overview of WiQA 2006 (2006) — contributed 1 concept elements to this direction.
The University of Sheffield TREC 2002 Q&A System (2002) — contributed 1 concept elements to this direction.
History & citationVersion 2 · 2026-10-04

Current · Permanent link

  1. Version 2 (latest) · 2026-10-04 · 16 papers
  2. Version 1 · 2026-10-04 · 19 papers
Cite this direction

Latest version (a living page):

@misc{lacuna_semantic_redundancy_detection_03a6c86b,
  title = {semantic redundancy detection},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/semantic-redundancy-detection/rdr_03a6c86b15b24e058be923cbd7c67804},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 2, frozen text):

@misc{lacuna_semantic_redundancy_detection_03a6c86b_v2,
  title = {semantic redundancy detection},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/semantic-redundancy-detection/rdr_03a6c86b15b24e058be923cbd7c67804/v/2},
  note = {Version 2, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake