Semantic Novelty Detection and Redundancy Filtering

You are reading version 1 of this direction, frozen 2026-10-04. The latest version is version 2.

Version 1 of 219 papers23 conceptsfrozen 2026-10-04

In information retrieval and summarization, "relevance" has long been the primary metric of success. If a user asks about a news event, a system should return documents about that event. However, relevance alone is insufficient for a good user experience. If a search engine returns ten identical news reports, it is perfectly relevant but entirely redundant.

The research direction explored here shifts the focus from mere relevance to novelty. The goal is to identify information that is not only related to a topic but also provides "new" value relative to what the user has already seen. This is critical for tasks like "update summarization"—where a user wants to know what has changed since they last checked the news—and for maintaining concise knowledge bases like Wikipedia.

The Redundancy Penalty

The most intuitive way to handle redundancy is to treat it as a "tax" on a document's relevance score. In Redundancy Removal to Selectively Diversify Information Retrieval Results, Xiaolin Wang and Hai Zhao introduce RedRem, an iterative algorithm that re-ranks search results.

The intuition is simple: as the system builds a list of results, it compares each new candidate against the documents already selected. If a candidate shares too many unigrams (words) with the "already accepted" set, its score is penalized. This forces the system to skip over near-duplicates and surface documents that cover different facets of a query, a concept known as "intent recall."

Similarly, in the context of Wikipedia, Sisay Fissaha Adafre, Maarten de Rijke, and Valentin Jijkoun developed systems for the WiQA task to find "perfect snippets." They used lexical overlap as a filter: if a candidate sentence for an article shared too much vocabulary with the existing text, it was discarded.

From Lexical Overlap to Semantic Units

While counting shared words is a good start, redundancy is often semantic rather than literal. Two sentences can use entirely different words to convey the same fact. Ruihua Song and Shaoping Ma noted in their TREC 2002 experiments that identifying redundancy is difficult because it requires understanding if a new sentence provides a genuinely new concept or just a synonym-swapped version of an old one.

To solve this, researchers moved toward more structured comparisons:

  1. Basic Elements (BE): Shafiq Joty and Yllias Chali found that comparing "Basic Elements"—minimal semantic units derived from syntactic dependencies—was more effective than n-gram overlap for update summarization.
  2. Wikification: Chunping Li and Yajie Miao proposed WikiSummarizer, which maps sentences to Wikipedia concepts. This allows the system to recognize that two sentences are redundant if they link to the same underlying concepts, even if their wording differs.
  3. Transformer Embeddings: Modern approaches, such as those by Jaromír Savelka and Kevin D. Ashley, use models like RoBERTa to capture nuanced linguistic patterns. By feeding the model both a candidate sentence and the source text, the transformer can learn to identify and down-rank repetitions that are semantically identical but lexically distinct.

Update Summarization and Temporal Novelty

A major application of this research is Update Summarization, where the goal is to summarize a set of documents (Cluster B) under the assumption that the user has already read an earlier set (Cluster A).

Sujian Li and colleagues at Peking University explored how to mathematically model this "history." They found that averaging the similarity of a new sentence against all sentences in the history was more robust than just checking against the single most similar sentence. This "global" penalty ensures that information which is generally "old news" across the entire background corpus is suppressed.

In real-time scenarios, like the TREC Real-Time Summarization track, Qingwei Meng and Zhen Yang applied these thresholds to Twitter streams. To prevent user fatigue, they discarded any incoming tweet that exceeded a vocabulary overlap threshold with tweets already pushed to the user that day.

Entailment as the Ultimate Novelty Filter

The most sophisticated framing of novelty is through Recognizing Textual Entailment (RTE). If sentence A entails sentence B, then B contains no information that isn't already present in A. Therefore, B is redundant.

Luisa Bentivogli, Danilo Giampiccolo, and Ido Dagan organized several RTE challenges that integrated entailment into summarization workflows. They demonstrated that semantic inference is a powerful tool for filtering: if a system can logically determine that a new fact is already supported by the existing corpus, it can safely ignore it.

This logic extends to fairness and coverage. Haoyuan Li and Snigdha Chaturvedi recently proposed using entailment-based coverage to ensure that multi-document summaries represent different viewpoints fairly. By using entailment, they can verify if the unique information from different sources is present in the summary, rather than just counting how many times a redundant fact is repeated.

Summary of the Intuition

To build a system that understands novelty, researchers have moved through three stages:

  1. Lexical: "Does this sentence use the same words I've seen before?"
  2. Structural/Semantic: "Does this sentence refer to the same entities or semantic units?"
  3. Logical: "Is the information in this sentence already logically implied by what I know?"

By applying these filters, systems can move beyond simple retrieval and toward true "authoring" and "updating," providing users with a stream of information that is always relevant and, crucially, always new.

Papers in this version

History & citationVersion 2 · 2026-10-04

Current · Permanent link

  1. Version 2 (latest) · 2026-10-04 · 16 papers
  2. Version 1 · 2026-10-04 · 19 papers
Cite this direction

Latest version (a living page):

@misc{lacuna_semantic_redundancy_detection_03a6c86b,
  title = {semantic redundancy detection},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/semantic-redundancy-detection/rdr_03a6c86b15b24e058be923cbd7c67804},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_semantic_novelty_detection_and_redundanc_03a6c86b_v1,
  title = {Semantic Novelty Detection and Redundancy Filtering},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/semantic-novelty-detection-and-redundancy-filtering/rdr_03a6c86b15b24e058be923cbd7c67804/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}