Improved Near-Duplicate Detection for Aggregated and Paywalled News-Feeds

Siddharth Tumre, Sangameshwar Patil, Alok KumarNAACL 2025 · DOI

Abstract

As news outlets increasingly move behind paywalls, news aggregators are often restricted to headlines and short snippets. Traditional near-duplicate detection (NDD) methods like MinHash struggle with these short, paraphrased texts. This paper introduces NDD-MAC (Near-Duplicate Detection Using Metadata Augmented Communities), a system that combines deep-learning embeddings with latent metadata and graph-based community detection. NDD-MAC significantly outperforms traditional hashing methods, providing a robust solution for modern news-feed redundancy.

The Challenge of Paywalled News

In the past, NDD was relatively straightforward because full-text webpages were accessible. Today, the landscape has shifted. News aggregators and enterprise intelligence tools often consume data via APIs that provide only a headline and a short "blurb" because the full article is paywalled.

Traditional techniques, such as Locality Sensitive Hashing (LSH) or MinHash, rely heavily on "surface-level" features—essentially looking for overlapping sequences of words (n-grams). When articles are heavily paraphrased or extremely short, these methods fail. The authors aim to solve this redundancy problem using only restricted snippets.

The NDD-MAC Pipeline

The authors propose a multi-stage pipeline designed to handle the constraints of short-text snippets by moving beyond simple word overlap to semantic and structural analysis.

The NDD-MAC system architecture for detecting near-duplicate news stories.

1. Neural Embedding and Metadata Extraction

Each news snippet is converted into a 768-dimensional vector using Sentence-BERT (specifically the all-mpnet-base-v2 model), capturing semantic meaning rather than just specific words. Simultaneously, an ensemble of classifiers extracts "latent" metadata, such as the news-type (e.g., Merger & Acquisition vs. Product Launch), industry domain, and specific technologies involved.

2. Metadata Augmentation and Community Detection

The system constructs a graph where news articles are nodes. Initial edges are weighted by the cosine similarity of their neural embeddings. These weights are then "boosted" or "penalized" based on the overlap of extracted metadata using a Jaccard index. Finally, instead of a simple similarity threshold, the system applies graph partitioning algorithms like Louvain or Leiden to identify cohesive "communities" of articles. This allows the system to identify duplicates even if some pairs within a group have slightly lower direct similarity.

Experimental Evaluation

The approach was evaluated on two datasets: a large, private NewsAggregator-Vendor Dataset (34,801 records) and a challenging public NDD-NS dataset (1,205 articles) characterized by heavy paraphrasing and low word overlap.

Key Results

NDD-MAC consistently outperformed baseline methods like MinHash and SimHash across both datasets. As shown in the performance analysis on the vendor dataset, NDD-MAC maintains high Micro-F1 scores even at similarity thresholds above 0.8, where traditional methods' performance typically plummets.

Comparison of Micro-F1 scores across different NDD methods and similarity thresholds.

In the NDD-NS dataset, where the average word overlap between duplicates is as low as 5.14%, the semantic approach of NDD-MAC proved far more effective than surface-level hashing. Furthermore, the use of a sliding window (optimally 4 days) allowed the system to scale efficiently while capturing related stories that break within the same news cycle.

Discussion and Future Work

NDD-MAC demonstrates that metadata can act as a powerful tie-breaker, distinguishing between snippets that look similar but describe different events. However, the system's performance is tied to the accuracy of its metadata classifiers. While NDD-MAC is largely unsupervised, making it easier to deploy, future work may explore hybridizing it with supervised BERT-based models to further improve accuracy while managing the costs of labeled data.

Summary updated

BibTeX
@inproceedings{tumre2025improved,
  author = {Siddharth Tumre and Sangameshwar Patil and Alok Kumar},
  title = {{Improved Near-Duplicate Detection for Aggregated and Paywalled News-Feeds}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2025},
  doi = {10.18653/v1/2025.naacl-industry.73},
}
Figure