Quantifying the Effects of Text Duplication on Semantic Models

Alexandra Schofield, Laure Thompson, David M. MimnoEMNLP 2017 · DOI

Abstract

Text duplication is a ubiquitous form of noise in natural language datasets, yet its quantitative impact on unsupervised semantic models is often overlooked. This paper investigates how Latent Semantic Analysis (LSA) and Latent Dirichlet Allocation (LDA) respond to varying levels of document and template duplication. Our findings reveal that these models are surprisingly robust; they tend to "sequester" repeated text into a small number of dedicated topics or components. This mechanism protects the broader semantic structure, suggesting that aggressive deduplication may not always be necessary if model parameters, such as the number of topics, are appropriately tuned.

Introduction

In real-world Natural Language Processing (NLP), data is rarely pristine. From social media "retweets" and news wire boilerplate to multiple editions of digital texts, duplication is a constant presence. While researchers typically employ heuristic deduplication methods, these processes can be computationally expensive and sometimes imprecise. This study seeks to quantify the actual cost of duplication: do these "noisy" repetitions fundamentally break topic models and word embeddings, or do the underlying algorithms manage to filter the noise?

Methodology

To isolate the effects of duplication, the researchers utilized two clean, manually deduplicated news corpora: the New York Times Annotated Corpus (NYT) and the Spanish Language News Text Corpus (REUSL). They then systematically introduced two types of artificial duplication:

  1. Exact Document Duplication: A percentage ($p\%$) of the corpus was selected, and each selected document was copied $c$ times.
  2. Template String Duplication: Fixed strings (e.g., "Lorem Ipsum" or random snippets) were prepended to documents to simulate headers or boilerplate.

The study evaluated the performance of Latent Semantic Analysis (LSA) and Latent Dirichlet Allocation (LDA) using metrics for loss (model fit), concentration (entropy of document-topic distributions), and expressivity (the degree to which topics are dominated by repeated text).

The "Trash Topic" Defense and Model Robustness

A primary discovery of this research is the "Trash Topic" defense mechanism inherent in LDA. When a specific document is repeated many times, the model does not allow this noise to permeate all semantic dimensions. Instead, it isolates the repeated text into one or two dedicated topics.

As long as the total number of topics ($K$) is sufficiently high, these "duplicate topics" act as sinks for the noise, leaving the remaining topics to capture the genuine semantic patterns of the corpus. This robustness is illustrated in the relationship between topic count and model performance under duplication stress.

Effect of topic count (K) on model performance under increasing text duplication.

As shown in the figure above, increasing the number of topics ($K$) significantly improves the model's ability to handle heavy duplication. For higher values of $K$, the model's loss remains stable even as the number of copies increases exponentially, whereas models with fewer topics see performance degrade rapidly.

Semantic Integrity of Singular Documents

The study also distinguished between the model's treatment of duplicated documents and "singular" (non-repeated) documents. While models may "memorize" duplicated content, the semantic representation of singular documents remains remarkably stable.

Semantic stability of singular (non-duplicated) documents as corpus duplication increases.

Even when a significant proportion of the corpus consists of duplicates, the semantic integrity of the non-repeated documents—measured by their similarity to a gold-standard representation—shows minimal degradation. In some cases, the "sequestering" of noise into specific components even allows the model to maintain or slightly refine the representation of the remaining documents.

Key Insights

  • Diversity vs. Repetition: LSA is more sensitive to many different documents being repeated a few times than to a single document being repeated many times. The former requires the model to sacrifice more dimensions to capture the noise.
  • Boilerplate Context: Template text is easily separated if its vocabulary differs significantly from the main content. However, semantically similar boilerplate (e.g., a news header in a news corpus) is more disruptive.
  • Saturation Thresholds: Models eventually reach a saturation point where duplication causes a sharp drop in entropy, indicating the model has shifted from learning general patterns to memorizing noise.

Conclusion and Practical Recommendations

The research suggests that the necessity of deduplication depends heavily on the model being used. For LDA users, duplication can often be mitigated by simply increasing the topic count ($K$), allowing the model to naturally sequester noise. For LSA, however, more aggressive deduplication of unique documents is recommended, as the model's linear nature makes it more susceptible to diverse noise. Future work should extend these findings to neural models like Transformers, which may process local context and duplication differently than bag-of-words models.

Summary updated

BibTeX
@inproceedings{schofield2017quantifying,
  author = {Alexandra Schofield and Laure Thompson and David M. Mimno},
  title = {{Quantifying the Effects of Text Duplication on Semantic Models}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2017},
  doi = {10.18653/v1/d17-1290},
}
Figure