Improving Model Reliability through Data Denoising and Selection

31 concepts24 papers1 proposal

In the era of large-scale machine learning, the prevailing wisdom often suggests that more data leads to better performance. However, a growing body of research demonstrates that the quality of training data is just as critical as its quantity. When datasets contain "noise"—such as mislabeled examples, irrelevant information, or systemic biases—models do not just ignore these errors; they often learn from them, leading to unreliable predictions and poor generalization.

The Fragility of Predictions

The impact of poor data quality is not merely a marginal reduction in accuracy. As Meyer et al. demonstrate through the concept of "dataset multiplicity," even minor inaccuracies in training labels can ripple through a model, making a significant portion of its test-time predictions unreliable. This fragility suggests that models are highly sensitive to the specific composition of their training sets.

This sensitivity is particularly evident in complex tasks like abstractive summarization. Jiang et al. and Wang et al. argue that noisy datasets—such as those containing colloquialisms or irrelevant formatting—hinder a model's ability to generate factually consistent and fluent summaries. In fact, Liu notes that a noise rate as low as 15% can significantly confuse a model, underscoring the necessity of rigorous data cleaning.

Differentiable Rewards for Data Selection

If data quality is paramount, how can we efficiently identify which examples are beneficial? Traditional methods often rely on manual curation or expensive reinforcement learning. To address this, Wang et al. proposed Differentiable Data Selection (DDS).

The intuition behind DDS is to treat data selection as an optimization problem. Instead of treating every training example equally, the framework learns to weight examples based on their similarity to a small, high-quality validation set. By comparing the gradients of the training data with the gradients of the validation data, the model receives a real-time reward signal: if a training example's gradient points in the same direction as the validation gradient, it is deemed beneficial and given a higher weight. This allows the model to automatically filter out noise and focus on the most relevant features for the target task.

Denoising in Specialized Domains

The "garbage in, garbage out" principle is especially acute in specialized fields like software engineering and medical imaging. In software engineering, Tufano et al. emphasize that the effectiveness of models for tasks like test generation depends entirely on the relevance of the training pairs. Shi et al. found that widely-used code summarization benchmarks are often riddled with noise, and by introducing automated cleaning tools like CAT, they were able to substantially improve model accuracy.

Similarly, in computer vision, Gröger et al. highlight that while self-supervised learning has improved near-duplicate detection, identifying label errors in complex medical data remains a significant hurdle. In the context of facial analysis, Lin et al. demonstrated that creating a cleaned version of standard datasets (such as IMDB-Clean) is essential for achieving robust performance in "in-the-wild" scenarios where data distributions vary wildly.

Shifting Focus from Quantity to Integrity

The collective evidence from these studies suggests a shift in the research paradigm. As Arcelli Fontana et al. suggest, the reliability of predictive models is constrained by "data smells"—structural issues in the data that undermine performance regardless of dataset size.

Whether through manual denoising by experts and citizen scientists (Soni et al.), the development of high-quality synthetic datasets (Jiang et al.), or the use of data-influence methods to prune corpora (Dau et al.), the goal remains the same: ensuring that the patterns a model learns are representative of the underlying task rather than the artifacts of a noisy collection process. As Yang et al. conclude, future progress in predictive modeling will likely depend less on building larger models and more on developing robust techniques for handling the inherent imbalances and noise in real-world data.

Key Foundational Work

Correcting Category Errors in Text Classification (2004) — contributed 1 concept elements to this direction.
Combining Human and Machine Transcriptions on the Zooniverse Platform (2018) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_data_quality_impact_c855b6e4,
  title = {data quality impact},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/data-quality-impact/rdr_c855b6e4e87548628dc46c5bd48f14c1},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-22},
}

This exact version (version 1, frozen text):

@misc{lacuna_data_quality_impact_c855b6e4_v1,
  title = {data quality impact},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/data-quality-impact/rdr_c855b6e4e87548628dc46c5bd48f14c1/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-22},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake