Label Purification Before Model Training

36 concepts30 papers1 proposal

Real-world datasets are inherently messy. When training deep neural networks on data containing corrupted or incorrect labels, a common instinct is to design noise-tolerant architectures or robust loss functions. However, this "learning-centric" approach forces the model to navigate corrupted signals throughout the entire optimization loop, often leading to confirmation bias and error propagation.

A powerful alternative is data-centric label purification: identifying and removing or correcting mislabeled instances before the final model training begins. By treating data cleaning as an independent preprocessing step, researchers can provide a clean, high-quality training signal to standard, unmodified architectures. This paradigm shift simplifies the optimization landscape and frequently yields superior accuracy compared to complex, noise-robust training algorithms.

The Intuition: Breaking the Feedback Loop

Why does filtering data beforehand work so well? In a standard training loop, a model exposed to noisy labels can enter a destructive feedback loop: early in training, it memorizes simple patterns, but eventually, it overfits to the corrupted labels, distorting its decision boundaries.

By pruning or cleansing the dataset upfront, we reduce the search space for the learning algorithm. As Shuo He et al. demonstrate in Candidate Label Set Pruning: A Data-centric Perspective for Deep Partial-label Learning, removing false candidate labels before training prevents models from selecting incorrect labels early on, making downstream disambiguation strategies vastly more effective. This proactive approach ensures that the final model learns from high-quality examples where the underlying features truly align with the assigned class label, as highlighted by Charles A. Kantor et al. in Gradient-Based Localization and Spatial Attention for Confidence Measure in Fine-Grained Recognition using Deep Neural Networks.

How to Detect Noise Without a Clean Model

The central challenge of pre-training purification is a chicken-and-egg problem: how do we identify mislabeled data without already having a perfectly trained model? Researchers have developed several clever strategies to bypass this issue.

Multi-Algorithm and Accuracy-Driven Filters

Early foundational work by Carla E. Brodley et al. in Identifying Mislabeled Training Data established that using multiple learning algorithms as "noise filters" can significantly boost final model accuracy, even when datasets contain up to 30% labeling errors. Crucially, Xindong Wu et al. showed in An Empirical Study of the Noise Impact on Cost-Sensitive Learning that for specialized tasks (like cost-sensitive learning), it is better to use a standard, accuracy-driven classifier to filter noise rather than the specialized model itself. Standard classifiers are less biased by task-specific constraints and better at isolating pure label noise.

Feature Similarity and Neighborhoods

Instead of training complex models, we can look at the geometry of the data. Zhaowei Zhu et al. introduced a training-free approach in Detecting Corrupted Labels Without Training a Model to Predict that detects corrupted labels by analyzing feature similarity among neighboring data points. Because clean samples naturally cluster together in representation space, an isolated sample with a conflicting label stands out immediately.

Tracking Training Dynamics

Another approach is to run a brief, proxy training phase to watch how the model learns. Qi Zhang et al. leverage this in DynClean: Training Dynamics-based Label Cleaning for Distantly-Supervised Named Entity Recognition by analyzing how a model's confidence in a label evolves over time. By injecting "fake labels" to automatically calibrate noise thresholds, they can cleanly separate true annotations from distantly-supervised noise.

[ Raw Noisy Data ] ──> [ Purification Step ] ──> [ Clean Dataset ] ──> [ Standard Model Training ]
                             │
              ┌──────────────┴──────────────┐
              ▼                             ▼
     [Feature Geometry]            [Proxy Dynamics]
   (Clustering & Neighbors)       (Confidence Tracking)

Leveraging Pretrained Knowledge and Oracles

When raw data features are insufficient, external knowledge or specialized models can act as gatekeepers to sanitize datasets prior to downstream training.

In text and code domains, pretrained models are highly effective filters. For instance, Hiroki Asano et al. utilize a pretrained error detection model in The AIP-Tohoku System at the BEA-2019 Shared Task to filter out noisy sentence pairs in grammatical error correction datasets, ensuring the generative model only learns from truly corrected targets. Similarly, Qi Cao et al. use pretrained knowledge in PKAD: Pretrained Knowledge is All You Need to Detect and Mitigate Textual Backdoor Attacks to detect and exclude samples with suspicious text-label relationships, neutralizing backdoor attacks before training even begins.

Alternatively, data-influence pipelines can be constructed to score data quality. Anh T. V. Dau et al. in Towards Using Data-Influence Methods to Detect Noisy Samples in Source Code Corpora propose a pipeline that computes influence scores via a validation set to systematically identify and drop low-scoring, noisy samples from source code corpora.

The Power of Decoupling

The primary advantage of proactive label purification is that it treats noise reduction as a standalone preprocessing step, as argued by Jiayi Ma et al. in Hyperspectral Image Classification in the Presence of Noisy Labels. Because the cleansing process is entirely independent of the final classification architecture, it avoids compounding architectural biases. Standard, off-the-shelf models trained on a purified dataset consistently outperform highly specialized, noise-tolerant architectures trained on raw, corrupted data.

Remaining Bottlenecks

Despite its success, automated data cleaning is not infallible. Rigid heuristic rules, often used to clean specialized datasets like code-comment pairs, can suffer from low precision or require heavy domain engineering, a challenge noted by Anh Dau et al. in DocChecker: Bootstrapping Code Large Language Model for Detecting and Resolving Code-Comment Inconsistencies and Lin Shi et al. in Are we building on the rock? on the importance of data preprocessing for code summarization.

Furthermore, automated filtering mechanisms can introduce their own classification errors—potentially discarding valid, complex examples or retaining sophisticated noise, as observed by Saad Obaid ul Islam et al. in Tackling Hallucinations in Neural Chart Summarization. Balancing aggressive noise removal with the preservation of rare, legitimate data points remains an active frontier in data-centric AI.

Key Foundational Work

Identifying Mislabeled Training Data (1999) — contributed 2 concept elements to this direction.
Identifying and Eliminating Mislabeled Training Instances (1996) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_automatic_data_cleaning_92f4edff,
  title = {automatic data cleaning},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/automatic-data-cleaning/rdr_92f4edff46f649578d59b8c1876e5247},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-21},
}

This exact version (version 1, frozen text):

@misc{lacuna_automatic_data_cleaning_92f4edff_v1,
  title = {automatic data cleaning},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/automatic-data-cleaning/rdr_92f4edff46f649578d59b8c1876e5247/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-21},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake