The Warm-up Phase in Learning with Noisy Labels

This direction is split into other directions: warm-up phase dependency. Its last version stays readable and citable here.

Version 1 of 168 papers70 conceptsfrozen 2026-10-04

Deep neural networks are remarkably "good students"—perhaps too good. When trained on a dataset where some images are incorrectly labeled (e.g., a truck labeled as a "car"), a standard network will eventually memorize these mistakes, treating errors as absolute truths. This phenomenon, often called the memorization effect, is a central challenge in robust machine learning.

To combat this, researchers have converged on a shared strategy: the warm-up phase. This initial period of training is not just a preliminary step; it is the critical window where a model establishes the "discriminative power" necessary to tell the difference between a legitimate pattern and a labeling error.

The Intuition of the Memorization Effect

The core insight behind most robust learning frameworks is that neural networks learn simple, generalized patterns before they learn complex, idiosyncratic ones. In a noisy dataset, the "clean" samples usually represent the simple patterns (because they are consistent across the data), while the "noisy" samples represent the idiosyncratic ones.

As Hansang Lee and colleagues observe in Noisy Label Classification Using Label Noise Selection, a model will initially achieve high accuracy on clean data while its loss on noisy data remains high. This creates a "small-loss" signature: clean samples result in lower training losses than noisy ones during the early stages of training.

The Small-Loss Hypothesis and Sample Selection

Most modern noise-robust methods, such as RankMatch by Lechao Cheng, exploit this signature. By training the model for a few "warm-up" epochs using standard cross-entropy loss, researchers can identify a subset of "confident" samples.

Once this baseline is established, the framework can pivot:

  1. Filtering: Discarding high-loss samples entirely to prevent the model from memorizing them.
  2. Re-weighting: Down-weighting suspicious samples, as seen in the Fast Sample Re-weighting (FSR) method developed by Tomas Pfister.
  3. Correction: Attempting to "fix" the label. For instance, Hwanjun Song introduced SELFIE, which "refurbishes" samples that show consistent predictive patterns over time rather than just throwing them away.

The Goldilocks Problem of Warm-up Duration

The success of these methods hinges on a delicate timing problem. If the warm-up period is too short, the model hasn't learned enough to distinguish noise from hard-but-clean samples. If it is too long, the model begins to overfit and memorize the noise, causing the "small-loss" gap to disappear.

Nannan Wu and the team behind FedA3I highlight that in complex scenarios like medical imaging, an incorrectly specified warm-up can lead to "inaccurate noise estimation." Similarly, Xinglong Wu notes in the Class Prototype-based Cleaner that excessive training causes the model to internalize the very noise it was intended to filter.

When Warm-up Fails: The Initialization Bottleneck

The reliance on an initial "clean" signal creates a significant vulnerability: extreme noise. If the ratio of incorrect labels is high enough (e.g., 60-80%), the model may never achieve better-than-random accuracy during the warm-up phase.

In Pseudo-Loss Selection (PLS), Paul Albert and co-authors warn that extreme noise leads to "confirmation bias." If the initial model is poorly calibrated, it will generate incorrect pseudo-labels, which the model then "learns" as truth, creating a feedback loop of error propagation. This is why Shuo Yang emphasizes that methods estimating a transition matrix (the probability of a label flipping from "Cat" to "Dog") are highly sensitive to the quality of the distilled subset of examples identified during those first few epochs.

Beyond Simple Labels: Structural and Semantic Warm-up

The concept of the warm-up phase has expanded beyond simple image classification. In Federated Graph Learning, De Li uses a warm-up to establish a global model that can resist "poisoning" from noisy clients. In self-supervised contrastive learning, Vicente Balmaseda shows that a baseline of "feature coherence" must be established before the model can reliably identify "false negatives" (semantically similar images incorrectly treated as opposites).

Ultimately, the research direction suggests that robustness is a temporal property. A model is not robust by default; it must be carefully guided through an initial phase of "skeptical" learning to build the internal representations required to eventually correct its own training data.

Papers in this version

Showing 60 of 68 papers.

History & citationVersion 1 · 2026-10-04

Split into other directions · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_the_warm_up_phase_in_learning_with_noisy_38d5184c,
  title = {The Warm-up Phase in Learning with Noisy Labels},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/the-warm-up-phase-in-learning-with-noisy-labels/rdr_38d5184c20c24c75887f7cb6268a1e88},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-21},
}

This exact version (version 1, frozen text):

@misc{lacuna_the_warm_up_phase_in_learning_with_noisy_38d5184c_v1,
  title = {The Warm-up Phase in Learning with Noisy Labels},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/the-warm-up-phase-in-learning-with-noisy-labels/rdr_38d5184c20c24c75887f7cb6268a1e88/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-21},
}