Deep learning models are notoriously sensitive to "noisy" labels—data points where the assigned category is incorrect or misleading. While the internet provides a nearly infinite supply of training data, and Federated Learning (FL) allows models to learn from decentralized devices, both sources are plagued by these inaccuracies.
The core challenge is that standard training algorithms cannot easily distinguish between a "hard" example (a legitimate but difficult image) and a "noisy" one (a mislabeled image). Recent research across both centralized web-supervised learning and decentralized federated learning has converged on a shared solution: leveraging a global perspective to identify and rectify local errors.
The Failure of Local Filtering
In traditional training, we process data in small mini-batches. However, noise is rarely distributed evenly. If a specific mini-batch happens to contain a high concentration of mislabeled images, the model’s gradients will fluctuate wildly, leading to training instability.
Chuanyi Zhang, Yazhou Yao, and colleagues argue in Web-Supervised Network with Softly Update-Drop Training for Fine-Grained Visual Classification that filtering noise within these small batches is fundamentally flawed. They propose that noise should be evaluated against the distribution of the entire dataset. By tracking how much a network's predictions for an image fluctuate over time—a metric they call "probability cross-entropy"—they can identify irrelevant samples that the model fails to learn consistently.
This global approach is echoed in the GRIP framework, where researchers use a dynamic threshold calculated from the entire dataset's loss distribution. This ensures that a clean sample isn't accidentally discarded just because it appeared in a "clean" batch, and a noisy sample isn't kept just because it was the least-noisy part of a "dirty" batch.
The Global Model as a Robust Teacher
In Federated Learning, the problem is amplified. Data stays on local devices (clients), and the central server only sees model updates, not the raw data. If a client has "toxic" labels, its updates can corrupt the shared global model.
The breakthrough intuition, explored by Yuxin Tian and Tongliang Liu in their work on FedGR, is that global models memorize noise much slower than local ones. Because the global model aggregates updates from many different sources, the random errors of one client are often canceled out by the correct labels of others.
This makes the global model a highly effective "reviser." Instead of trusting local labels blindly, the system uses the global model to:
- Identify Noisy Samples: If the global model's loss on a local sample is exceptionally high, that sample is likely mislabeled.
- Generate Pseudo-labels: The global model provides a "consensus" prediction that can replace the suspect local label.
This mechanism, described in Learning Locally, Revising Globally, allows clients to clean their own datasets using the collective intelligence of the federation.
From Discarding to Distillation
Early methods for handling noise simply "dropped" suspicious samples. However, Jingyi Xu and Tony Quek demonstrate in FedCorr that correcting and retaining these samples yields superior performance. Even a mislabeled image contains valuable structural information (e.g., the features of a bird are still there, even if it's labeled "airplane").
When noise levels are extreme (e.g., >90%), even simple correction might fail. In these cases, Yang Lu suggests a strategy called Negative Distillation. Instead of trying to fix the labels, the global model identifies "toxic" clients and treats them as "bad teachers." The global model is updated to intentionally diverge from the incorrect predictions of these clients, effectively learning what not to do.
Synchronizing Local and Global Knowledge
The most robust modern systems, such as FedNoRo and Federated Label-mixture Regularization (FLR), use knowledge distillation to bridge the gap.
In FLR, Taehyeon Kim and Se-Young Yun propose blending local and global predictions into a single robust target. This prevents the local model from over-fitting to its own noisy data while still allowing it to learn unique local features. By using the global model's "cleaner" early-stage knowledge to regularize client updates, the system creates a feedback loop where even highly corrupted data sources can eventually be rehabilitated.
By shifting the focus from local batch-level filtering to global distribution-level guidance, these researchers have turned "dirty" data from a liability into a scalable resource for training high-performance vision models.