Federated Label-Noise Learning with Local Diversity Product Regularization

Xiaochen Zhou, Xudong WangAAAI 2024 · DOI

Abstract

Federated Learning (FL) often encounters noisy labels due to the decentralized nature of data collection. Existing label-noise learning techniques struggle in FL because individual clients typically have very few samples, leading to unstable estimation of the Noise Transition Matrix (NTM). We present FedLNL, a framework that stabilizes NTM estimation using Bayesian inference and a novel Local Diversity Product (LDP) regularizer. FedLNL allows for effective local optimization that contributes to global model accuracy. Our experiments show that FedLNL significantly outperforms state-of-the-art methods, particularly in data-scarce regimes, improving accuracy by up to 26%.

Introduction: The "Small Data, Big Noise" Problem

In Federated Learning, we often assume that while data is decentralized, labels provided by users are accurate. In reality, user-generated labels are frequently "noisy" due to mistakes or varying expertise. While centralized learning has established methods for handling label noise—often by estimating a Noise Transition Matrix (NTM) to "undo" the corruption—these methods often fall apart in federated settings.

The primary hurdle is data scarcity. In FL, a single client may only have a few hundred training samples. When clients attempt to estimate their local NTM using standard gradient descent (SGD), the resulting gradients are unstable, leading to massive estimation errors and model "memorization" of noise.

Table 1: Performance comparison across different client data scales. FedLNL (represented as Accurate NTM) maintains high accuracy even as samples per client drop from 5000 to 500, whereas baseline methods show significant degradation.

Methodology: The FedLNL Framework

FedLNL introduces two primary innovations to stabilize learning in low-data federated environments:

1. Alternating Bayesian Updates

Instead of treating the NTM as a set of weights learned via SGD, FedLNL employs an alternating Bayesian update strategy. The NTM is modeled using Dirichlet distributions. Clients use their current classifier to estimate "clean" labels, construct a confusion matrix, and update the NTM's prior distribution. This approach is significantly more stable than backpropagation when data is limited because the updates are naturally bounded and less sensitive to individual outliers.

2. Local Diversity Product (LDP) Regularizer

To ensure the NTM doesn't converge to a trivial solution (like an identity matrix), researchers typically use a "minimal volume" constraint. In centralized settings, this requires a global view of the dataset. FedLNL introduces the Local Diversity Product (LDP), a mathematically decomposable version of this constraint. This allows clients to perform local optimization that remains globally consistent, enforcing the minimal volume principle without needing to aggregate raw data across the network.

Experimental Results

The researchers evaluated FedLNL against state-of-the-art (SOTA) FL schemes such as RoFL, FedCorr, and FedLSR using CIFAR-10, CIFAR-100, and the real-world Clothing1M dataset.

Figure 1: Test accuracy on CIFAR-10 with 40% label noise (flip-04). FedLNL consistently outperforms existing SOTA methods across both 10-client and 100-client configurations.

Key Findings

  • Superior Accuracy: On CIFAR-100 with 40% noise, FedLNL achieved 68.14% accuracy, while the next best FL-specific method, FedCorr, only managed 48.69%.
  • Robustness to Scarcity: As shown in Table 1, FedLNL maintains near-peak performance even when clients have as few as 500 samples, a regime where other methods fail.
  • Real-World Performance: On the Clothing1M dataset, which contains natural label noise from online shopping metadata, FedLNL outperformed all other federated methods in both i.i.d. and non-i.i.d. settings.

Conclusion

FedLNL addresses a critical gap in Federated Learning by enabling robust model training on noisy, decentralized data. By replacing unstable gradient-based updates with Bayesian inference and introducing the LDP regularizer, the framework effectively solves the "Small Data, Big Noise" problem. This work represents a significant step toward deploying reliable AI in privacy-sensitive environments where data quality cannot be guaranteed.

This article is a summary of "Federated Label-Noise Learning with Local Diversity Product Regularization" (AAAI 2024).

Summary updated

BibTeX
@inproceedings{zhou2024federated,
  author = {Xiaochen Zhou and Xudong Wang},
  title = {{Federated Label-Noise Learning with Local Diversity Product Regularization}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2024},
  doi = {10.1609/aaai.v38i15.29659},
}
Figure