Abstract
Federated Learning (FL) often encounters noisy labels due to the decentralized nature of data collection. Existing label-noise learning techniques struggle in FL because individual clients typically have very few samples, leading to unstable estimation of the Noise Transition Matrix (NTM). We present FedLNL, a framework that stabilizes NTM estimation using Bayesian inference and a novel Local Diversity Product (LDP) regularizer. FedLNL allows for effective local optimization that contributes to global model accuracy. Our experiments show that FedLNL significantly outperforms state-of-the-art methods, particularly in data-scarce regimes, improving accuracy by up to 26%.
Introduction: The "Small Data, Big Noise" Problem
In Federated Learning, we often assume that while data is decentralized, labels provided by users are accurate. In reality, user-generated labels are frequently "noisy" due to mistakes or varying expertise. While centralized learning has established methods for handling label noise—often by estimating a Noise Transition Matrix (NTM) to "undo" the corruption—these methods often fall apart in federated settings.
The primary hurdle is data scarcity. In FL, a single client may only have a few hundred training samples. When clients attempt to estimate their local NTM using standard gradient descent (SGD), the resulting gradients are unstable, leading to massive estimation errors and model "memorization" of noise.
Methodology: The FedLNL Framework
FedLNL introduces two primary innovations to stabilize learning in low-data federated environments:
1. Alternating Bayesian Updates
Instead of treating the NTM as a set of weights learned via SGD, FedLNL employs an alternating Bayesian update strategy. The NTM is modeled using Dirichlet distributions. Clients use their current classifier to estimate "clean" labels, construct a confusion matrix, and update the NTM's prior distribution. This approach is significantly more stable than backpropagation when data is limited because the updates are naturally bounded and less sensitive to individual outliers.
2. Local Diversity Product (LDP) Regularizer
To ensure the NTM doesn't converge to a trivial solution (like an identity matrix), researchers typically use a "minimal volume" constraint. In centralized settings, this requires a global view of the dataset. FedLNL introduces the Local Diversity Product (LDP), a mathematically decomposable version of this constraint. This allows clients to perform local optimization that remains globally consistent, enforcing the minimal volume principle without needing to aggregate raw data across the network.
Experimental Results
The researchers evaluated FedLNL against state-of-the-art (SOTA) FL schemes such as RoFL, FedCorr, and FedLSR using CIFAR-10, CIFAR-100, and the real-world Clothing1M dataset.
Key Findings
- Superior Accuracy: On CIFAR-100 with 40% noise, FedLNL achieved 68.14% accuracy, while the next best FL-specific method, FedCorr, only managed 48.69%.
- Robustness to Scarcity: As shown in Table 1, FedLNL maintains near-peak performance even when clients have as few as 500 samples, a regime where other methods fail.
- Real-World Performance: On the Clothing1M dataset, which contains natural label noise from online shopping metadata, FedLNL outperformed all other federated methods in both i.i.d. and non-i.i.d. settings.
Conclusion
FedLNL addresses a critical gap in Federated Learning by enabling robust model training on noisy, decentralized data. By replacing unstable gradient-based updates with Bayesian inference and introducing the LDP regularizer, the framework effectively solves the "Small Data, Big Noise" problem. This work represents a significant step toward deploying reliable AI in privacy-sensitive environments where data quality cannot be guaranteed.
This article is a summary of "Federated Label-Noise Learning with Local Diversity Product Regularization" (AAAI 2024).
Summary updated