Federated Partial Label Learning with Local-Adaptive Augmentation and Regularization

Yan Yan, Yuhong GuoAAAI 2024 · OpenReview

TL;DR

This paper introduces "Federated Partial Label Learning" (FedPLL), a new scenario where data with noisy, incomplete labels is distributed across multiple clients under privacy constraints. They propose FedPLL-LAAR, a novel method that tackles this by adaptively disambiguating labels, augmenting local data, and regularizing client models to align with a global model, achieving strong performance.

What Problem Does This Paper Address?

Traditional supervised learning relies on precise labels for every data point. However, obtaining such high-quality labels is often expensive and time-consuming. Partial Label Learning (PLL) offers a solution by allowing each data point to have a set of candidate labels, where only one is the true label. While effective, existing PLL methods assume all data is centrally located.

The paper addresses a critical gap: applying PLL in a federated learning (FL) setting. FL allows multiple clients to collaboratively train a global model without sharing their raw data, preserving privacy. However, standard FL assumes clean labels. Combining these two challenges, the paper introduces Federated Partial Label Learning (FedPLL), where data is distributed across clients, each with partial, noisy labels, and privacy is paramount. This scenario is more practical for real-world applications where data is inherently distributed and labeling is imperfect. The core problem is how to train a robust global model under these combined constraints of distributed data, noisy partial labels, and privacy.

Methods

The authors propose FedPLL-LAAR, a novel method designed to tackle the FedPLL problem. It integrates three key components within the standard federated learning framework (like FedAvg):

  • Local Model Learning with Disambiguated Pseudo-labels: Instead of directly using the noisy candidate labels, FedPLL-LAAR iteratively refines "pseudo-labels" for each instance. These pseudo-labels are updated using a moving average of the local model's predictions. This process helps to gradually disambiguate the true label from the noisy candidates. The local models are then trained using these refined pseudo-labels as targets via a cross-entropy loss.
  • Local-Global Gradient Alignment Regularization: To combat "client drift" (where local models diverge significantly from the global model due to non-IID data), this component enforces alignment between local and global model gradients. It adds a regularization term that penalizes dissimilarity between the gradient of the local loss and the gradient of the global model. This encourages local updates to stay closer to the global objective.
  • MixUp-based Local-Adaptive Data Augmentation: To address data insufficiency and improve robustness on local clients, FedPLL-LAAR employs MixUp. This technique creates synthetic training samples by linearly interpolating existing data points and their labels. Crucially, it's adaptive: the weights for mixing are determined by the confidence of the global model on the instances. This allows the augmentation to focus on more informative samples and better align with the global model's understanding, while also helping to mitigate label noise.

These components are combined into an overall local training loss function that is minimized by each client.

Datasets

The experiments were conducted on three standard benchmark datasets:

  • CIFAR-10: 10 classes, 6,000 images per class.
  • CIFAR-100: 100 classes, 600 images per class.
  • SVHN (Street View House Numbers): Contains real-world digit images.

To create the FedPLL setting, these datasets were modified:

  1. Partial Labels: A procedure was used to corrupt labels, ensuring each image had a set of candidate labels with a probability p of containing the true label and additional noisy ones.
  2. Non-IID Data: A standard method was employed to distribute data across clients such that each client's data distribution was not identical, simulating real-world scenarios.

The paper does not introduce a new benchmark dataset but rather adapts existing ones to the novel FedPLL scenario.

Key Insights and Ideas

  • The Power of Pseudo-label Refinement: Iteratively updating pseudo-labels using a moving average of local model predictions is a surprisingly effective way to disambiguate noisy partial labels in a federated setting, even without direct access to true labels.
  • Gradient Alignment as a Client Drift Antidote: Forcing local model gradients to align with global model gradients is a crucial technique to prevent local models from diverging too much, especially in non-IID scenarios, thereby improving global model performance.
  • Adaptive MixUp for Targeted Augmentation: Applying MixUp augmentation adaptively, guided by the global model's confidence, is more effective than standard MixUp. It helps focus on informative samples and better aligns local learning with the global objective.
  • Synergy of Components: The paper strongly emphasizes that the effectiveness of FedPLL-LAAR comes from the synergistic interaction of its three core components, rather than any single one being a silver bullet.

Key Results

  • Superior Performance: FedPLL-LAAR consistently outperformed state-of-the-art federated learning methods (FedCorr, RoFL, FedX) and baselines (FedAvg) on CIFAR-10, SVHN, and CIFAR-100 datasets across various noise levels and non-IID settings.
  • Significant Gains: For example, on SVHN with a noise level of 0.5, FedPLL-LAAR outperformed the best alternative by 13.49%. On CIFAR-10, it achieved gains of up to 16.9% over the best method.
  • Robustness to Noise: The performance gains of FedPLL-LAAR increased with higher noise levels, demonstrating its effectiveness in handling noisy partial labels.
  • Efficiency: FedPLL-LAAR required fewer communication rounds to converge compared to other methods, even though its per-round communication might be slightly higher. This suggests better overall efficiency.
  • Scalability: The method showed good performance even with a large number of clients (up to 80), maintaining reasonable prediction accuracy.
  • Ablation Study Validation: Removing any of the core components (label disambiguation, adaptive MixUp, gradient alignment) led to significant performance degradation, validating their importance.

Limitations and Future Work

  • Hyperparameter Tuning: While the paper presents a robust method, the performance of FedPLL-LAAR relies on hyperparameters like η and δ (trade-off for loss terms) and μ (momentum for pseudo-labeling). The paper mentions setting η and δ to 1 by default, but optimal tuning might be required for different scenarios.
  • Communication Overhead: Although more efficient in terms of rounds, the method might have a slightly higher communication cost per round due to gradient exchange.
  • Future Work: The paper implicitly suggests future work could involve exploring more sophisticated adaptive weighting schemes for MixUp, investigating more advanced gradient alignment techniques, and potentially extending FedPLL to other complex learning scenarios. The authors also highlight the practical implications of FedPLL for real-world applications where data is inherently noisy and distributed.

Summary updated

Abstract

Partial label learning (PLL) expands the applicability of supervised machine learning models by enabling effective learning from weakly annotated overcomplete labels. Existing PLL methods however focus on the standard centralized learning scenarios. In this paper, we expand PLL into the distributed computation setting by formalizing a new learning scenario named as federated partial label learning (FedPLL), where the training data with partial labels are distributed across multiple local clients with privacy constraints. To address this challenging problem, we propose a novel Federated PLL method with Local-Adaptive Augmentation and Regularization (FedPLL-LAAR). In addition to alleviating the partial label noise with moving-average label disambiguation, the proposed method performs MixUp-based local-adaptive data augmentation to mitigate the challenge posed by insufficient and imprecisely annotated local data, and dynamically incorporates the guidance of global model to minimize client drift through adaptive gradient alignment regularization between the global and local models. Extensive experiments conducted on multiple datasets under the FedPLL setting demonstrate the effectiveness of the proposed FedPLL-LAAR method for federated partial label learning.
BibTeX
@inproceedings{yan2024federated,
  author = {Yan Yan and Yuhong Guo},
  title = {{Federated Partial Label Learning with Local-Adaptive Augmentation and Regularization}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2024},
  url = {https://openreview.net/forum?id=rvvUIb5xry},
}
Figure