ADVERSARY-AWARE PARTIAL LABEL LEARNING WITH LABEL DISTILLATION

Cheng Chen, Yueming Lyu, Ivor Tsang

TL;DR

This paper introduces "Adversary-Aware Partial Label Learning" (AAPL), a new framework for training models with noisy, incomplete labels while simultaneously protecting data privacy. By incorporating "rival" labels and a novel "Adversarial Teacher within Momentum" (ATM) algorithm, AAPL aims to improve robustness against label noise and prevent adversaries from inferring sensitive information from leaked data.

What Problem Does This Paper Address?

The core problem this paper tackles is the challenge of training accurate machine learning models when the provided labels are not entirely trustworthy. Specifically, it addresses "partial label learning" (PLL), where each data point is associated with a set of candidate labels, but only one is the true label. This scenario is common when data collection is expensive or when privacy is a concern, leading to noisy or deliberately obscured labels. Conventional PLL methods struggle with a high ratio of noisy labels and, crucially, are vulnerable to data breaches, where adversaries could potentially infer sensitive information from the partial labels. This paper aims to create a more robust and privacy-preserving PLL approach.

Methods

The paper introduces several key innovations:

  • Adversary-Aware Partial Label Learning (AAPL): This is the overarching framework. It augments the standard partial label set with a "rival" label. This rival label acts as a form of encryption, designed to obscure the true label and protect against privacy breaches.
  • Adversarial Teacher within Momentum (ATM): This is a novel disambiguation algorithm. It addresses the intractability and inconsistency issues that arise from the introduction of rival labels. ATM uses a momentum-based approach to learn a more consistent classifier and risk function by leveraging instance embeddings and a self-attention prototype mechanism.
  • Label Distillation: The method incorporates a form of label distillation, where the "teacher" (ATM) guides the learning of the "student" (the main classifier) by providing refined pseudo-labels.
  • Contrastive Learning Integration: The framework leverages contrastive learning to improve representation learning, using a carefully constructed positive sample set that is made more accurate by the ATM disambiguation.

Datasets

The proposed method was evaluated on the following standard image classification datasets:

  • CIFAR-10
  • CIFAR-100
  • CUB-200

The paper also discusses the generation of "adversary-aware partial label datasets" for these benchmarks, which differ from standard partial label datasets by including the introduced "rival" labels.

Key Insights and Ideas

  • Rival Labels for Privacy: The introduction of "rival" labels is a clever way to simultaneously introduce noise (making it harder for adversaries) and structure (allowing the model to learn). It's a dual-purpose mechanism for privacy and robustness.
  • Bridging the Gap with ATM: The core challenge in AAPL is the intractability of the transition matrix introduced by rival labels. The ATM algorithm is a significant contribution, as it provides a practical way to overcome this by using momentum and self-attention to approximate the necessary information.
  • Contrastive Learning Synergy: The paper highlights how improved label disambiguation (via ATM) leads to better positive sample sets for contrastive learning, creating a virtuous cycle where better labels improve representations, which in turn improve label disambiguation.
  • Provable Consistency: A key theoretical contribution is the proof that the proposed method leads to a provably consistent classifier and risk estimator, which is crucial for reliable learning.

Key Results

  • The proposed Adversary-Aware Partial Label Learning (AAPL) method, particularly with the ATM algorithm, consistently outperforms previous state-of-the-art methods on challenging partial label learning scenarios.
  • On CIFAR-10, the method achieved an 8.17% superior classification performance at a 0.5 partial rate compared to prior work.
  • For CUB200, AAPL showed a 5.95% improvement at a 0.1 partial rate.
  • The method demonstrated strong performance across various partial label rates (e.g., 0.1, 0.3, 0.5) on CIFAR-10, CIFAR-100, and CUB200.
  • Ablation studies confirmed the indispensable role of the adversary-aware matrix and the effectiveness of the proposed loss function.

Limitations and Future Work

  • Transition Matrix Approximation: While ATM approximates the transition matrix, it's still an approximation. The paper suggests that future work could explore variational inference methods to more accurately approximate the intractable transition matrix.
  • Hyperparameter Sensitivity: While the paper explores hyperparameter tuning (specifically 'a'), the optimal settings might still require careful tuning for different datasets and scenarios.
  • Rival Generation: The current rival generation is based on a predefined noise transition matrix. More sophisticated, adaptive rival generation strategies could be explored.

Summary updated

Abstract

To ensure that the data collected from human subjects is entrusted with a secret, rival labels are introduced to conceal the information provided by the participants on purpose. The corresponding learning task can be formulated as a noisy partial-label learning problem. However, conventional partial-label learning (PLL) methods are still vulnerable to the high ratio of noisy partial labels, especially in a large labelling space. To learn a more robust model, we present Adversary-Aware Partial Label Learning and introduce the $\textit{rival}$, a set of noisy labels, to the collection of candidate labels for each instance. By introducing the rival label, the predictive distribution of PLL is factorised such that a reasonably good predictive label is achieved with less uncertainty coming from the transition matrix, assuming its generation process is known. Nonetheless, the predictive accuracy is still insufficient to produce an adequately good set of positive samples to minimise the loss function. Moreover, the inclusion of rivals also brings an inconsistency issue for the classifier and risk function due to the intractability of the transition matrix. Consequently, the immature teacher within momentum (ITWM) disambiguation algorithm is proposed to cope with the situation. We utilise the confidence score mapping from the instance space to approximate the intractable term, allowing us to obtain a provably consistent classifier and risk function. Extensive experiments demonstrate that our method achieves promising results on the CIFAR10, CIFAR100 and CUB200 datasets.
BibTeX
@misc{chen2023adversary,
  author = {Cheng Chen and Yueming Lyu and Ivor Tsang},
  title = {{ADVERSARY-AWARE PARTIAL LABEL LEARNING WITH LABEL DISTILLATION}},
  year = {2023},
  eprint = {2304.00498},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2304.00498},
}
Figure