Self-PU: Self Boosted and Calibrated Positive-Unlabeled Training

Xuxi Chen, Wuyang Chen, Tianlong Chen, Ye Yuan, Chen Gong, Kewei Chen, Zhangyang Wang

TL;DR

This paper introduces "Self-PU," a novel framework that enhances Positive-Unlabeled (PU) learning by integrating self-training techniques. It addresses the challenge of learning from data where only positive examples are labeled, and the rest are a mix of positive and negative, by adaptively selecting confident examples and using self-supervision to improve classifier robustness and accuracy.

What Problem Does This Paper Address?

Positive-Unlabeled (PU) learning is a critical problem in many real-world scenarios where obtaining labeled negative examples is difficult or impossible. For instance, in medical diagnostics, identifying positive cases (e.g., patients with a disease) might be straightforward, but the vast majority of the population is unlabeled, containing both truly negative (healthy) individuals and individuals in the early, asymptomatic stages of the disease. Treating all unlabeled data as negative can lead to biased classifiers that underestimate risks. Existing PU learning methods often rely on importance reweighting to estimate risks, but they largely ignore the inherent learning capacity of the model itself, which could provide valuable supervision. This paper aims to bridge this gap by leveraging the model's own learning capabilities to generate more reliable supervision.

Methods

The proposed "Self-PU" framework introduces three key "self"-oriented components:

  • Self-Paced Learning for Trusted Set Augmentation: This component adaptively selects confident positive and negative examples from the unlabeled pool. As training progresses, the model identifies examples it is most certain about and adds them to a "trusted set" (D_trust), effectively turning them into labeled data for supervised learning. This process is dynamic, allowing the model to "regret" earlier selections if they become less confident.
  • Self-Calibrated Instance-Aware Loss: For the remaining unconfident examples, a meta-learning approach is used to fine-tune the loss functions. This helps to better handle the inherent uncertainty and noise in the unlabeled data.
  • Self-Distillation for Consistency: To further improve robustness and reduce variance, a self-supervised consistency scheme is employed. This involves training multiple "student" models with slightly different learning paces (i.e., different rates of adding examples to D_trust). Their predictions are then forced to be consistent with each other. Additionally, "teacher" models, which are moving averages of the student models' weights, are introduced to provide another layer of consistency, acting as a stable reference.

Datasets

The framework was evaluated on:

  • MNIST: A standard benchmark for PU learning, where odd digits were treated as positive and even digits as negative.
  • CIFAR-10: Another common benchmark, with specific vehicle classes as positive and animal classes as negative.
  • ADNI (Alzheimer's Disease Neuroimaging Initiative): A novel, real-world benchmark for PU learning, focusing on classifying brain MRI images for Alzheimer's Disease diagnosis. This dataset is particularly challenging due to the complexity of medical imaging and the nuanced nature of disease progression, where early-stage patients might be mislabeled as negative.

Key Insights and Ideas

  • Leveraging Model's Own Learning: The core idea is that the model itself can be a powerful source of supervision, rather than solely relying on external labeling or fixed reweighting schemes.
  • Adaptive Data Selection: Dynamically identifying and labeling confident examples from unlabeled data is more effective than fixed sampling strategies, leading to a more reliable "trusted set."
  • Handling Uncertainty with Meta-Learning: Calibrating loss functions for uncertain examples is crucial for improving performance when dealing with noisy unlabeled data.
  • Consistency as Regularization: Self-supervised distillation between models with different learning paces and with their moving averages significantly improves stability and robustness, especially in complex real-world scenarios.
  • ADNI as a Realistic PU Benchmark: The paper highlights the ADNI dataset as a more challenging and practically relevant benchmark for PU learning, moving beyond saturated traditional datasets.

Key Results

  • State-of-the-Art Performance: Self-PU achieved state-of-the-art results on MNIST and CIFAR-10, outperforming existing methods like uPU, nnPU, and DAN. For example, on CIFAR-10, it surpassed nnPU by over 1%.
  • Improved ADNI Classification: On the ADNI dataset, Self-PU significantly outperformed baseline methods, achieving 79.50% accuracy, validating its effectiveness on complex, real-world data.
  • Enhanced Training Stability: The proposed framework demonstrated a more stable training process compared to existing methods, which often suffer from overfitting and unstable validation accuracy.
  • Ablation Study Findings: The paper's ablation studies confirmed the importance of each component: dynamic rate sampling and the "in-and-out" trusted set strategy for self-paced learning, and the diversity of student learning paces for distillation.

Limitations and Future Work

  • Hyperparameter Sensitivity: While the paper explores several hyperparameters (e.g., α for hard sample mining, β for teacher smoothing, γ for loss reweighting), optimal settings might still require careful tuning for different datasets.
  • Computational Cost: The integration of multiple student and teacher models for distillation, along with the self-paced learning mechanism, likely increases the overall training time and computational resources required compared to simpler PU learning methods.
  • Future Work: The authors suggest exploring more realistic PU learning settings and believe this will motivate new algorithmic findings. They also note that the selection of one teacher model over another for final prediction was for simplicity, implying that ensemble or weighted fusion of teachers could be explored.

Summary updated

Abstract

Many real-world applications have to tackle the Positive-Unlabeled (PU) learning problem, i.e., learning binary classifiers from a large amount of unlabeled data and a few labeled positive examples. While current state-of-the-art methods employ importance reweighting to design various risk estimators, they ignored the learning capability of the model itself, which could have provided reliable supervision. This motivates us to propose a novel Self-PU learning framework, which seamlessly integrates PU learning and self-training. Self-PU highlights three "self"-oriented building blocks: a self-paced training algorithm that adaptively discovers and augments confident positive/negative examples as the training proceeds; a self-calibrated instance-aware loss; and a self-distillation scheme that introduces teacher-students learning as an effective regularization for PU learning. We demonstrate the state-of-the-art performance of Self-PU on common PU learning benchmarks (MNIST and CIFAR-10), which compare favorably against the latest competitors. Moreover, we study a real-world application of PU learning, i.e., classifying brain images of Alzheimer's Disease. Self-PU obtains significantly improved results on the renowned Alzheimer's Disease Neuroimaging Initiative (ADNI) database over existing methods. The code is publicly available at: https://github.com/TAMU-VITA/Self-PU.
BibTeX
@inproceedings{chen2020self,
  author = {Xuxi Chen and Wuyang Chen and Tianlong Chen and Ye Yuan and Chen Gong and Kewei Chen and Zhangyang Wang},
  title = {{Self-PU: Self Boosted and Calibrated Positive-Unlabeled Training}},
  booktitle = {International Conference on Machine Learning},
  year = {2020},
}
Figure