Positive unlabeled learning via wrapper-based adaptive sampling

Pengyi Yang, Wei Liu, Jean Yee Hwa YangIJCAI 2017 · DOI

TL;DR

This paper introduces AdaSampling, a novel approach to positive unlabeled learning (PUL) that adaptively refines training data by iteratively using a classifier's prediction probabilities. This method effectively distinguishes between positive and unlabeled instances, leading to improved classification accuracy for both single models and ensembles, even in challenging scenarios with imbalanced data.

What Problem Does This Paper Address?

Many real-world applications involve situations where obtaining labeled data is difficult, particularly for negative instances. This is known as the Positive Unlabeled Learning (PUL) problem. For example, identifying genes related to a specific disease is straightforward (positive instances), but definitively identifying genes unrelated to the disease is much harder, leading to a contaminated "negative" set. Traditional supervised learning requires both positive and negative labels, making it unsuitable here. Existing PUL methods often rely on heuristics with arbitrary thresholds, introduce biases that can be detrimental with limited data, or treat all unlabeled data as negative, which can lead to noise and reduced performance. This paper aims to address these limitations by proposing a more robust and flexible PUL method.

Methods

The core contribution is AdaSampling, a wrapper-based adaptive sampling technique. Here's how it works:

  • Initialization: Initially, all unlabeled instances are treated as potential negative examples. A base classifier is trained on the labeled positive data and these initially selected negative instances.
  • Iterative Refinement: The trained classifier then predicts the probability of each unlabeled instance belonging to the positive class. AdaSampling uses these probabilities to adaptively "sample" instances for the next iteration. Instances with a higher predicted probability of being positive are less likely to be selected as negative in subsequent training rounds. This process iteratively refines the training set, reducing the chance of misclassifying true positives as negatives.
  • Wrapper Approach: The term "wrapper" signifies that AdaSampling wraps around an existing classification model. It doesn't require modifying the base classifier itself but rather uses its output (prediction probabilities) to improve the training data. This makes it highly flexible and applicable to various classifiers that output probabilities (e.g., SVM, kNN, logistic regression).
  • Ensemble Learning: AdaSampling can be extended to create ensembles of models. By using the refined sampling probabilities, multiple diverse base classifiers can be trained on different subsets of the adaptively selected negative instances, leading to more robust predictions.

Datasets

The authors evaluated their method on:

  • Synthetic Datasets: Two synthetic datasets were generated to simulate "easy" and "hard" scenarios with varying ratios of labeled positives, unlabeled positives, and unlabeled negatives. This allowed for controlled analysis of AdaSampling's behavior.
  • Real-World Datasets: Five standard benchmark datasets were used: Breast Cancer Diagnosis (Breast), Ionosphere, Sonar Mines vs. Rocks (Sonar), Wisconsin Diagnostic Breast Cancer (WDBC), and Pima Indians Diabetes. For each dataset, two configurations were created: "easy" (50% of positive instances' labels removed) and "hard" (67% of positive instances' labels removed) to simulate PUL scenarios.

Key Insights and Ideas

  • Adaptive Probability-Driven Sampling: The key innovation is using the classifier's own confidence (prediction probabilities) to guide the selection of negative examples. This is more principled than arbitrary thresholds used in heuristic methods.
  • Noise Reduction: By iteratively down-weighting instances that are likely positive, AdaSampling effectively mitigates the problem of label noise inherent in PUL, where unlabeled data can contain hidden positives.
  • Flexibility and Generality: The wrapper approach makes AdaSampling compatible with a wide range of classification algorithms, offering a general solution to PUL.
  • Ensemble Benefits: Extending AdaSampling to ensembles further boosts performance, suggesting that combining models trained on adaptively sampled data is more effective than single models.
  • Potential for Label Noise Correction: The paper hints that AdaSampling might even be capable of identifying and correcting existing label noise in the original datasets, a significant potential benefit beyond standard PUL.

Key Results

  • Rapid Convergence: AdaSampling demonstrated effective learning and convergence in very few iterations, even on challenging "hard" synthetic datasets with high positive-to-negative ratios.
  • Superior Performance: Across all tested real-world datasets and both SVM and kNN classifiers, AdaSampling-based methods (both single and ensemble) consistently outperformed baseline approaches (treating all unlabeled as negative) and other state-of-the-art methods like "BiasModel" and "BagModel."
  • High Accuracy: AdaSampling achieved the highest F1 scores and geometric means, indicating a good balance between sensitivity and specificity, particularly in scenarios where other methods struggled.
  • Outperforming Gold Standard: In some cases, AdaSampling even surpassed the performance of models trained with complete label information, suggesting its ability to recover missing labels and potentially correct noise.

Limitations and Future Work

  • Limitations: The paper notes that while AdaSampling generally performs well, the "BiasModel" approach sometimes suffered from low specificities on the Pima dataset, indicating a potential issue with over-correction. This highlights the delicate balance in PUL.
  • Future Work: The authors suggest that AdaSampling can be easily extended for:
  • Multi-class classification: Adapting the method to handle more than two classes.
  • Class label noise identification and correction: Further developing its capability to detect and fix noisy labels in datasets.

Summary updated

BibTeX
@inproceedings{yang2017positive,
  author = {Pengyi Yang and Wei Liu and Jean Yee Hwa Yang},
  title = {{Positive unlabeled learning via wrapper-based adaptive sampling}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2017},
  doi = {10.24963/ijcai.2017/457},
}
Figure