Rethinking Class-Prior Estimation for Positive-Unlabeled Learning

Yu Yao, Tongliang Liu, Bo Han, Mingming Gong, Gang Niu, Masashi Sugiyama, Dacheng TaoICLR 2022 · OpenReview

Introduction

Positive-Unlabeled (PU) learning—training a classifier using only positive and unlabeled data—is a crucial technique when confirmed negative examples are unavailable (e.g., diagnosing patients where the general population is unlabeled). A fundamental building block of PU learning is Class-Prior Estimation (CPE): estimating $\pi$, the proportion of true positives hidden within the unlabeled data.

Current state-of-the-art CPE methods rely heavily on a hidden theoretical requirement known as the irreducibility assumption. This assumption dictates that the positive data distribution cannot be entirely "contained" within the support of the negative data distribution. Unfortunately, in real-world scenarios, this assumption is frequently violated (e.g., the features of a healthy person might completely overlap with early disease stages). Worse, it is mathematically impossible to verify this assumption using only positive and unlabeled data. When violated, existing methods systematically overestimate the class prior, leading to poor downstream classification.

To address this, the authors introduce Regrouping CPE (ReCPE), a wrapper method that creates an auxiliary distribution to satisfy these theoretical assumptions, significantly reducing estimation bias.

Methods: Regrouping CPE (ReCPE)

Instead of attempting to estimate the prior on the original, potentially "broken" distributions, ReCPE transforms the problem into a new one where the irreducibility assumption is guaranteed to hold. ReCPE acts as a meta-method that can be applied on top of almost any existing CPE algorithm.

The algorithm follows these core steps:

  1. Initial Classification: Train a simple binary classifier (treating the unlabeled set as negative) to rank how "positive" each unlabeled sample appears.
  2. Regrouping: Take a small percentage ($p$) of the unlabeled samples that the classifier is most confident are positive and "copy" them into the positive set.
  3. Auxiliary Distribution: This creates a new, auxiliary positive distribution. By manually injecting these high-confidence samples, the method essentially "forces" the positive distribution to have a region distinct from the latent negative distribution.
  4. Base Estimation & Correction: Run any existing CPE method (like AlphaMax or KM2) on this new "regrouped" data, and use derived formulas to map the auxiliary estimate back to the original class prior.
Illustration of original distributions versus the regrouped auxiliary distributions, ensuring a distinct positive region.

Datasets and Experiments

The researchers evaluated ReCPE using both synthetic and real-world data:

  • Synthetic Datasets: Two 10-dimensional Gaussian mixture datasets were used—one designed to satisfy the irreducibility assumption and one designed to violate it (the "reducible" case).
  • Real-World Datasets: Nine benchmarks from the UCI machine learning database (including Adult, Bank, Covtype, and Magic04) were converted into PU tasks by treating specific classes as positive and others as negative/unlabeled.

Key Results and Insights

ReCPE provides a mathematically sound "safe wrapper" for existing estimators. The authors prove that if the irreducibility assumption already holds, ReCPE does not alter the result. However, if the assumption is violated, ReCPE is guaranteed to reduce the overestimation bias.

Performance difference showing ReCPE reduces error on reducible data while remaining safe on irreducible data.

Empirical results strongly support this theory:

  • Bias Reduction: On "reducible" synthetic data, ReCPE drastically reduced the estimation bias from 0.2 down to approximately 0.01.
  • Broad Improvement: ReCPE was tested against seven state-of-the-art baseline methods (including AlphaMax, DEDPUL, and KM2). It improved the accuracy of all of them across the majority of the nine real-world datasets, suggesting that the irreducibility assumption is frequently violated in practice.
  • Top Performer: The combination of AlphaMax with the regrouping wrapper (ReAM) achieved the lowest average absolute estimation error across all tested scenarios.

Limitations and Future Work

While highly effective, ReCPE introduces a hyper-parameter $p$ (the fraction of data to copy). The authors found $p=10\%$ worked well across all datasets, but the optimal value might vary depending on the specific domain. Additionally, in practical implementation, the authors "copy and paste" samples to create the auxiliary distribution, which is an approximation of the theoretical regrouping of probability masses. Future work will explore more sophisticated ways to generate the auxiliary samples to further refine the estimation process.

Summary updated

Abstract

Given only positive (P) and unlabeled (U) data, PU learning can train a binary classifier without any negative data. It has two building blocks: PU class-prior estimation (CPE) and PU classification; the latter has been well studied while the former has received less attention. Hitherto, the distributional-assumption-free CPE methods rely on a critical assumption that the support of the positive data distribution cannot be contained in the support of the negative data distribution. If this is violated, those CPE methods will systematically overestimate the class prior; it is even worse that we cannot verify the assumption based on the data. In this paper, we rethink CPE for PU learning—can we remove the assumption to make CPE always valid? We show an affirmative answer by proposing Regrouping CPE (ReCPE) that builds an auxiliary probability distribution such that the support of the positive data distribution is never contained in the support of the negative data distribution. ReCPE can work with any CPE method by treating it as the base method. Theoretically, ReCPE does not affect its base if the assumption already holds for the original probability distribution; otherwise, it reduces the positive bias of its base. Empirically, ReCPE improves all state-of-the-art CPE methods on various datasets, implying that the assumption has indeed been violated here.
BibTeX
@inproceedings{yao2022rethinking,
  author = {Yu Yao and Tongliang Liu and Bo Han and Mingming Gong and Gang Niu and Masashi Sugiyama and Dacheng Tao},
  title = {{Rethinking Class-Prior Estimation for Positive-Unlabeled Learning}},
  booktitle = {International Conference on Learning Representations},
  year = {2022},
  url = {https://openreview.net/forum?id=MQxTAL60akp},
}
Figure