Abstract
Positive and Unlabeled (PU) learning is a challenging machine learning task where only a small set of labeled positive samples and a large set of unlabeled samples are available. Traditional PU learning methods typically rely on an explicit estimation of the class prior—the proportion of positive samples within the unlabeled set. However, accurate prior estimation is often difficult, and errors can lead to significantly biased classifiers. This paper proposes a novel strategy, LPUb, which bypasses explicit class prior estimation. By minimizing the Bhattacharyya coefficient to reduce the overlap between positive and negative class distributions modeled via an implicit mixture of Restricted Boltzmann Machines (RBMs), the method achieves superior robustness and performance across synthetic and real-world datasets.
Introduction
In standard supervised learning, models are trained on datasets where every instance is labeled (e.g., "cat" vs. "not-cat"). However, many real-world scenarios—such as identifying rare land-cover types in satellite imagery or document retrieval—only provide a handful of known positives and a massive pool of unlabeled data that contains both hidden positives and negatives. This is the domain of Positive and Unlabeled (PU) learning.
Historically, solving PU problems required a two-step process: first, estimate the "class prior" ($\pi$), and second, use that estimate to weight samples during training. The class prior represents the exact percentage of positives hidden in the unlabeled pile. If this estimate is inaccurate, the resulting model becomes either too conservative or too aggressive, leading to poor generalization.
Methodology: Bypassing the Class Prior
The core innovation of this research is shifting the focus from estimating a numerical scalar ($\pi$) to minimizing a statistical distance. The authors propose using the Bhattacharyya Coefficient ($B$), a measure of the overlap between two statistical distributions ranging from 0 (no overlap) to 1 (identical distributions).
Implicit Mixture of RBMs (IRBM)
To model complex data distributions, the researchers utilize an Implicit Mixture Model of Restricted Boltzmann Machines. This allows for the direct representation of the conditional probability density functions (PDFs) for both positive and negative classes. The objective is to find a negative distribution that is as distinct as possible from the positive distribution while still accurately explaining the unlabeled data manifold.
Optimization Strategy
Because the objective function involves non-differentiable terms like the Heaviside function, the authors employ a "soft" optimization approach using the arctan function. This enables the use of gradient-descent-based training to simultaneously learn the distribution shapes and the decision boundary in an end-to-end fashion.
Experimental Results
The LPUb method was evaluated against state-of-the-art benchmarks, including nnPU, CSLPU, and GDD, across synthetic 2D distributions, MNIST handwritten digits, and the Scene dataset.
Key Findings
- Robustness to Overlap: On synthetic data, LPUb significantly outperformed other methods when classes were highly mixed (small $\mu$). As shown in the experimental plots, LPUb maintains lower empirical error and higher F-measure/G-mean scores in high-overlap scenarios.
- Prior Independence: While competing methods showed fluctuating performance based on the accuracy of prior estimation, LPUb remained remarkably stable across a wide range of class prior values (from 0.1 to 0.5).
- Small Labeled Sets: The model performed exceptionally well even when the labeled positive set was very small (e.g., using only 1% of a digit class in MNIST), demonstrating its ability to leverage the structure of unlabeled data effectively.
Conclusion
The LPUb framework offers a powerful alternative to traditional PU learning by treating class separation as a geometric optimization problem rather than a parameter estimation task. By minimizing the Bhattacharyya coefficient, the model avoids the "chicken and egg" problem of class prior estimation. Future work may involve integrating more modern generative architectures, such as Variational Autoencoders (VAEs), to further enhance the model's scalability and performance on high-dimensional, non-linear data manifolds.
Summary updated