Learning A Disentangling Representation For PU Learning

Omar Zamzam, Haleh Akrami, Mahdi Soltanolkotabi, Richard M. Leahy

Abstract

Positive-Unlabeled (PU) learning is a common challenge in real-world machine learning where only a subset of positive instances is labeled, and the remaining data is unlabeled. In high-dimensional spaces, such as image datasets, positive and negative distributions often become deeply entangled, making traditional classification difficult. We propose a novel representation learning method, VQ K-means, which forces unlabeled data into two distinct, easily separable clusters. By combining a specialized "tug-of-war" loss function with vector quantization, our approach simplifies high-dimensional PU learning into a manageable clustering problem. Experimental results on benchmarks like AFHQ demonstrate that our method significantly outperforms current state-of-the-art models in both accuracy and convergence speed.

Introduction

In many practical scenarios—ranging from medical diagnosis to fraud detection—confirmed negative labels are unavailable. We typically possess a set of "positive" examples and a large pool of "unlabeled" data that contains both hidden positives and negatives. This is the core of PU Learning. While low-dimensional PU problems can often be solved by identifying distinct modes in a distribution, high-dimensional data (like images) lacks this inherent simplicity. Current deep learning methods often struggle to find a clear decision boundary without explicit negative supervision, leading to slow training and suboptimal performance.

Methodology: VQ K-means

Our approach focuses on learning a new representation of the data that mimics the simplicity of low-dimensional toy problems where K-means clustering is effective.

The "Tug-of-War" Loss

The central innovation is a loss function designed to create distributional contrast. We define two target vectors in the representation space: a "high-magnitude" vector for labeled positives and a "low-magnitude" vector for all unlabeled samples.

  • Labeled Positives: Pulled toward the high-magnitude target.
  • Unlabeled Samples: Pulled toward the low-magnitude target.

Because the unlabeled set contains both positives and negatives, the unlabeled positive samples experience a "tug-of-war" between the two targets, eventually settling into a linear combination. Unlabeled negatives, however, are only pulled toward the low-magnitude target. This mechanism naturally separates the two classes into distinct clusters.

Vector Quantization (VQ)

To prevent these clusters from overlapping and to sharpen the decision boundary, we incorporate vector quantization. As shown in the architecture diagram below, the learned representations are forced to "snap" to specific codebook vectors.

Figure 3: The VQ K-means architecture maps input $x$ to a representation $v$, which is then quantized to the nearest codebook vector $c_i$.

Stopping Criterion

A major hurdle in PU learning is the lack of a labeled validation set. We observed that as the model begins to overfit, the centers of the learned clusters start to converge. By monitoring the distance between these cluster centers, we can identify the optimal stopping point—the peak of accuracy—without needing any negative labels for validation.

Experimental Results

We evaluated VQ K-means on several image benchmarks, including MNIST, Fashion-MNIST, CIFAR-10, and AFHQ (Animal Faces).

Disentanglement and Accuracy

The primary goal was to achieve clear disentanglement in high-dimensional space. On the AFHQ dataset (Cats vs. Dogs), our method achieved a state-of-the-art accuracy of 95.3%, surpassing baselines like Observer-GAN and TEDn. The effectiveness of the disentanglement is clearly visible in t-SNE projections of the learned space.

Figure 4: t-SNE visualization of the learned representation space, showing clear separation between positive and negative samples.

Training Efficiency

Beyond accuracy, VQ K-means offers a dramatic improvement in training speed. On the AFHQ dataset, our model reached 90% of its peak accuracy in just 11 epochs. In contrast, competing methods required between 43 and 220 epochs to reach similar performance levels.

Conclusion

By reframing high-dimensional PU learning as a representation learning problem, VQ K-means provides a robust and efficient solution. The combination of a contrastive loss and vector quantization successfully "disentangles" positive and negative distributions, making them as separable as a simple 1D toy problem. While the method currently relies on the "Selected Completely At Random" (SCAR) assumption, its superior speed and accuracy make it a powerful tool for real-world unlabeled datasets.

Summary updated

Abstract

In this paper, we address the problem of learning a binary (positive vs. negative) classifier given Positive and Unlabeled data commonly referred to as PU learning. Although rudimentary techniques like clustering, out-of-distribution detection, or positive density estimation can be used to solve the problem in low-dimensional settings, their efficacy progressively deteriorates with higher dimension due to the increasing complexities in the data distribution. In this paper we propose to learn a neural network-based data representation using a loss function that can be used to project the unlabeled data into two (positive and negative) clusters that can be easily identified using simple clustering techniques, effectively emulating the phenomenon observed in low-dimensional settings. We adopt a vector quantization technique to the learned representations to amplify the separation between the learned unlabeled data clusters. We conduct experiments on simulated PU data that demonstrate the improved performance of our proposed method compared to the current state-of-the-art approaches. We also provide some theoretical justification for our two cluster-based approach and some of our algorithmic choices.
BibTeX
@misc{zamzam2023learning,
  author = {Omar Zamzam and Haleh Akrami and Mahdi Soltanolkotabi and Richard M. Leahy},
  title = {{Learning A Disentangling Representation For PU Learning}},
  year = {2023},
  eprint = {2310.03833},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2310.03833},
}
Figure