Who Is Your Right Mixup Partner in Positive and Unlabeled Learning

Changchun Li, Ximing Li, Lei Feng, Jihong OuyangICLR 2022 · OpenReview

TL;DR

Positive and Unlabeled (PU) learning often suffers from "decision boundary deviation," where the classifier's boundary shifts too far toward the positive side because it mistakes unlabeled positive samples for negatives. This paper introduces P³Mix, a novel method that uses a heuristic "Mixup" strategy to pair ambiguous unlabeled instances with positive instances near the boundary. This effectively "pushes" the decision boundary back to its rightful place, significantly improving classification accuracy.

What Problem Does This Paper Address?

In many real-world scenarios—like medical diagnosis or fraud detection—we have a small set of confirmed Positive cases (e.g., diagnosed patients) and a massive pile of Unlabeled data (e.g., the general population). The unlabeled data is a mix of both hidden positives and true negatives. This setup is known as PU Learning.

The standard "naive" approach is to treat all unlabeled data as negative. However, the authors observed a consistent problem: the resulting decision boundary deviates toward the positive side. Because the model sees some positive examples labeled as "negative" (the unlabeled ones), it becomes overly cautious, shrinking the "positive" region. The authors' goal was to find a way to use data augmentation (specifically Mixup) to correct this bias without needing to know the true labels of the unlabeled set.

Key Insights and Ideas

  • The Boundary Deviation Phenomenon: The "aha moment" of this research was visualizing that PU models don't just fail randomly; they systematically "squish" the positive class. As training progresses, the predicted positive prior (PP rate) drops below the real positive prior, leading to increased error rates.
PP rate and Error rate over epochs demonstrating the boundary deviation phenomenon.
  • Mixup as Supervision Correction: While Mixup is usually just for data augmentation, here it acts as a bridge. By mixing a "maybe positive" (unlabeled) with a "definitely positive" (labeled), the resulting sample has a soft label that is more "correct" than the original "pseudo-negative" label.
  • Strategic Partnering: The paper argues that who you mix with matters. Mixing ambiguous samples with boundary-dwelling positives provides the most informative signal for the model to refine its decision threshold.

Methods: Introducing P³Mix

The authors propose Positive and unlabeled learning with Partially Positive Mixup (P³Mix). The core innovation is a heuristic mixup partner selection strategy:

  1. Identify Marginal Pseudo-Negatives ($X_{mpn}$): These are unlabeled instances that the current model is "unsure" about (predictive scores near the threshold). They are the most likely to be hidden positives causing boundary deviation.
  2. Build a Candidate Mixup Pool ($X_{cnd}$): The model maintains a pool of labeled positive instances that have high entropy (meaning they are near the current decision boundary).
  3. Heuristic Mixup: Instead of mixing random pairs, $X_{mpn}$ instances are specifically paired with instances from $X_{cnd}$. This creates augmented samples that are "partially positive" and located right where the boundary needs to be corrected.

Robustness Variants: * P³Mix-E: Adds "Early-learning regularization" to prevent the model from memorizing the noisy labels in the unlabeled set. * P³Mix-C: Adds "Pseudo-negative instance correction," where unlabeled instances with very high positive confidence have their labels flipped to positive before the Mixup process.

Datasets and Key Results

The researchers evaluated their method on three standard image benchmarks, modified into synthetic PU tasks (Fashion-MNIST, CIFAR-10, STL-10), as well as a real-world Credit Card Fraud Detection dataset from Kaggle to demonstrate practical utility in unbalanced, non-image scenarios.

P³Mix consistently outperformed state-of-the-art PU learning methods (like nnPU and Self-PU) across all benchmarks:

  • Accuracy Gains: On datasets like CIFAR-10 and STL-10, P³Mix achieved improvements of 1% to 5% over existing methods.
  • Robustness: The P³Mix-E and P³Mix-C variants showed superior stability, especially when the "pseudo-negative" labels were particularly noisy.
  • Real-World Performance: On the credit card fraud dataset, P³Mix-E achieved the highest Recall (87.7%), which is critical for fraud detection where missing a "positive" (a fraud case) is very costly.

Limitations and Future Work

  • Hyperparameter Sensitivity: The method relies on a threshold parameter ($\gamma$) and a coefficient ($\beta$). While the authors provide a sensitivity analysis, finding the optimal values for a brand-new dataset might require careful tuning.
  • Computational Overhead: Maintaining a candidate mixup pool and calculating entropy for positive instances adds some computational complexity compared to simpler PU methods.
  • Future Directions: The authors suggest that this heuristic mixup approach could be extended to other weakly supervised tasks, such as multi-label learning or learning with noisy labels.

Summary updated

Abstract

Positive and Unlabeled (PU) learning targets inducing a binary classifier from weak training datasets of positive and unlabeled instances, which arise in many real-world applications. In this paper, we propose a novel PU learning method, namely Positive and unlabeled learning with Partially Positive Mixup (P3Mix), which simultaneously benefits from data augmentation and supervision correction with a heuristic mixup technique. To be specific, we take inspiration from the directional boundary deviation phenomenon observed in our preliminary experiments, where the learned PU boundary tends to deviate from the fully supervised boundary towards the positive side. For the unlabeled instances with ambiguous predictive results, we select their mixup partners from the positive instances around the learned PU boundary, so as to transform them into augmented instances near to the boundary yet with more precise supervision. Accordingly, those augmented instances may push the learned PU boundary towards the fully supervised boundary, thereby improving the classification performance. Comprehensive experimental results demonstrate the effectiveness of the heuristic mixup technique in PU learning and show that P3Mix can consistently outperform the state-of-the-art PU learning methods.
BibTeX
@inproceedings{li2022who,
  author = {Changchun Li and Ximing Li and Lei Feng and Jihong Ouyang},
  title = {{Who Is Your Right Mixup Partner in Positive and Unlabeled Learning}},
  booktitle = {International Conference on Learning Representations},
  year = {2022},
  url = {https://openreview.net/forum?id=NH29920YEmj},
}
Figure