Class Prior Overestimation Bias in Positive Unlabeled Learning

16 concepts8 papers1 proposal

Imagine you are tasked with training a machine learning model to identify a rare disease from medical scans. You are handed a small set of confirmed "positive" cases, alongside a massive pile of "unlabeled" scans. This unlabeled mixture contains both healthy individuals (negatives) and undiagnosed patients who actually have the disease (hidden positives).

Before you can train a standard classifier, you must answer a foundational question: What fraction of the unlabeled data is actually positive? This fraction is known as the class prior ($\pi$).

If the features of healthy and diseased patients were perfectly distinct, finding $\pi$ would be trivial. You would simply look for the region of the feature space where only diseased patients live, count them, and extrapolate. In the real world, however, feature distributions overlap. A patient might show mild, ambiguous symptoms that appear in both healthy individuals and early-stage disease cohorts.

When these distributions overlap, standard class-prior estimation algorithms fail in a highly predictable way: they systematically overestimate the proportion of positive samples. Understanding why this distribution overlap bias occurs—and how to fix it—is crucial for learning effectively from positive and unlabeled (PU) data.

Why Distribution Matching Fails

The most intuitive way to estimate the class prior is through partial distribution matching. Mathematically, we treat the unlabeled distribution $p_U(x)$ as a mixture of the true positive distribution $p_P(x)$ and the true negative distribution $p_N(x)$:

$$p_U(x) = \pi p_P(x) + (1-\pi) p_N(x)$$

Because we lack labeled negative data, standard algorithms attempt to find a value for $\pi$ that allows a scaled version of the positive distribution, $\pi p_P(x)$, to "fit" inside the unlabeled mixture $p_U(x)$ as snugly as possible without exceeding it anywhere.

As explained by Marthinus Christoffel du Plessis et al. in Class Prior Estimation from Positive and Unlabeled Data and Gang Niu et al. in Class-prior estimation for learning from positive and unlabeled data, this matching process becomes overly greedy when distributions overlap. Because the estimator cannot explicitly see the negative distribution, it cannot distinguish between true positive samples and negative samples that happen to look similar to the positive class.

To understand the mechanics of this bias, consider the optimization landscape. Christoffel Marthinus et al. demonstrated in Learning from Positive and Unlabeled Data(2)Computationally Efficient Estimation of Class Priors that for standard divergence-matching objectives (such as Kullback-Leibler or Pearson divergences), the derivative of the objective function at the true prior value is negative rather than zero. In optimization, a negative derivative acts as a directional force telling the algorithm to keep increasing the parameter to minimize the loss. This forces the convex objective function to yield an inflated, positively biased estimate of $\pi$.

The Collapse of Idealized Assumptions

Why did early theoretical frameworks miss this systematic bias? The answer lies in common convenience assumptions that rarely survive contact with real-world data.

Historically, theoretical guarantees for identifying class proportions relied on the disjoint support assumption. As noted by Saurabh Garg et al. in Mixture Proportion Estimation and PU Learning:A Modern Approach, this assumption requires that there are regions of the feature space where the negative class exists but the positive class probability drops to absolute zero. Similarly, Xutao Wang et al. observe in PUe: Biased Positive-Unlabeled Learning Enhancement by Causal Inference that correcting for selection biases often requires a "local certainty condition"—assuming certain regions contain exclusively positive instances.

When these pure regions do not exist, we violate what is known as the irreducibility assumption. This core concept dictates that the positive distribution cannot be entirely contained within or explained by the negative distribution.

In a comprehensive analysis, Yu Yao et al. in Rethinking Class-Prior Estimation for Positive-Unlabeled Learning showed that when the negative class is diverse enough to overlap heavily with the positive class, the mathematical framework becomes fundamentally unidentifiable. The model cannot distinguish between a genuinely high positive prior and a high degree of distributional overlap. Crucially, the authors proved that this assumption cannot be mathematically verified using only positive and unlabeled data, meaning standard algorithms will blindly output biased estimates without any internal warning mechanism.

Counteracting the Bias

To overcome distribution overlap bias, researchers have developed two primary intuition-driven strategies: penalizing the greedy matching behavior, or mathematically transforming the data to satisfy the missing assumptions.

1. Penalized Divergences

If the overestimation bias is caused by the objective function's derivative pulling the estimate upward, the most direct fix is to apply a counteracting mathematical force. Marthinus Christoffel du Plessis et al. introduced this in Class-prior Estimation for Learning from Positive and Unlabeled Data by utilizing properly penalized $f$-divergences. By incorporating a penalized $L_1$-distance into the objective function, they introduced a mathematical "brake" that penalizes the model for greedily absorbing overlapping negative samples into the positive estimate, resulting in an efficient, unbiased analytical solution.

2. Distribution Regrouping

Instead of changing the loss function, an alternative approach changes the distribution itself. The authors of Rethinking Class-Prior Estimation for Positive-Unlabeled Learning proposed a wrapper method called Regrouping CPE (ReCPE). ReCPE constructs an engineered, auxiliary distribution from the mixture data designed to artificially satisfy the irreducibility assumption. By forcing the data into a structure where a distinct positive region exists, it allows standard estimators to run safely without generating a positive bias.

The Flip Side: The Density Chasm

Interestingly, the challenges of distribution overlap cut both ways in machine learning. While too much overlap causes overestimation bias in PU learning, too little overlap creates severe failure modes in other domains.

For example, Akash Srivastava et al. explore the opposite extreme in Estimating the Density Ratio between Distributions with High Discrepancy using Multinomial Logistic Regression. They show that when two distributions have minimal overlap (the "density-chasm problem"), standard binary classifiers used for density ratio estimation learn a sharp decision boundary too quickly. This overconfidence causes them to completely miss tail information, leading to a severe underestimation of the true ratio.

Whether dealing with the overestimation caused by overlapping classes in PU learning, or the underestimation caused by disjoint classes in density estimation, the lesson remains the same: standard statistical estimators are deeply sensitive to the geometric overlap of distributions in feature space, and correcting for these geometric realities is essential for robust real-world performance.

History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_distribution_overlap_bias_353d6636,
  title = {distribution overlap bias},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/distribution-overlap-bias/rdr_353d6636736846cd828af915d7c40d98},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_distribution_overlap_bias_353d6636_v1,
  title = {distribution overlap bias},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/distribution-overlap-bias/rdr_353d6636736846cd828af915d7c40d98/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake