In supervised machine learning, we rely on the Empirical Risk Minimization (ERM) framework. To train a model, we minimize an objective function—the empirical risk—which approximates the true expected loss over a labeled data distribution. But what happens when our labels are incomplete, corrupted, or entirely missing?
The naive approach is to use heuristics: treating unlabeled data as negative examples, or guessing labels via pseudo-labeling. However, as du Plessis et al. demonstrated, these shortcuts introduce systematic biases, causing standard convex loss functions to draw incorrect decision boundaries.
Instead of guessing missing labels, a powerful alternative is unbiased risk rewriting. This paradigm mathematically transforms the standard supervised risk function into an algebraically equivalent form that depends only on the observed, weakly supervised data distributions. By leveraging the linearity of expectation, we can train models that provably converge to the exact same optimal decision boundaries as if they had access to fully supervised ground truth.
The Intuition: Exploiting Distribution Mixtures
To understand why risk rewriting works, consider the classic Positive-Unlabeled (PU) learning scenario. We have a set of confirmed positive samples (e.g., diagnosed patients) and a massive pool of unlabeled data containing a mix of hidden positives and true negatives.
We cannot directly compute the standard classification risk because we lack explicit negative samples to evaluate the model's negative-class loss ($R_N^-$). However, we do know that the unlabeled distribution ($P_U$) is a clean geometric mixture of the true positive ($P_P$) and true negative ($P_N$) distributions, governed by the class prior $\pi$ (the proportion of positives in the unlabeled set):
$$P_U = \pi P_P + (1 - \pi) P_N$$
Because expectation is a linear operator, this mixture relationship extends directly to our loss functions. If we evaluate our model on the unlabeled dataset while treating every single instance as a negative example, the resulting unlabeled risk ($R_U^-$) is simply a weighted combination of how the model performs on the hidden positives and the true negatives:
$$R_U^-(f) = \pi R_P^-(f) + (1 - \pi) R_N^-(f)$$
Here lies the breakthrough: we do not have a dataset of true negatives to compute $R_N^-(f)$, but we do have an unlabeled dataset to compute $R_U^-(f)$ and a positive dataset to compute $R_P^-(f)$. By basic algebra, we can isolate the missing negative risk term:
$$(1 - \pi) R_N^-(f) = R_U^-(f) - \pi R_P^-(f)$$
By substituting this expression back into the standard supervised risk formula, the authors derived an objective function composed entirely of observable data:
$$R(f) = \pi R_P^+(f) + R_U^-(f) - \pi R_P^-(f)$$
This formulation allows a classifier to approximate the true binary decision boundary without seeing a single explicit negative label, as highlighted by Niu. The model learns to subtract the expected loss of the hidden positive samples from the total unlabeled loss, effectively neutralizing the bias of the unlabeled pool.
Generalizing Beyond Positive-Unlabeled Data
The algebraic magic of risk rewriting is not restricted to PU learning. Any weakly supervised setting where the observed data can be modeled as a linear transformation of the true data distribution is a candidate for an unbiased risk estimator.
Complementary-Label Learning
In multi-class problems, specifying what an object is can be time-consuming, but specifying what it is not (a complementary label) is often trivial. Takashi et al. and Ishida et al. proved that if complementary labels are selected uniformly from the non-ground-truth classes, the true multi-class risk can be rewritten using only these negative indicators. This mathematical guarantee ensures that models trained on "negative-only" labels still converge to the true optimal classifier, a framework Gao et al. successfully extended to multi-label settings.
Unlabeled-Unlabeled (UU) Learning
Remarkably, it is possible to train a binary classifier with zero labeled data. Lu et al. demonstrated that if we possess two distinct unlabeled datasets collected from environments with different class priors (e.g., two different cities), we can set up a system of linear equations. By algebraically weighting the losses across both unlabeled datasets, we can isolate the underlying positive and negative risks. Tang et al. later generalized this principle to multi-class classification using known class proportions.
Aggregate Observations
In applications constrained by privacy or data collection limits, data may only be available as group-level statistics (e.g., "this batch of 100 samples contains 30% positives"). Wei et al. showed that risk rewriting techniques can reconstruct individual-level classification risk from these aggregate observations, matching the performance of individual supervision.
The Practical Catch: Non-Negative Risk Correction
While risk rewriting works perfectly in expectation, implementing it in deep learning introduces a severe practical challenge: negative empirical risk.
Because the rewritten risk functions involve subtracting stochastic terms (such as subtracting $\pi R_P^-(f)$ from $R_U^-(f)$), the empirical estimate computed over a finite mini-batch can mathematically drop below zero. In a standard setting, risk cannot be negative. When a powerful deep neural network overfits the positive samples during a training iteration, the subtracted term can become artificially large, pushing the total estimated risk into negative territory.
Standard Risk: [ Labeled Positive Risk ] + [ Labeled Negative Risk ] >= 0 (Stable)
Rewritten Risk: [ Labeled Positive Risk ] + [ Unlabeled Risk ] - [ Positive Risk as Negatives ]
^-- Can cause total risk to drop < 0
When the risk becomes negative, gradient descent reverses its behavior. Instead of minimizing the classification error, the optimizer begins to maximize the loss on the unlabeled data to push the negative number even lower, causing the model's performance to collapse.
To resolve this instability, Lu et al. introduced a consistent risk correction approach. They proposed decomposing the total empirical risk into its class-specific components and wrapping the potentially negative terms in a non-negative constraint, effectively clipping them at zero:
$$\widetilde{R}(f) = \pi R_P^+(f) + \max\left(0, R_U^-(f) - \pi R_P^-(f)\right)$$
This simple, ReLU-like mathematical adjustment prevents the objective function from dropping below zero, stabilizing the gradients and allowing deep architectures to train reliably under weak supervision.
Real-World Impact
Unbiased risk estimation has transitioned from a statistical curiosity to a core tool for solving real-world data bottlenecks across diverse domains:
- Implicit Feedback in Recommender Systems: In recommendation engines, users rarely click "dislike"; they simply interact with items (positives) or ignore them (unlabeled). Wang et al. leveraged risk rewriting to eliminate the systematic bias introduced by traditional negative sampling, treating unobserved interactions as a true mixture distribution.
- Open-Set Domain Adaptation: When deploying models to new environments, models often encounter entirely novel classes. Zhang et al. reframed open-set domain adaptation as a multi-class PU problem, treating the target domain as a mixture of known and unknown categories to bound target risk without explicit labels for the novel classes.
- Bypassing Class Priors in Evaluation: In highly imbalanced scenarios, estimating the exact class prior can be as difficult as labeling the data itself. Li et al. proved that for Area Under the ROC Curve (AUC) optimization, the supervised risk can be reconstructed by treating the entire unlabeled dataset as positive and negative simultaneously, achieving unbiased semi-supervised AUC optimization without needing to guess labels or estimate class priors.
- Domain-Specific Annotations: From predicting biological functions using incomplete gene ontologies (Zhapa-Camacho et al.) to detecting software functional clones (Wei et al.), risk rewriting provides a mathematically rigorous foundation for learning from the incomplete data distributions inherent to real-world systems.