Many clinical prediction problems have confirmed cases and a large pool of everyone else. Some of the "everyone else" are cases that were never diagnosed. Learning a classifier from this kind of data is called positive-unlabeled (PU) learning. The standard trick estimates the error on the unseen negatives by taking the loss over the unlabeled pool and subtracting the loss over the known positives. That subtraction can go negative on finite data, and flexible models such as boosted trees and deep networks exploit it. Existing fixes clip, reflect, or rebalance the negative term. All of them assume the known positives look, feature for feature, like the hidden positives in the unlabeled pool. In electronic health records this fails for a mundane reason: once a patient is diagnosed, they get tested more. Their records have different missing cells. After imputation (filling the gaps with plausible values), they also carry the statistical fingerprints of imputed cells.
Our claim is that this gap between diagnosed and undiagnosed patients is a hidden channel. A flexible model can use it to tell "diagnosed" apart from "sick," which drives the negative risk term below zero for reasons unrelated to the disease. Clipping hides the symptom while the classifier still learns the labeling process. We propose a simple correction. Before computing the subtracted term, each known positive is passed through the same mask-then-impute pipeline that undiagnosed patients experience. We call this re-masking. It uses multiple imputation draws, and we test it in regimes where redundant, collinear lab panels make the hidden channel wider.
Core Claim
The PU risk estimator is unbiased only if the labeled positives and the hidden positives in the unlabeled pool share a feature distribution. This is the "selected completely at random" (SCAR) assumption. Missingness and imputation act on the features. Suppose SCAR holds for the true, complete features. It can still fail for the observed-and-imputed features whenever the missingness pattern depends on the label status $S$ (diagnosed or not), and not only on the patient's physiology. In that case, a high-capacity model can push the term $\hat R_u^-(g) - \pi_p \hat R_p^-(g)$ negative by recognizing the labeling signature. Here $\hat R_u^-$ is the loss on the unlabeled data treated as negatives, $\hat R_p^-$ is the same loss on labeled positives, and $\pi_p$ is the positive class prior. Two quantities follow from this view. First, a held-out estimate of that term becomes a measurable canary for imputation-induced SCAR violation. Second, the violation can be repaired by making the transformation identical for both groups.
Let $T(x, m)$ denote applying a missingness mask $m$ to features $x$ and then imputing. Let $\hat q(m \mid x_c)$ be a mask model fitted on unlabeled records, where $x_c$ is a core block of always-recorded variables (demographics, vitals, admission labs). The re-masked non-negative risk is
$$\tilde R(g) = \pi_p \hat{\mathbb E}_P\big[\ell(g(T(x,m_x)),+1)\big] + \max\Big\{0,\ \hat{\mathbb E}_U\big[\ell(g(T(x,m_x)),-1)\big] - \pi_p \hat{\mathbb E}_{P,\ \tilde m \sim \hat q}\big[\ell(g(T(x,m_x \cup \tilde m)),-1)\big]\Big\},$$
where $\ell$ is the classification loss, $g$ the classifier, $m_x$ the record's own mask, and $\tilde m$ an extra mask sampled for each labeled positive.
All expectations are averaged over $M$ multiple-imputation draws, so the imputation noise in the subtracted term matches that in the unlabeled term. Consider the assumption that, among undiagnosed patients, test ordering depends on disease only through recorded physiology ($m \perp Y \mid x_c, S=0$). Under that assumption, re-masked positives match the hidden positives in distribution, and the correction restores unbiasedness in the space the model actually sees. Re-masking can only remove cells. Labeled positives with cells missing that $\hat q$ would have observed are left as is, and we report how often this happens.
How this differs from existing work
The non-negative correction line motivates the work and supplies the base estimators. Positive-Unlabeled Learning with Non-Negative Risk Estimator clips the negative term at zero. Learning from Positive and Unlabeled Data with Arbitrary Positive Shift uses an absolute value instead and handles shift between training and test positives. Balancing Positive and Negative Classification Error Rates in Positive-Unlabeled Learning replaces the zero floor with a moving lower bound plus an error-parity constraint, and evaluates on an Alzheimer's cohort. Positive-Unlabeled Learning using Random Forests via Recursive Greedy Risk Minimization builds the clipped estimator into tree splits. Every one of these treats negative risk as finite-sample or capacity noise on fully observed features. None asks whether the feature pipeline itself creates the violation. Our proposal leaves their loss geometry alone and changes what the labeled positives are fed through, so it composes with any of them.
The imputation literature constrains the design. Prediction-based Inference in Electronic Health Record (EHR)-linked Biobanks shows that naive plug-in imputation gives a 100% Type I error rate when test ordering depends on illness. This is the inference-side version of our concern, but it addresses association testing with fully labeled outcomes. RefiDiff and the MICE pipeline in Machine learning vs. traditional regression analysis for fluid overload prediction in the ICU supply the imputers we reuse. Both treat imputation as a label-agnostic preprocessing step and never examine how imputed cells differ across label groups.
The redundancy work defines the regime. Deep Tabular Representation Corrector finds that redundant, high-entropy tabular representations hurt accuracy. Gaussian Image Anomaly Detection with Greedy Eigencomponent Selection shows that high-variance directions often carry nuisance structure. We hypothesize that redundant lab panels widen the leak in a specific way: one post-diagnosis test order fills many correlated columns at once, and imputed values shrink toward their correlates in a pattern trees can detect. The non-linear-interaction literature explains why the leak matters most for flexible models. The fluid-overload study shows XGBoost finding signal that logistic regression misses, through interactions. The capacity that finds clinical interactions also finds labeling artifacts, and Mitigating Overfitting in Supervised Classification from Two Unlabeled Datasets reports that low-capacity linear models are largely immune to negative-risk overfitting.
Research Program
The first part is a controlled demonstration. We start from complete tabular benchmarks used in the cited work. Covtype and UNSW-NB15 come from the PU Extra Trees study. Adult, Default, News and Shoppers come from the RefiDiff suite and include mixed-type and wider tables. On each we create a PU split that satisfies SCAR in the full feature space, so ground-truth labels for the unlabeled pool are known. We then inject missingness through three knobs. The first is physiology-driven missingness, which depends on feature values and on $Y$ only through $x$. The second is label-driven extra observation, where diagnosed records get additional columns revealed. The third is redundancy, added by appending correlated noisy copies of the label-relevant columns at controlled correlation and count.
We sweep the strength of label-driven observation from zero upward, crossed with redundancy level and model family. The model families are logistic regression, PU Extra Trees, XGBoost trained with the non-negative objective, and an MLP. The first prediction is that leakage and held-out negative risk rise with label-driven observation and with redundancy, mainly for the non-linear models. The second is that clipping and parity constraints stop training-loss divergence without restoring accuracy on true labels.
The second part builds and analyzes the correction. The mask model $\hat q$ is a per-column logistic or small boosted model fitted on unlabeled records, conditioned on the core block. Imputation uses MICE and RefiDiff, each with $M$ draws, and the imputer is fitted blind to $S$. We test whether the correction's benefit tracks the size of the violation, and how it degrades when the conditional-ignorability assumption is broken on purpose (test ordering depending directly on hidden $Y$). We also check whether held-out negative risk can serve as a model-selection and early-stopping signal when true labels are unavailable.
The third part is a real-data check where the mechanism is plausible. The candidate is All of Us, used in the prediction-based inference paper: diabetes diagnosis codes as labeled positives, and HbA1c and related panels as features whose measurement depends on diagnosis. A second candidate is the high-dimensional Alzheimer's proteomics setting in Applying machine learning to high-dimensional proteomics datasets, if missingness metadata are available. In both, ground truth is incomplete. The real-data claims are therefore limited to two checks: whether the leak diagnostic fires, and whether re-masking changes predictions on a subgroup with confirmatory measurements taken before diagnosis.
Baselines
The naive reference point is the uncorrected unbiased PU estimator on MICE-imputed data, alongside logistic regression with the non-negative objective. The first shows how badly flexible models hack the loss. The second shows what a model gets when it has too little capacity to exploit the leak and too little to capture interactions. Beating the naive reference means higher AUROC on the true labels of the unlabeled pool at matched imputation.
The strongest existing competitors are DC-PU from Balancing Positive and Negative Classification Error Rates in Positive-Unlabeled Learning for neural models and PU Extra Trees from Positive-Unlabeled Learning using Random Forests via Recursive Greedy Risk Minimization for trees. They are the strongest because they are the most developed answers to negative risk in, respectively, deep clinical PU and tabular PU. Each is run on exactly the same imputed datasets, with the same $M$ draws, the same true prior $\pi_p$, and an equal hyperparameter search budget, with validation done only on PU data. No result is reported on Lacuna for DC-PU or PU Extra Trees on incomplete tables. The file also reports no extractable F-score values for PU Extra Trees on Covtype, Epsilon or UNSW-NB15, nor exact F1 or GAP values for DC-PU on its Alzheimer's data. We will therefore reproduce both on the complete-data versions of our benchmarks as a sanity anchor. The closest number the file provides is from a different, fully supervised setting: XGBoost with MICE reached AUROC 0.78 on ICU fluid overload, according to the fluid overload study. It shows the scale of interaction gains, but no comparison is drawn against it. Beating these baselines means a statistically reliable gain in true-label AUROC and F1, a reduction in the leak score defined below, and no worse error gap, at every label-driven-observation level above zero. Results should match the baselines at level zero.
Three ablations isolate the key ingredient. The first applies re-masking with a single imputation instead of $M$ draws, to test whether matching imputation noise matters. The second uses an S-blind imputer without re-masking, to test whether blinding alone suffices. The third uses random re-masking drawn from the marginal unlabeled mask distribution instead of $\hat q(m \mid x_c)$, to test whether conditioning on physiology matters.
Why not just…?
Why not just use features recorded before diagnosis? When the diagnosis time is known and pre-diagnosis records are rich, this is the cleanest fix, and it is a real threat. In PU data, however, the unlabeled pool has no diagnosis time, so the cutoff must be imposed by an arbitrary index date. That discards the post-index measurements that carry interaction signal. We run this as a baseline in the semi-synthetic study, where "pre-label" columns are known exactly. If a temporal cutoff matches re-masking at equal information, we will recommend it and restrict our claim to settings without reliable timestamps.
Why not just let the model see missingness indicators, or use XGBoost's native missing-value handling? This is the standard practice when missingness is informative, but here the informative part includes the labeling process itself. Indicators hand the model the label fingerprint directly, so we expect the leak score to rise. This is tested directly as an arm of the sweep. The prediction-based inference result, where naive use of ML-predicted values gave a 100% Type I error under illness-driven missingness, is the closest evidence that informative missingness used naively corrupts downstream conclusions.
Why not just rely on nnPU, abs-PU or DC-PU, since they already prevent negative risk? They prevent the training loss from diverging, which is what their papers measure. For example, nnPU kept CIFAR-10 test error around 11–12% where the unbiased estimator stayed above 40%, according to the nnPU paper. Clipping a term at zero does not change which features the classifier uses to make the term small. Our decisive measurement is the leak score under each correction. If DC-PU's parity constraint already drives the leak score to chance, the proposal's added value collapses, and we will say so.
Why not just prune or compress the redundant features first? Compression in the style of TRC or greedy eigencomponent selection removes duplicate channels. It does not remove the core columns that diagnosis causes to be observed, and pruning before imputation discards the correlated columns imputation needs. We include PCA-then-PU and correlation-filter-then-PU arms at each redundancy level. The prediction is that they narrow but do not close the gap to re-masking, and that they cost true-label accuracy when the pruned columns carried interaction signal.
Proposed Experiments and Decisive Checks
The central measurement is the leak score. In the semi-synthetic setting the hidden positives are known, so this is the AUROC of the trained classifier's score at separating labeled positives from unlabeled positives. Under SCAR in the model's input space it should be 0.5. Secondary metrics are true-label AUROC and Macro-F1 on the unlabeled pool, the gap between false-negative and false-positive rates, and the held-out value of $\hat R_u^- - \pi_p \hat R_p^-$ averaged over imputation draws. Each configuration uses ten random masks, following the RefiDiff protocol, and five PU splits.
The proposal stands or falls on four checks. First, with no label-driven observation, all methods should tie. Otherwise the correction is doing something other than what we claim. Second, as label-driven observation grows, the leak score of clipped and parity-constrained flexible models should rise well above 0.5, more steeply at higher redundancy, while re-masking keeps it near 0.5 and recovers true-label AUROC. Third, held-out negative risk should correlate with the leak score across configurations, which would justify it as a label-free diagnostic. Fourth, when test ordering is made to depend on hidden $Y$ directly, the correction should degrade gracefully and remain no worse than the baselines. If the first or second check fails, the mechanism is wrong. If only the third fails, the method survives but the diagnostic claim is dropped.
References
- Positive-Unlabeled Learning with Non-Negative Risk Estimator
- Learning from Positive and Unlabeled Data with Arbitrary Positive Shift
- Balancing Positive and Negative Classification Error Rates in Positive-Unlabeled Learning
- Positive-Unlabeled Learning using Random Forests via Recursive Greedy Risk Minimization
- Prediction-based Inference in Electronic Health Record (EHR)-linked Biobanks
- RefiDiff: Progressive Refinement Diffusion for Efficient Missing Data Imputation
- Machine learning vs. traditional regression analysis for fluid overload prediction in the ICU
- Deep Tabular Representation Corrector
- Gaussian Image Anomaly Detection with Greedy Eigencomponent Selection
- Mitigating Overfitting in Supervised Classification from Two Unlabeled Datasets: A Consistent Risk Correction Approach
- Applying machine learning to high-dimensional proteomics datasets for the identification of Alzheimer’s disease biomarkers