Hospitals increasingly want image models that act as safety filters. The model sorts incoming cases into "needs a human look" and "safe to set aside," and the only error that really matters is a missed disease, because a false alarm costs a few minutes of a reviewer's time. Such models are usually trained and checked on labels harvested from routine clinical work, such as the lesion a radiologist happened to measure or the finding recorded in a report, because dense expert annotation is too expensive. Those labels are one-sided. When a label is present, the disease is almost certainly there. When it is absent, the disease may still be there. The consequence is that the miss rate computed on harvested labels is optimistic in exactly the way that matters. The diseases the model fails to find are disproportionately the ones the clinician also failed to record.
We propose a procedure that chooses a triage threshold with a statistical guarantee of the form "with 95% confidence, the true miss rate is at most α." It uses the incomplete labels wherever they are reliable and spends a small, targeted expert budget only where they are silent. The core idea is to split the miss rate into two parts. The part on recorded positives can be counted exactly at no cost. The part on unrecorded positives is estimated from a stratified expert audit of the unlabeled cases. The aim is to certify safety for far fewer expert labels than an audit of a fully labeled random sample would need.
Why this gap is real
Two observations in the file motivate this work. In mining DeepLesion, Yan et al. collected 33,688 bookmarked CT images from hospital archives and found that radiologists bookmark only the primary measurable lesion. Smaller co-existing lesions went unrecorded, and a detector that found them was scored as producing false positives. Evaluation on harvested labels is therefore biased, and the direction of the bias hides misses. Separately, the systematic-review screening work of Wallace, Trikalinos et al. achieved 100% yield (no eligible study missed) with roughly 40–50% workload reduction. It got there by having experts label about half the corpus up front, and it offers no rule that certifies the unreviewed half is clean. The file's open-problems section names the lack of distribution-free stopping rules under low or unknown prevalence as a standing limitation of triage filtering.
Existing work at the intersection addresses training, not certification. Asymmetric Loss suppresses gradients from likely-unlabeled positives in multi-label classification. False Positive Rectification cleans over-activated weak segmentation maps. Online asymmetric active learning biases queries toward the minority class to train a better streaming classifier. None of them says anything verifiable about how many positives a deployed filter discards. Our proposal is agnostic to how the model was trained. It treats any of these models as the input to a certification step, and its contribution is the audit design and the bound.
Core Claim
Let $P_w$ and $F_w$ be the number of weakly labeled (recorded) positives and the number of those the filter dismisses. Both are observed exactly. Let $P_u$ and $F_u$ be the number of true positives among unrecorded cases and the number of those dismissed. Both are unknown. The true miss rate at threshold $\tau$ is
$$M(\tau) = \frac{F_w(\tau) + F_u(\tau)}{P_w + P_u}.$$
We estimate $F_u$ and $P_u$ by auditing unrecorded cases, stratified by model score. Audit effort is concentrated in the strata just below and just above $\tau$, where unrecorded positives are most likely to sit. We then form a one-sided upper confidence bound $\hat M^{+}(\tau)$.
To pick the threshold, we scan candidates from least to most aggressive dismissal and stop before the first one whose bound exceeds α. This fixed-order testing keeps the overall guarantee valid even though many thresholds are examined.
The claim is that $M(\tau) \le \alpha$ holds with the stated confidence, and that the number of expert labels required falls roughly in proportion to the fraction of positives the weak labels already cover. The reason is that recorded positives enter the denominator without sampling error, so the audit only has to resolve the unrecorded remainder.
Here is why audit-only certification is expensive at low prevalence. If no misses are observed among $n$ audited positives, the 95% upper bound on the miss rate is about $3/n$, the "rule of three" for binomial proportions. Certifying α = 5% therefore needs about 60 audited positives. At 1% prevalence, that means auditing about 6,000 random cases.
Research Program
The work has three parts.
The first is the estimator and its guarantee. We will derive a stratified upper bound for the ratio $M(\tau)$ under sampling without replacement within strata. We will prove validity when the audit is random within each stratum, with no assumption about how the clinician's labels went missing. This independence from the missingness mechanism is the important property, because missingness here is clearly not random: small lesions are omitted more often.
The second is the audit allocation rule. It chooses how many cases to audit in each score stratum so that the width of $\hat M^{+}$ at the candidate threshold is as small as possible, given a pilot estimate of positive density per stratum. The rule is adaptive in rounds, but each round is randomized within strata so the bound remains valid.
The third is an empirical study of when weak labels actually help. It varies the fraction of positives that are recorded and how strongly the recording depends on lesion size or salience.
The models being certified will include a standard detector or classifier, the same model trained with Asymmetric Loss margin shifting to tolerate unlabeled positives, and, for histology, a standard weakly supervised localizer. This tests whether better training reduces audit cost. We expect it lowers the certified dismissal threshold but cannot replace the audit.
Baselines
The naive reference point is to tune the threshold on the harvested labels themselves. We pick the most aggressive τ whose miss rate on recorded positives is at most α. This is what one would do using DeepLesion-style labels at face value. We will run it on identical data and models, and we expect it to violate the target miss rate whenever missingness correlates with difficulty. Showing the frequency and size of that violation is itself a result.
The strongest principled competitor is audit-only certification. We draw a uniformly random sample of cases, have every one fully expert-labeled, and choose τ with an exact binomial upper bound on the miss rate. This is the textbook way to get a valid guarantee, and it ignores the weak labels. We give it exactly the same expert-label budget and the same fixed-order threshold scan. Beating it means achieving the same validity, meaning the empirical violation rate at or below 5% across repeated runs, with either fewer expert labels for the same dismissed fraction, or a larger dismissed fraction (more workload saved) for the same labels.
The strongest existing triage method in the file is the patient active learning and undersampling approach of Wallace, Trikalinos et al.. It reports 100% yield across Proton Beam, COPD and Micronutrients, workload reduction of "nearly 50% for the Proton Beam and Micronutrients datasets, and by approximately 40% for the COPD dataset," and requires labeling approximately 50% of the corpus. We will rerun it on those three datasets and compare at matched labeling. Beating it means certifying a miss rate of at most α while labeling well under 50% of the corpus, with comparable workload reduction.
We will also run Asymmetric Loss training as a competitor in its own right. Its model will be thresholded naively, to test whether better handling of missing labels during training removes the evaluation bias.
The key ablation isolates the two ingredients. The first variant keeps the decomposition but audits uniformly instead of by score stratum. The second keeps stratified auditing but discards the weak labels, putting every case into the audit population. This separates how much gain comes from the free recorded positives and how much from audit placement.
Why not just…?
Why not just lower the threshold until sensitivity is high enough? Lowering the threshold is the decision our procedure makes. The question is what evidence justifies a particular threshold. The diabetic retinopathy deployment of ARDA in India reached 97.0% sensitivity with PPV of 50.7%, but it had a consensus-adjudicated test set of 4,537 images to verify that. Without such a set, a threshold tuned on harvested labels inherits their blind spots, and the naive baseline above measures by how much. Threshold tuning remains a component of our method, not an alternative to it.
Why not just train with an asymmetric or missing-label-aware loss? Asymmetric Loss shows that margin shifting improves learning under unlabeled positives, reaching 86.6% mAP on MS-COCO. Better ranking does not, however, tell you how many positives remain in the dismissed pool. Its evaluation still relies on the labels it distrusts during training. We include it as both a baseline and a certified model. If its naive miss rate turned out to be reliably within target, that would weaken our motivation, and the experiment reports this directly.
Why not just fully label a random validation set? This is the real threat. It is valid, simple, and exactly our audit-only baseline. Our advantage exists only when weak labels cover a substantial share of positives. When coverage is low, the two methods should coincide, and we will report the coverage level below which our method gives no benefit rather than hide it.
Why not use the dense-segmentation weak-supervision methods directly? Methods such as RECIST co-segmentation (90.3% mean Dice) and AHRNet with regional level sets (92.6% Dice) are evaluated only on lesions that were bookmarked, so they cannot reveal misses of unbookmarked lesions. The co-segmentation paper's finding that CRF refinement sharply cut recall is a reminder that untested post-processing tends to remove true positives quietly.
Proposed Experiments
The first setting uses the three systematic-review screening datasets: Proton Beam with 4,751 citations, COPD with 1,606, and Micronutrients with 4,010. All are fully labeled, so the truth is known. We simulate weak labels by revealing a subset of eligible citations, either at random or biased toward those the model scores highest. This mimics an incomplete seed set of known-relevant studies. We then measure the empirical violation rate over 1,000 resampled audits, the expert labels used, and the workload reduction, comparing against Wallace et al.'s numbers above.
The second setting is CAMELYON16, with 399 whole slides and pixel masks. We simulate clinical-style weak labels by recording only slides with large metastases as positive. This creates the size-dependent missingness that DeepLesion exhibits. We certify a slide-level triage filter built on a weakly supervised localizer. The histology WSOL survey reports average localization PxAP "around 39.01%" against "over 80%" for a supervised U-Net, which shows how much small tumor area these localizers miss. We found no reported result on Lacuna for certified slide-level miss rate on CAMELYON16, so audit-only certification is our reference there.
The third setting is DeepLesion itself, where missingness is real rather than simulated. We use per-study lesion detection with bookmarks as weak positives. The decisive check is a small real radiologist audit of a few hundred randomly stratified studies. For reference, the original multi-class detector reached 64.30% accuracy at IoU 0.5 against recorded bookmarks only, and we found no reported result on Lacuna for true lesion-level miss rate including unbookmarked lesions.
The proposal succeeds if three things hold together: the naive weak-label threshold measurably violates α in the second and third settings; our bound stays valid in all three; and, at weak-label coverage above roughly half of positives, our bound needs substantially fewer expert labels than audit-only certification for the same dismissed fraction. If the last of these fails, the honest conclusion is that a small random fully labeled audit is all a triage deployment needs, and the weak labels are useful for training only.
References
- DeepLesion: automated mining of large-scale lesion annotations and universal lesion detection with deep learning
- Semi-automated screening of biomedical citations for systematic reviews
- Asymmetric Loss For Multi-Label Classification
- FPR: False Positive Rectification for Weakly Supervised Semantic Segmentation
- Online Adaptive Asymmetric Active Learning With Limited Budgets
- Performance of a Deep Learning Diabetic Retinopathy Algorithm in India
- Weakly Supervised Lesion Co-Segmentation on Ct Scans
- Weakly-Supervised Universal Lesion Segmentation with Regional Level Set Loss
- Deep weakly-supervised learning methods for classification and localization in histology images: a survey