A diagnostic model ends in a cut-off: above some score, call the disease. Clinical medicine keeps learning that one cut-off for everyone fails. Prostate MRI scores miss more cancers in Black men, PSA reference ranges differ for older Taiwanese men, and BMI cut-offs differ for Asian cohorts. Machine learning has learned a parallel lesson about rare classes. A model trained on data where a few diseases dominate under-calls the rare ones, and pushing rare-class decisions outward costs accuracy on the common ones, the so-called head-tail seesaw. The two problems meet in the case that matters most and is least studied: a rare disease in an under-represented patient group. A threshold must be set for each disease-by-group cell, and the worst cells may hold only a handful of labeled positives.
We propose to treat both corrections as the same mathematical object and to split each cell's threshold into two parts. The first part is the shift that would occur if only disease prevalence differed between groups. Bayes' rule gives it in closed form, and it can be estimated from a group's unlabeled cases. The second part is a residual shift: the score itself means something different in that group, as in the lower MRI sensitivity in Black men. Only this residual needs scarce labels, and we estimate it by borrowing strength across diseases and groups. If the split is right, the compound-tail cells get usable thresholds with far fewer labels than per-cell tuning needs, and the gain in those cells does not come at the expense of common diseases or majority groups.
Core Claim
Long-tail "logit adjustment" (subtracting the log training frequency of each class so rare classes are not drowned out) and population-specific thresholds (adding the log prevalence of each class in a group) are both corrections to a class prior. They differ only in which prior they target. Let $s_k(x)$ be the log-odds a pooled, calibrated model assigns to disease $k$ for case $x$. Let $\pi_k$ be the pooled prevalence of $k$ and $\pi_{k\mid a}$ its prevalence in group $a$. We then act on
$$ s_k(x) + \log\frac{\pi_{k\mid a}}{\pi_k} + \delta_{k,a} \;>\; \log c_k , $$
where $c_k$ is the clinical harm-to-benefit ratio that sets the decision point for disease $k$, and $\delta_{k,a}$ is the residual shift. The middle term is exact when the appearance of disease given its presence is the same across groups (pure prevalence shift, by Bayes' rule). The residual $\delta_{k,a}$ captures everything else.
The claim has three parts, and each can be tested. The prevalence term can be estimated without labels: the standard expectation-maximization procedure re-estimates class priors on a group's unlabeled cases from a calibrated model. That procedure transfers to the tail better than label-based tuning. The residual is small and low-rank, meaning group effects and disease effects interact through a few shared factors. Shrinking it toward that structure beats per-cell estimation in sparse cells. Finally, because the decision rule follows from calibrated probabilities and stated clinical costs rather than parity constraints, gains in tail cells should not reduce net benefit in head cells. That would be a seesaw that is bounded rather than traded away.
This position builds directly on Net benefit, calibration, threshold selection, and training objectives for algorithmic fairness in healthcare. That study of ASCVD risk in 5.8 million patients found that equalized-odds training penalties broke calibration and collapsed net benefit at the 7.5% statin cut-off. It concluded that equity should come from calibrated models plus population-specific threshold choice. That paper supplies our design constraint, but it worked with one common binary outcome and millions of records. It offers no way to set thresholds when a cell has ten positives, and that is our problem.
Relation to Prior Work
Three lines of work motivate the proposal. Clinical audits document population threshold failures but fix them by hand-binning a single biomarker. Negative Predictive Value of a Prostate MRI in Black Men found PIRADS ≥3 sensitivity of 90.7% versus 96.3% and proposed lowering the PSA-density deferral cut-off to 0.09 for PIRADS 1–2. Similar recalibrations appear for PSA in Ethnic differences in the age-related distribution of serum prostate-specific antigen values and for BMI in He et al.. The prostate case is informative for our split: a sensitivity difference at a fixed score is a residual (likelihood) shift, not a prevalence shift. So the residual term cannot be dropped a priori.
Long-tail methods supply mechanisms but ignore patient groups. The Devil Is in Classification (SimCal) showed the seesaw lives mainly in the classifier rather than the features: rare-class mask AP on LVIS rose from 2.7 to 16.4 by retraining only the head on a frozen backbone. This licenses our choice to keep one pooled backbone and move all population-specific work into the decision layer. Two Fists, One Heart argues that head and tail errors are competing objectives on a Pareto frontier. We accept this, and our evaluation therefore reports frontiers rather than single points. MONICA benchmarks over 30 long-tail methods on medical images but tracks no demographics, which leaves the compound tail unmeasured.
Post-hoc fairness methods supply the closest competitor and a constraint. Group-Aware Threshold Adaptation for Fair Classification (GSTAR) learns per-group thresholds after training under a fairness penalty. It was evaluated on balanced binary tasks and assumes enough validation data per group. Conformal Prediction Meets Long-tail Classification shows that marginal guarantees hide tail under-coverage, but it partitions only by class frequency. Fairness Under Demographic Scarce Regime shows that hard imputation of missing group labels hurts and that uncertainty-filtering helps. We reuse this when group labels are partly missing. MIMIC-IF constrains interpretation: it found shorter ventilation durations for Black and Hispanic patients. Some apparent "prevalence" differences may therefore reflect care access rather than biology, so we report the prevalence term as an empirical quantity and not as a physiological fact.
Research Program
We would build a post-hoc decision layer on top of any pooled, long-tail-trained backbone. The first step is calibration. Starting from a backbone trained with a strong long-tail loss, we fit class-wise temperature scaling on pooled validation data so that $s_k$ is a calibrated log-odds. This addresses the concern from the GSTAR gap analysis that tail scores are squashed toward zero.
The second step is the prevalence term. For each group, we run prior re-estimation by expectation-maximization on that group's unlabeled cases, including test-time cases where deployment allows. We place a weak hierarchical prior on log prevalence ratios so that a tiny group's estimate for a very rare disease is pulled toward the pooled value. This estimate is noisy for rare classes in small groups, and we measure that noise directly.
The third step is the residual. We parameterize $\delta_{k,a} = u_a^\top v_k + \epsilon_{k,a}$ with rank one or two, and put a Gaussian penalty on $\epsilon$. We fit it by penalized likelihood on the labeled validation cases, with the penalty strength chosen by cross-validation across groups. When group labels are missing, we weight each case by its predicted group probability and drop high-uncertainty cases, following FairDSR.
The analytical deliverable is a variance account. For each cell, we compare the label-count requirement of per-cell threshold tuning with that of the decomposed estimator, and we locate the cell size below which per-cell tuning becomes worse than a global threshold.
Baselines
The do-nothing reference is the pooled backbone with a single global threshold per disease, chosen at the clinical cost ratio. Every method must beat it in the worst compound-tail cell without lowering net benefit in head cells.
The strongest long-tail competitor is decoupled classifier retraining in the SimCal style, combined with global logit adjustment on a backbone trained with GCL. The file reports GCL reaching 71.33% tail accuracy on ISIC-2019-LT versus 12.67% for ERM (MONICA). This baseline fixes the class axis as well as current methods do, so any further gain must come from the group axis.
The strongest population-threshold competitor is GSTAR. It reports cutting the CelebA false-positive-rate gap from 0.235 to 0.014 at a 2.2% accuracy cost (GSTAR). We extend it one-versus-rest to each disease, and run it once with its fairness penalty and once with only a net-benefit objective so that the penalty does not confound the comparison. We also add plain per-cell threshold tuning on validation labels.
All methods share the same frozen backbone, the same validation split, and the same hyperparameter search budget. Beating a baseline means one of two things. Either the method gets higher net benefit in the bottom decile of cells by labeled-positive count, with bootstrap intervals that exclude zero, and no statistically detectable loss in head cells. Or it reaches the same tail net benefit with fewer labeled tail positives.
Two ablations isolate the key ingredient. One uses only the prevalence term. The other uses only the shrunk residual, which is hierarchical shrinkage of thresholds with no label-free component.
Why not just…?
Why not just tune a separate threshold per group and disease on validation data? In the compound tail this means setting a cut-off from a few positives, and the ROC point it lands on has huge variance. The clinical literature names data scarcity in tail subgroups as the main open problem. Our variance account measures exactly where per-cell tuning crosses below the global threshold, and the label-budget sweep below shows it directly.
Why not just add group membership as an input and let the network learn group-specific behavior? This is a serious alternative, because a network could learn both prevalence and residual shifts. But it must learn the compound-tail interaction from the same scarce labeled cells, and SimCal's evidence suggests such interaction errors sit in the classifier rather than the features. We include a backbone trained with group as an input, with and without our decision layer on top. If it matches us in tail cells, the claim that a post-hoc split is needed fails.
Why not just shrink per-cell thresholds hierarchically, without the prevalence machinery? This is the most dangerous alternative, and it is our residual-only ablation. Our bet is that much of the group difference is prevalence, which unlabeled data can identify, so the shrinkage has less work to do. If residual-only shrinkage matches the full method on both real data and semi-synthetic data with a known prevalence shift, we will report that the prevalence term adds nothing in practice and recommend the simpler estimator.
Why not just retrain with fairness penalties or stronger re-balancing losses? Net benefit, calibration, threshold selection, and training objectives showed equalized-odds training destroys the calibration that clinical thresholds depend on. Fairness Improves Learning from Noisily Labeled Long-Tailed Data shows some re-balancing losses drive minority-subpopulation accuracy toward zero under noise. We include one in-processing variant for completeness and report its calibration error per cell.
Proposed Experiments
The first experiment is semi-synthetic and has ground truth. On CIFAR-100-LT we create synthetic "groups" by resampling classes with known prior shifts, and separately by applying known class-specific feature perturbations that create likelihood shift. We vary group size across orders of magnitude. This tells us whether the method recovers the true prevalence term, whether it detects residual shift when present, and where each competitor breaks.
The main real-data experiment uses Fitzpatrick17k-C, the cleaned split from Investigating the Quality of DermaMNIST and Fitzpatrick17k. It has a long-tailed set of skin conditions and skin-tone groups, which directly yields compound-tail cells. No reported result was found on Lacuna for any classifier on Fitzpatrick17k-C, so we establish the baselines ourselves. We repeat the experiment on MONICA datasets wherever patient age and sex fields are available, using age and sex strata as groups. We also run a binary tabular check on MIMIC-IV in-hospital mortality by ethnicity, where the file reports an IMV-LSTM AUROC of 0.955 (MIMIC-IF). On MIMIC-IV only the prevalence-versus-residual split is at stake, not the class tail.
Metrics are reported per cell. They are net benefit at each disease's cost ratio, sensitivity at fixed specificity, and per-cell calibration error, all with bootstrap intervals. We summarize them by the worst-cell value and by head-versus-tail net-benefit frontiers.
The decisive checks are three. The first is a label-budget sweep: subsample tail-cell validation positives from all down to five, and plot net benefit against labels used for every method. The method must dominate per-cell tuning and GSTAR at small budgets. The second is the two ablations: the full method must beat residual-only shrinkage in cells where the semi-synthetic or estimated prevalence shift is large. The third is a head-cost check: head-cell net benefit must stay within its bootstrap interval of the do-nothing baseline.
If the first check fails, the premise that unlabeled data can substitute for scarce labels is wrong. If the second fails, the decomposition is unnecessary and we recommend the simpler shrinkage estimator.
References
- Net benefit, calibration, threshold selection, and training objectives for algorithmic fairness in healthcare
- Negative Predictive Value of a Prostate MRI in Black Men: Implications for Biopsy Decision-Making
- Ethnic differences in the age-related distribution of serum prostate-specific antigen values: A study in a Taiwanese male population
- Prevalence of overweight and obesity in 15.8 million men aged 15–49 years in rural China from 2010 to 2014
- The Devil Is in Classification: A Simple Framework for Long-Tail Instance Segmentation
- Two Fists, One Heart: Multi-Objective Optimization Based Strategy Fusion for Long-tailed Learning
- MONICA: Benchmarking on Long-tailed Medical Image Classification
- Group-Aware Threshold Adaptation for Fair Classification
- Conformal Prediction Meets Long-tail Classification
- Fairness Under Demographic Scarce Regime
- MIMIC-IF: Interpretability and Fairness Evaluation of Deep Learning Models on MIMIC-IV Dataset
- Robust Learning from Noisily Labeled Long-Tailed Data via Fairness Regularizer
- Investigating the Quality of DermaMNIST and Fitzpatrick17k Dermatological Image Datasets