TL;DR
Standard evaluation metrics like macro-mAP often hide severe underdiagnosis of rare diseases within specific demographic subgroups when machine learning models are deployed in clinical workflows. This paper demonstrates that the score-to-decision thresholding step is a massive, underutilized lever for fairness: by implementing tail-aware thresholding, researchers significantly reduced missed positives for vulnerable subgroups without retraining the underlying model.
What Problem Does This Paper Address?
When deploying computer vision models for chest X-ray (CXR) triage, continuous risk scores must eventually be converted into binary decisions (e.g., "surface for immediate radiologist review" vs. "place in standard queue"). However, medical datasets are notoriously long-tailed; rare conditions have very few positive examples, and intersectional subgroups (e.g., a rare condition in a specific age or racial group) have fewer still.
The core gap the authors address is that standard performance metrics—like macro-mean Average Precision (macro-mAP)—evaluate the ranking of scores but completely mask what happens after a decision threshold is applied. A model can achieve an acceptable macro-mAP while simultaneously missing the vast majority of rare-positive cases within a vulnerable demographic subgroup. The authors formalize this pre-deployment fairness issue as thresholded class-subgroup underdiagnosis.
Methods
Rather than proposing a single new algorithm, the authors construct a diagnostic ladder consisting of six model variants (M1–M6) to isolate exactly where missed positives enter the clinical pipeline. All models share a standard ConvNeXt-Tiny backbone but vary in their loss functions, weighting schemes, or thresholding rules:
- M1 (BCE): Standard Binary Cross-Entropy baseline.
- M2 (ASL): Asymmetric Loss to mitigate class-level long-tail imbalances.
- M3 (CB-ASL): Adds effective-number class weighting to ASL.
- M4 (GroupDRO): A distributionally robust optimization baseline targeting group fairness.
- M5 (GT-ASL): Group-Tail ASL, which multiplies ASL by a clipped class-subgroup weight based on training counts.
- M6 (+TA): Reuses the exact same model scores as M5 but replaces standard $F_1$-optimal thresholds with Tail-Aware (TA) thresholding. For tail classes, it shifts the operating point to maximize worst-group recall, subject to a strict minimum precision floor ($\tau = 0.05$).
By comparing M5 and M6, the authors isolate the impact of the score-to-decision thresholding step from the model training process itself.
Datasets
The approach was audited across two major chest X-ray cohorts:
- VinDr-CXR: A dataset containing 15K training images and a 1,500-image test split with 15 labels (5 defined as tail classes). Subgroup metadata is limited to sex and age group extracted from DICOM headers.
- MIMIC-CXR/CXR-LT: A much larger dataset utilizing a 78,946-image test split with 40 labels (13 tail classes). This dataset links to Electronic Health Records (EHR), enabling richer subgroup analyses across sex, age, race, and insurance status.
Key Insights and Ideas
- The Threshold is the Action Boundary: The transition from M5 to M6 proves that changing the decision rule for rare classes alters the False Negative Rate (FNR) far more effectively than complex, class-level long-tail training objectives.
- Group Robustness $\neq$ Rare Subgroup Safety: Optimizing for aggregate group fairness (like GroupDRO) does not automatically protect rare subgroup cells. The audit showed that aggregate group robustness and thresholded rare-positive recall are entirely distinct dimensions.
- Fairness is a Workflow Policy Choice: Lowering thresholds to catch more rare diseases inherently increases the "alert burden" (the number of cases flagged for review). The choice of where to set the precision floor ($\tau$) is a clinical workflow policy decision, not a static mathematical constant.
Key Results
The transition to tail-aware thresholding yielded dramatic reductions in missed cases across both datasets:
- VinDr-CXR: Moving from the baseline (M1) to tail-aware thresholding (M6) reduced the overall tail FNR from 0.665 to 0.269. More importantly, it slashed the sex worst-group FNR from 0.705 to 0.157, and the age worst-group FNR from 0.822 to 0.133.
- MIMIC-CXR/CXR-LT: The same tail-aware adjustment on M5 scores reduced the tail FNR from 0.866 to 0.741 and consistently lowered the worst-group FNR across all four demographic axes (sex, age, race, insurance).
The absolute reduction in missed positives per 100 true positives across these cohorts is visualized below:
However, this safety benefit comes at a clear operational cost. As shown in the operating-point trade-off curves, catching more missed positives requires flagging significantly more studies for human review:
Limitations and Future Work
The authors emphasize that their framework is a diagnostic measurement tool rather than a universal cure for algorithmic bias. Key limitations include:
- Low Resolution in the Extreme Tail: In the VinDr-CXR test set, some tail classes contain as few as 3 to 19 positive cases, making disease-level conclusions statistically noisy.
- Missing Metadata: Real-world clinical data is messy; for instance, DICOM headers in VinDr lacked informative sex data in 51% of cases and age data in 69% of cases.
- Proxy Attributes: EHR-derived variables like race and insurance status function as imperfect administrative and social proxies rather than direct causal biological attributes.
Future work will need to explore how these thresholding policies perform under real-world clinical deployment constraints and across varying hospital architectures.
Summary updated