Medical staging systems are rules. A thyroid nodule's TI-RADS risk level, which runs from TR1 to TR5, is computed from five scored ultrasound features. A diabetic retinopathy grade is computed from counts and locations of retinal lesions. A CheXpert training label depends on whether a radiologist's "uncertain" mention is mapped to positive or to negative. Committees revise these rules often: new editions, regional variants, new labeling policies.
Most diagnostic models learn the stage directly from pixels, so the rule in force at labeling time gets baked into their weights. When the rule changes, such a model is wrong in ways that are hard to see. It is also wrong unevenly across patients. Patient groups differ in where their findings sit relative to the moved boundary, and in how accurately the model detects those findings in the first place.
We propose to treat a diagnostic criterion as an explicit, versioned program. The program runs on findings: the fine-grained observations a model can detect, such as margin type, echogenic foci, microaneurysm counts, or an "uncertain effusion" mention. A model that outputs findings can be re-staged under any criteria version by swapping the program, with no retraining. More importantly, this split makes the uneven harm predictable. The change in stage accuracy each subgroup will suffer under a new rule can be computed before deployment, from two quantities measured under the old rule: how often each subgroup's findings fall near the boundary the new rule moved, and how often the model misreads exactly those findings for that subgroup.
Core Claim
The claim has two parts. First, a findings model paired with a versioned rule matches or beats direct stage classifiers fine-tuned on a small set of new-criteria labels, and it needs no new-criteria labels at all. Second, and this is what we think is new, the split yields a pre-deployment disparity forecast that a direct stage classifier cannot produce.
Write $f$ for the vector of findings, $R_v$ for the staging rule of version $v$, and $\hat f$ for the model's predicted findings. For subgroup $g$, the forecast of stage error under a new version $v'$ is
$$ \widehat{\mathrm{Err}}_g(v') = \sum_k \Pr\big(R_{v'}(\hat f) \neq R_{v'}(f) \text{ via finding } k \mid g\big) \approx \sum_k b_k^{v'}(g)\, e_k(g). $$
Here $e_k(g)$ is the model's error rate on finding $k$ within subgroup $g$, measured on old-version data. The term $b_k^{v'}(g)$ is the fraction of subgroup $g$'s cases for which flipping finding $k$ changes the stage under $R_{v'}$. It can be computed from the old data's ground-truth findings alone, because the rule is known.
The first-order form assumes finding errors rarely co-occur on a case. We test that assumption rather than presume it. The decisive prediction is that the forecast ranks subgroups by their realized post-revision error, and by their realized gap, better than the obvious proxy of "current subgroup error carries over."
Where This Sits in Prior Work
Work that motivates the proposal.
- The AJCC 8th-edition thyroid study (Clinical study of ultrasonic evaluation of T/N staging of differentiated thyroid carcinoma using AJCC 8th staging criteria) shows the harm concentrates at moved boundaries. It evaluated human ultrasonographers on 1,206 patients and found high T-stage accuracy overall, but only 29.5% sensitivity for the redefined T3b category and 24.3% positive predictive value for N1b. It did not ask whether those failures fell unevenly on its 338 male and 868 female patients, and it involved no learned model.
- The ICD transition study (Transitions between versions of the International Classification of Diseases and chronic disease prevalence estimates from administrative health data: a population-based study) and the eating-disorder meta-analysis (An update on the prevalence of eating disorders in the general population: a systematic review and meta-analysis) show criteria revisions shift cohort definitions substantially: 75% of tracked chronic conditions in the former, a DSM-5 surge in the latter. Both work at the level of population statistics, not model behavior.
- The causal account of imaging bias (A causal perspective on dataset bias in machine learning for medical imaging) names annotation disparities, including those from evolving guidelines, as a distinct bias pathway. It argues no generic mitigation fits all pathways. That supports our choice to model the annotation pathway explicitly. Its analysis is conceptual and binary, and we make it empirical and multi-level.
Work that supplies the method.
- CMCNet (CMCNet: Aligning Ultrasound Image Embeddings with Textual TI-RADS Representations for Fine-Grained Thyroid Classification) aligns ultrasound embeddings to text templates of TI-RADS feature scores. It reaches 75.17% balanced accuracy and 91.78% weighted F1 on five-level staging of the 600-nodule STN set. A text-only ModernBERT reaches 98% accuracy from the feature descriptions alone. That gap tells us staging error is almost entirely a findings-reading problem, which is the premise of the split.
- FairerOPTH (Fairer AI in ophthalmology via implicit fairness learning for mitigating sexism and ageism) routes disease prediction through a branch that detects 67 lesion features. It reports 92.0% mAP on OculoScope with a narrowed sex gap (91.8% female versus 90.8% male).
- The FGADR benchmark (A Benchmark for Studying Diabetic Retinopathy: Segmentation, Grading, and Transferability) shows that lesion masks improve grading kappa by up to 8.04%.
All three use findings to improve or de-bias a single fixed label set. None changes the label rule, and none uses findings to forecast subgroup harm.
Work that constrains the design.
- Subgroups Matter (Subgroups Matter for Robust Bias Mitigation) shows that mitigating on demographic groups instead of the true spurious attribute does worse than plain training in 10 of 16 settings, by up to 0.08 AUC. Mitigation helps by more than 0.07 AUC only when groups match the actual mechanism. We read this as a warning against defaulting to demographic reweighting, and as support for locating harm at specific findings near specific boundaries.
- The Rad-DINO adapter study (Subgroup performance analysis of adaptation strategies for chest X-ray foundation models) shows a more expressive head can raise overall AUROC (pneumonia 0.791 to 0.855) while widening race gaps. Overall accuracy gains therefore cannot stand in for subgroup safety, so we report both.
Research Program
We build a findings model on a frozen imaging foundation model. It has one small head per finding, trained on old-version annotations. A rule library encodes each criteria version as executable code, and an auditing tool computes the boundary-sensitivity terms $b_k^{v'}(g)$ and the forecast. We use three testbeds, chosen because each has finding-level labels and a believable rule revision.
Thyroid ultrasound (STN). The real revision is between TI-RADS variants. The file names ACR TI-RADS, EU-TIRADS and C-TI-RADS, which combine overlapping ultrasound features into different risk levels. We will encode each variant as far as STN's ACR-style feature labels allow. Where a variant needs a feature STN lacks, we mark the mapping as approximate and report that variant separately. STN has no demographic fields, so here we test only re-staging accuracy, with ultrasound view as the one available stratum.
Chest radiographs (CheXpert). The rule revision is the labeling policy itself (Structured dataset documentation: a datasheet for CheXpert). Each observation is negative, uncertain, positive or unmentioned, and mapping uncertain to positive versus negative are two criteria versions over a three-level finding. The data span 2002 to 2017 with sex and age attributes. We can also reuse the pacemaker annotation from Subgroups Matter, which gives one case with a known spurious attribute.
Retinal grading (FGADR and OculoScope). Grades are rules over lesion types and counts. We define revised rules that move a single boundary, such as the count threshold separating moderate from severe disease. We run these on FGADR's lesion masks, and on OculoScope's 67 features with age and sex strata if access is obtained. These revisions are semi-synthetic, the same strategy Subgroups Matter uses for spurious attributes, and we label them as such. ODIR-5K and Kaggle-EyePACS provide a cross-site check of whether finding-level errors measured at one site still forecast harm at another.
Baselines
Naive reference. The do-nothing point is a direct stage classifier trained under the old rule and deployed unchanged after the revision. Its stage accuracy, scored against new-rule labels, shows how much damage the revision causes. Every other method must at least recover from it.
Fine-tuned stage classifier. This is a direct classifier trained on the old rule and then fine-tuned on a small budget of new-rule labels: 50, 100 and 200 cases. It is the realistic competitor, because hospitals adopting a new edition usually relabel a little data and retrain. It shares our frozen backbone, training compute and hyperparameter search budget. Our method beats it if it matches or exceeds it at every label budget while using zero new-rule labels.
Re-templated CMCNet. This is the strongest existing method on the thyroid testbed. Its text anchors are feature descriptions, so they can be regenerated under a new rule. It is the most direct rival: it also exploits structure in the criteria, only implicitly through text alignment. CMCNet reports 75.17% balanced accuracy and 91.78% weighted F1 on STN under the original ACR rule, and that is the number our findings model must reproduce before the comparison under revised rules means anything. We run it with the same USFM backbone, the same data splits, and anchors regenerated under each variant. Beating it means higher balanced accuracy under the revised variants at equal or lower variance.
Lesion-guided models. On retina, the competitors are FairerOPTH's pathology-guided attention and the FGADR lesion-mask grading approach, with FairerOPTH's 92.0% mAP on OculoScope as its reported reference. On CheXpert, the competitor is the causal-subgroup mitigation from Subgroups Matter, which reports gains of more than 0.07 AUC. We use these to judge whether our split narrows post-revision gaps more than mitigation does, not only overall accuracy.
Forecast baselines. The forecast is compared against two simpler predictors of post-revision subgroup error. One is each subgroup's current stage error, carried over. The other is each subgroup's prevalence near any stage boundary, ignoring finding-level error.
Ablations. One ablation replaces the explicit rule with a learned map from predicted findings to stage. This isolates whether the explicit rule, rather than the findings bottleneck alone, is what enables label-free re-staging. A second ablation drops the $e_k(g)$ term from the forecast, to show that subgroup-specific finding error matters beyond boundary geometry.
Why not just…?
Why not just apply the new rule to the old finding annotations and retrain a direct classifier? This is the real threat. Wherever old data carry finding labels, which is exactly our setting, relabeling is free, and a direct classifier trained on relabeled data may match our re-staging accuracy. We will run it on every testbed. If it matches, our accuracy claim shrinks to convenience: no retraining per version, and one model serving many versions at once. The contribution then rests on the disparity forecast, which a retrained direct classifier does not provide because it exposes no finding-level errors to audit. We also compare the subgroup gaps of the two approaches, since the adapter study shows that matching overall accuracy says nothing about matching gaps.
Why not just treat the criteria era as another domain shift and use domain adaptation or an era indicator? An era indicator needs labeled data from the new era, which is exactly what is scarce at revision time. It also learns the new rule statistically instead of being told it. The fine-tuned stage classifier baseline approximates this family at fixed label budgets. The STN results, where the rule is known exactly, will show whether learning the rule from 100 cases ever matches executing it.
Why not just reweight by demographic group with group-robust training after the revision? Subgroups Matter found that reweighting on demographic attributes rather than the true mechanism falls below plain training in most of its settings. Our hypothesis is that the mechanism after a revision is finding-specific boundary error. If so, demographic reweighting should under-perform targeted correction of the flagged findings. We test this directly on CheXpert, where both group types are available.
Why not just prompt a general vision-language model with the revised criteria? CMCNet reports beating LLaVA-Med on STN, and Zihan Li et al. report general models such as GPT-4V falling behind task-specific adapters on fine-grained ophthalmic diagnosis. No reported result found on Lacuna for prompted vision-language models under a criteria revision specifically. We will include a prompted model on STN as a reference, without expecting it to be competitive.
Decisive Checks
The first check is re-staging. On STN, the findings model must reproduce CMCNet's 75.17% balanced accuracy under the ACR rule. Under each other TI-RADS variant, it must then match or beat the fine-tuned and re-templated baselines without new labels.
The second check is the forecast, and it is the one that decides whether the idea is worth anything. On CheXpert's uncertainty policies and the retinal threshold revisions, we compute $\widehat{\mathrm{Err}}_g$ from old-rule data and compare it with realized new-rule error by subgroup. We report rank correlation across sex and age strata and absolute error of the predicted gap, against the two forecast baselines. If the forecast does not beat "current error carries over," the claim fails.
The third check is cross-site. We test whether finding-level error rates measured at one site still forecast post-revision harm at another, moving from FGADR to ODIR-5K and Kaggle-EyePACS. The CcRCC analysis (Critical insights on statistical consistency, generalizability, and feature optimization in AI-driven CcRCC prognostic modeling) warns that baseline presentation shifts across ethnic groups, which could break the forecast precisely where it is most needed. We will report that failure if it occurs.
References
- Clinical study of ultrasonic evaluation of T/N staging of differentiated thyroid carcinoma using AJCC 8th staging criteria
- Transitions between versions of the International Classification of Diseases and chronic disease prevalence estimates from administrative health data: a population-based study
- An update on the prevalence of eating disorders in the general population: a systematic review and meta-analysis
- A causal perspective on dataset bias in machine learning for medical imaging
- CMCNet: Aligning Ultrasound Image Embeddings with Textual TI-RADS Representations for Fine-Grained Thyroid Classification
- Fairer AI in ophthalmology via implicit fairness learning for mitigating sexism and ageism
- A Benchmark for Studying Diabetic Retinopathy: Segmentation, Grading, and Transferability
- Subgroups Matter for Robust Bias Mitigation
- Subgroup performance analysis of adaptation strategies for chest X-ray foundation models
- Structured dataset documentation: a datasheet for CheXpert
- Critical insights on statistical consistency, generalizability, and feature optimization in AI-driven CcRCC prognostic modeling