Medical imaging models often reach expert-level accuracy overall while missing disease more often in some patient groups, and most fixes need every training image tagged with the patient's race or sex. Weight averaging is a training trick that needs no such tags. Instead of keeping the last or best checkpoint, it keeps the mean of the weights the network passes through late in training. This note asks a narrow question that nobody has tested. When does averaging shrink the gap between the best- and worst-served patient groups, and when does it widen it? We predict the answer from a single measurable quantity: how sharply each group's loss rises when the weights move in the directions the training trajectory actually wanders.
The second idea is about how we define the groups. Fairness work in imaging treats race and sex as fixed categories. Genetics shows that the anatomy behind many of these gaps varies continuously and has a measurable genetic basis. A generative phenotyping study on UK Biobank retinas found that an apparent vascular signal at the OCA2-HERC2 pigmentation locus was really melanin changing how visible the vessels are (Bridging the genotype-phenotype gap with generative artificial intelligence). We therefore audit retinal models along genotype-derived axes, such as ancestry components and pigmentation genotype, as well as along self-reported labels. This lets us ask whether the gaps, and whatever averaging does to them, follow the biology or the social category. The genotypes are used only for auditing and are never used in training.
Core Claim
Take $K$ checkpoints $w_1,\dots,w_K$ with mean $\bar w$ and spread covariance $\Sigma = \frac{1}{K}\sum_i (w_i-\bar w)(w_i-\bar w)^\top$. Let $L_g$ be the loss on patient group $g$ and $H_g$ its Hessian (the matrix of second derivatives, i.e. the curvature) at $\bar w$. A second-order expansion around $\bar w$ then gives
$$\frac{1}{K}\sum_{i=1}^K L_g(w_i) - L_g(\bar w) \approx \tfrac{1}{2}\,\mathrm{tr}(H_g \Sigma).$$
The first-order term cancels exactly, because the deviations from the mean sum to zero. The left side is how much a group gains from averaging compared with a typical checkpoint.
Subtracting this expression for two groups gives a falsifiable prediction. Averaging narrows the worst-group gap by about $\tfrac12[\mathrm{tr}(H_{\text{worst}}\Sigma) - \mathrm{tr}(H_{\text{best}}\Sigma)]$. So it helps exactly when the underserved group is sharper along the trajectory's wandering directions, and it does nothing or hurts otherwise.
This connects two results that appear to conflict:
- Averaging helps under distribution shift. Stochastic weight averaging (SWA) and simple moving averages land in flat, centered regions of the loss and generalize better across domains (Improving Generalization of Pre-trained Language Models via Stochastic Weight Averaging; Ensemble of Averages).
- Merging hurts some groups. Merging separately trained models amplifies unfairness, because subgroups with sharp loss surfaces suffer most when the weights are displaced (FairMerging).
Our claim is that both are the same second-order effect at different displacement sizes. Averaging along one trajectory has small spread $\Sigma$, so the expansion holds and sharp groups gain. Merging distinct fine-tunes, as in Model Stock (Model Stock) or uniform soups, has large spread. The averaged point can then fall off a minority group's basin entirely, which reproduces FairMerging's amplification.
Nothing in the file tests this in medical imaging. Nothing tests whether the sharp directions line up with genetically grounded anatomical variation.
Research Program
The work has three parts.
Measurement study. We train standard classifiers on public fairness benchmarks and save dense late-training trajectories. We compare four estimators that use no demographic labels:
- ERM: ordinary training that minimizes average loss, keeping the best or last checkpoint.
- Exponential moving average of weights (The Unusual Effectiveness of Averaging in GAN Training).
- SWA over a constant-learning-rate tail.
- Model Stock across two independent fine-tunes from the same pretrained anchor.
For each group we estimate $\mathrm{tr}(H_g\Sigma)$ directly. We project $\Sigma$ onto its top principal directions and estimate curvature along those directions with Hessian-vector products on held-out data from that group. We then test whether the predicted gap change matches the observed change, as a function of displacement size.
Genetic audit. This part uses UK Biobank retinal fundus images with linked genotypes. Diagnostic labels for glaucoma and diabetic eye disease would come from linked health records; how reliable those labels are is an open data question we would settle first. We define continuous audit axes:
- ancestry principal components;
- genotype at OCA2-HERC2;
- an image-derived pigmentation trait in the style of AIPheno.
The analysis asks two things. First, after conditioning on self-reported ethnicity, do false negative rates and per-patient curvature still vary along the pigmentation axis? Second, does averaging's gain follow that axis?
For a parallel test in another organ, we use dermatology. Skin tone is the sensitive attribute in Fitzpatrick-17k (FairAdaBN). If pigmentation is a shared driver of sharpness, the curvature pattern should replicate across organs.
Label-free averaging rule. The equation says gains concentrate on samples whose loss swings most across the trajectory. We therefore weight checkpoints toward those that perform well on the top decile of samples with the highest loss variance across checkpoints. No demographic tags are used. This is the single ingredient our ablation isolates.
Baselines
Naive reference. The naive reference is ERM with the checkpoint chosen by best validation AUC. Every claim is measured against it.
Plain averaging. The baseline that most threatens the proposal is plain uniform SWA. If it already closes gaps, our selection rule adds nothing, and the contribution reduces to the mechanistic explanation. That would still be worth reporting.
Label-using methods. These are the strongest existing competitors for the question "how small can the gap get":
- Fair Identity Normalization. It standardizes features group by group using patient identity. It reports equity-scaled AUC (overall AUC penalized by between-group spread) rising from 0.77 to 0.81, and Black-patient AUC rising from 0.77 to 0.82, on Massachusetts Eye and Ear glaucoma data (Equitable artificial intelligence for glaucoma screening with fair identity normalization).
- FairAdaBN. It gives each group its own batch-normalization parameters. It reports a fairness-accuracy trade-off efficiency of $48.79\times10^{-2}$ on Fitzpatrick-17k, against $22.63\times10^{-2}$ for the next baseline (FairAdaBN).
- Fair-FedMOE. It reports a worst-group-scaled AUC more than 7% above standard one-shot aggregation for glaucoma on its retinal benchmarks (Fair-FedMOE). No reported absolute result for Harvard-FairVLMed was found on Lacuna.
- FairMerging. It cuts the fairness gap from 1.49 to 0.42 on ViT-B/32 (FairMerging). That result is on natural images, so we would re-run it on our tasks.
Data rebalancing. We also include group rebalancing, which needs labels.
Fairness of comparison. All methods share the same backbone, splits and pretrained initialization. Every method gets the same total number of training runs. Model Stock and the ensembles use two runs; SWA and EMA use one. Each method also gets the same hyperparameter sweep budget. Label-using methods receive the labels; ours does not. We count a result as beating a label-free baseline if it achieves a statistically significant gain in worst-group AUC across five seeds without losing overall AUC. Against label-using methods, matching within seed variance counts as success.
Why not just…?
Why not just rebalance the training data? Balancing by age and sex in brain MRI reconstruction left a significant gap in reconstruction quality and lowered quality for the advantaged group without helping the underserved one (Unveiling Fairness Biases in Deep Learning-Based Brain MRI Reconstruction). FIN also beat oversampling on glaucoma. Rebalancing is a baseline here. Our curvature measurements can also test the explanation that rebalancing changes how often groups are seen but not how sharp their loss is.
Why not just ensemble the predictions of the checkpoints? This is a real threat. Averaging outputs avoids the risk of the averaged weights leaving a minority basin, and it may be fairer than any weight average. The cost is $K$ forward passes at inference (Ensemble of Averages). We run output ensembles of exactly the same checkpoints. If they close the gap and weight averaging does not, that difference is itself evidence for the off-basin regime the equation predicts. We would report it rather than hide it.
Why not just use self-reported race and skip the genetics? Self-report is cheaper and is what clinicians record. The genetic audit is justified only if it explains variance that self-report misses. The decisive regression asks whether pigmentation genotype predicts per-patient false negatives and curvature after controlling for self-reported ethnicity. If it does not, we drop the claim that the genetic axis adds anything.
Why not just remove demographic information with an adversary? Adversarial scrubbing needs labels. FairAdaBN reports that "fairness through unawareness" fails because correlated anatomy remains. The OCA2-HERC2 finding also suggests that pigmentation partly changes what is actually visible in the image, so scrubbing it could erase real signal. We would not argue this a priori. If the genetic audit shows pigmentation-linked errors, an adversarial baseline run on the same data would show whether scrubbing lowers overall AUC.
Proposed Experiments
Datasets. The main benchmarks are CheXpert and MIMIC chest X-rays, Harvard-FairVLMed and BRSET fundus images, and Fitzpatrick-17k. On chest X-rays, foundation models show strong underdiagnosis gaps for intersectional groups such as Black women (Demographic bias of expert-level vision-language foundation models in medical imaging). UK Biobank fundus images with genotypes are used for the genetic audit.
Metrics. We report overall AUC, worst-group AUC, equity-scaled AUC, false negative rate gaps including intersectional strata, and the measured $\mathrm{tr}(H_g\Sigma)$.
Decisive check 1. Across datasets, groups and averaging methods, predicted and observed gap changes should correlate positively at small displacement. Their sign should flip as displacement grows from SWA to Model Stock to uniform soups of independent runs. If gap changes are unrelated to measured curvature, the core claim fails.
Decisive check 2. The variance-weighted averaging rule must beat uniform SWA on worst-group AUC, with the same checkpoints and compute.
Decisive check 3. In the genetic audit, pigmentation genotype must carry signal about errors and averaging gains beyond self-reported ethnicity, and the pattern must replicate in skin-tone strata on Fitzpatrick-17k.
Outcome if all three hold. Clinics would get a fairness intervention that needs no demographic labels and costs nothing at inference, together with a diagnostic, curvature along the training trajectory, that says in advance when that intervention is safe.
References
- Bridging the genotype-phenotype gap with generative artificial intelligence
- Improving Generalization of Pre-trained Language Models via Stochastic Weight Averaging
- Ensemble of Averages: Improving Model Selection and Boosting Performance in Domain Generalization
- FairMerging: Rethinking Model Merging through the Lens of Fairness
- Model Stock: All We Need Is Just a Few Fine-Tuned Models
- The Unusual Effectiveness of Averaging in GAN Training
- FairAdaBN: Mitigating Unfairness with Adaptive Batch Normalization and Its Application to Dermatological Disease Classification
- Equitable artificial intelligence for glaucoma screening with fair identity normalization
- Fair-FedMOE: Group-Fair One-Shot Federated Learning via Prototype-Guided Experts for Medical Imaging Analysis
- Unveiling Fairness Biases in Deep Learning-Based Brain MRI Reconstruction
- Demographic bias of expert-level vision-language foundation models in medical imaging