Is "Anatomy as Destiny" Really "Voxel Count as Destiny"? Scale as the Hidden Driver of Demographic Gaps in Volumetric MRI

Deep networks that reconstruct or segment 3D brain MRI perform measurably differently across sex and age groups, and the usual fixes do not close the gap. Equalizing how many subjects of each group appear in training, or in each minibatch, fails. So far the leading explanation is a vague one: female and younger brains simply differ in anatomy. We propose a sharper and testable explanation. In a volumetric scan, anatomy differs above all in physical scale: how many voxels (3D pixels) the head and each structure occupy, and so how much empty background surrounds them. Scale affects three things that have nothing to do with demography. It changes how whole-image quality scores average over easy background. It changes how much weight each subject gets in a loss summed over voxels. It changes how much of the anatomy fits inside the network's field of view, whether that view is a 2D slice, a 3D patch, or a whole volume. Unlike sex or age, scale can be manipulated directly: we can resize a real subject and watch what happens.

The note proposes a causal audit and a matched intervention. The audit resizes subjects in controlled steps and measures how much of the demographic gap is carried by scale, and how that share changes as spatial context grows from 2D slices to full 3D networks. The intervention removes scale's influence through the pipeline itself (scale augmentation and per-subject loss normalization) and never touches demographic labels. If scale carries most of the gap, then the well-documented failure of rebalancing has a mundane cause: the gap is being attacked along the wrong axis.

Core Claim

We claim that the sex and age disparities that survive rebalancing in volumetric brain MRI are substantially mediated by per-subject physical scale. We also claim that this mediation depends on how much volumetric context the model sees.

Two consequences follow, and both are checkable. First, part of the reported gap may be an artifact of how quality is measured. Consider the direction of the gap reported by Du et al.: the smaller female brains reconstruct better (SSIM, a structural-similarity score, of 0.876 versus 0.868 for males). This fits a simple story. A smaller head leaves more nearly-empty background, which is easy to reconstruct and inflates a score averaged over the whole image. Second, whatever gap remains after masking to the brain should shrink when we remove the pipeline's sensitivity to scale. It should not shrink when we equalize group counts.

What is new here is the choice of target. Existing work treats the problem as demographic and intervenes on groups. We treat it as geometric and intervene on scale, and we test that choice causally with counterfactual resizing.

Where This Sits

The motivating evidence comes from the bias-mitigation literature. Du et al. (preprint here) audited a U-Net for undersampled brain MRI reconstruction on 162 OASIS subjects. They showed that dataset and minibatch rebalancing leave sex and age gaps significant, and often lower the favored group's quality instead of lifting the other group. They proposed "anatomy as destiny" after observing correlations with intracranial volume (eTIV) and normalized whole-brain volume (nWBV), but they explicitly left causation and architecture unexamined. We supply the causal test and the architecture sweep.

MEDFAIR benchmarked 11 debiasing methods across 10 medical datasets, including 3D ADNI and CT data. It found that none reliably beats plain empirical risk minimization (ERM, ordinary training), and that checkpoint selection and the flat-minimum optimizer SWAD matter more. Its tasks were all binary classification. This result constrains our design: we must beat SWAD and minimax selection, not just naive training.

Bell et al. supply a measurement tool. They define a difficulty amplification factor $k$: the gap between groups when a model is trained jointly, divided by the gap when each group is trained and tested in isolation. They measured $k = 1.19 \pm 0.12$ for ResNet-18 on a controlled CIFAR-100 setup. We reuse $k$ to ask whether larger spatial context amplifies or damps the scale-driven gap.

Qiu et al. and Ren et al. make the conceptual case we build on. Qiu et al. find that phenotype imbalance hidden inside balanced demographic groups sustains bias. Ren et al. find that acquisition and environment factors are not fixed by fixing label skew. Our claim is a specific instance of this pattern: scale is the hidden phenotype.

On the volumetric side, the file supplies both the architectures we sweep and a key mechanism. The architectures run from 3D convolutional segmentation (Dou et al.) and the 3D U-Net, which add inter-slice context, to SegVol, a 3D vision transformer that looks at a coarse zoomed-out view before refining. The mechanism comes from Volumetric Bias in Segmentation and Reconstruction, which proved that classical likelihood energies secretly favor equal-sized segments. There, correcting the bias cut GrabCut error from 14.41% to 7.87%. That paper concerns classical optimization and has no demographic dimension. We carry its core point, that voxel-summed objectives weight things by size, over to deep volumetric training and to subjects as the unit.

Finally, entropy-based active learning for fair brain segmentation found that restricting uncertainty to a region of interest keeps large anatomy from dominating sample selection. On synthetic SimBA data, this cut disparity by 75% under strong bias and 86% under weak bias. That result points the same way we do, but it acts on data acquisition, needs group labels, and does not isolate scale. Our interventions act on training and evaluation and use no demographic labels.

Research Program

Stage 1: measurement correction. We recompute every quality metric in two forms: over the whole image and restricted to a brain mask, with the mask fixed from the fully sampled image. For segmentation we additionally report boundary distance alongside Dice (an overlap score), because Dice is known to penalize small structures. This separates a gap that lives in background averaging from a gap that lives in the anatomy.

Stage 2: counterfactual scale audit. Each test subject's fully sampled volume is resized with an affine scaling. The scale factors map a male-typical intracranial volume to a female-typical one and back, and span the age range of nWBV. We then simulate undersampled k-space (the raw frequency-domain measurements) from the resized image, the same way the original data are undersampled, and rerun the trained model.

Let $G$ be the observed between-group gap in the brain-masked metric. Let $G_{\text{matched}}$ be the gap after each subject is rescaled to the other group's median scale. We report the scale-mediated share

$$ \rho = \frac{G - G_{\text{matched}}}{G}, $$

with bootstrap confidence intervals over subjects. A $\rho$ near 1 means scale carries the gap; a $\rho$ near 0 means it does not.

Resizing also changes effective resolution and noise texture. To separate "size" from "resampling," a control condition resamples at the subject's native scale through the same interpolation path.

Stage 3: context sweep. We repeat Stages 1 and 2 across five spatial-context regimes, matching parameter count within about 10% where feasible:

  1. a 2D slice U-Net;
  2. a 2.5D U-Net that stacks neighboring slices as input channels;
  3. a 3D U-Net on patches;
  4. a whole-volume 3D U-Net at reduced resolution;
  5. a zoom-out/zoom-in 3D transformer in the style of SegVol, for segmentation only.

For each regime we report $\rho$ and Bell's $k$. The competing predictions are clear. Wider context might lower $\rho$, because the model sees the whole head and can calibrate to its size. Alternatively, fixed-size 3D patches might raise it, because a small head fills a different fraction of each patch.

Stage 4: scale-targeted interventions, with no demographic labels. The first intervention is random isotropic scale augmentation over the observed eTIV range. The second is per-subject loss normalization: the training loss is averaged within each subject's brain mask before averaging across subjects, so every subject counts equally regardless of voxel count,

$$ L = \frac{1}{S}\sum_{s=1}^{S} \frac{1}{|M_s|}\sum_{v \in M_s} \ell_v , $$

where $S$ is the number of subjects in the batch, $M_s$ is subject $s$'s brain mask, and $\ell_v$ is the per-voxel loss. The third intervention combines both.

Datasets. The tasks are OASIS reconstruction, which directly extends Du et al., and SimBA synthetic brain segmentation. SimBA is used because its generator lets us plant morphological group differences of known size. For each model we can therefore check whether $\rho$ recovers the planted scale component and assigns nothing to a planted non-scale component, such as a shape change at fixed volume. That test validates the audit itself.

Baselines

Naive training. The reference point is plain ERM with the standard voxel-averaged loss, trained under the Du et al. protocol on the same OASIS subset, split, and undersampling. The reported ERM numbers for reconstruction are SSIM 0.876 (female) versus 0.868 (male), and 0.876 (under 40) versus 0.867 (over 65), with $p < 0.001$ Du et al.. Reproducing these within seed variance is the precondition for everything else. For SimBA segmentation, no reported Dice was found on Lacuna, so we establish our own ERM reference.

Group rebalancing. This baseline uses dataset-level and minibatch-level balancing across sex and age. It is the intervention the field reaches for first, and the one Du et al. showed fails. They report that the gap remains significant at $p < 0.001$; no reported gap magnitude after rebalancing was found on Lacuna.

Strongest existing competitor: SWAD with minimax Pareto checkpoint selection. SWAD is a weight-averaging optimizer that seeks flat minima. Minimax Pareto selection picks the checkpoint with the best worst-group score. MEDFAIR found this combination to be the most reliable route to worst-group performance in medical classification, improving average worst-group rank from 2.53 to 1.49 MEDFAIR. No reported result for dense 3D tasks was found on Lacuna.

Lipschitz gradient penalty. This baseline follows Bell et al., who report that constraining how fast the network's output can change brings $k$ below 1. It tests whether a generic simplicity-bias remedy does the job without targeting scale.

Registration to template (segmentation only). Every volume is affinely registered to a common template before training and inference, and predictions are mapped back to native space for scoring. This is the standard preprocessing answer to size differences.

Isolating ablation. The ablation replaces scale augmentation with rotation augmentation of matched strength. This separates a scale-specific effect from generic augmentation.

Fair comparison and success criterion. All methods share the same data, splits, architecture per context regime, compute budget, and number of tuning trials: equal hyperparameter search counts and five seeds each. An intervention "wins" if it meets two conditions. First, it reduces the brain-masked worst-group gap by more than SWAD with minimax selection does, with a bootstrap interval excluding zero. Second, it does not lower the best group's masked score beyond seed variance. The second condition rules out the "dominant group penalty" Du et al. describe.

Why not just…?

Why not just rebalance or reweight by group? Du et al. already ran exactly this on the same data and found the gap unchanged and the favored group degraded. MEDFAIR found rebalancing no better than ERM across 10 datasets. Group rebalancing equalizes subject counts, but it leaves a subject's influence proportional to its voxel count, and it does nothing about background inflating the whole-image metric. We include it as a baseline to confirm the replication, not as a live contender.

Why not just report brain-masked metrics and stop? This is the real threat, and it is why Stage 1 comes first. If masking removes the sex gap entirely, then part of the published disparity is a measurement artifact. That is itself worth establishing. It would also narrow our claim to whatever age gap remains, since atrophy plausibly produces a genuine within-brain difficulty. We commit in advance to reporting this outcome either way. Stages 2 through 4 proceed only on the masked residual.

Why not just register every scan to a template? For segmentation this is a strong, cheap alternative, and we run it as a baseline. For reconstruction it is unavailable. The network receives undersampled native-space measurements, so no registration can happen before the image exists. A remedy that works inside the pipeline is therefore needed there regardless. If registration matches our interventions on segmentation, we will say so, and we will restrict the intervention claim to reconstruction.

Why not just use SWAD and minimax selection, which MEDFAIR found best? These methods choose among, or average over, solutions that ordinary training produces. If the voxel-summed loss systematically underweights small subjects, every checkpoint inherits that weighting. The head-to-head comparison in Stage 4, under a matched tuning budget, settles whether this concern matters in practice.

Why not just switch to a full 3D model? More context might fix the problem on its own, and Stage 3 tests exactly that. The file offers no evidence either way. Bell et al. warn that more expressive models can amplify gaps, while Dou et al. and the 3D U-Net show that context resolves ambiguous boundaries. We treat the direction of the effect as an open empirical question and do not presuppose it.

Decisive Checks

The first check is validation on SimBA. The audit must recover a planted scale difference (predicted $\rho$ close to 1) and must assign almost none of a planted fixed-volume shape difference to scale (predicted $\rho$ close to 0). If it fails either test, the method is not trustworthy on real data and the project pivots to fixing it.

The second check is on OASIS. We ask whether the brain-masked sex and age gaps survive, and what $\rho$ is for each regime in the context sweep.

The third check is the intervention test. Scale augmentation plus per-subject normalization, using no demographic labels, must beat SWAD with minimax selection on the masked worst-group gap without penalizing the best group. The rotation-augmentation ablation must fail to match it.

The claim is falsified if $\rho$ is near zero after masking in every regime. That outcome would still be useful. It would rule out the simplest reading of "anatomy as destiny" and point the field toward acquisition or tissue-contrast explanations instead.