Auditing Generative Medical Images for Population-Specific Anatomical Hallucination with Shadow-Feature Tests

Generative models that rebuild medical images from incomplete scans can invent anatomy. When the measurements leave the answer open, the model fills the gap with what its training data made typical. We propose that these inventions are not random. They are systematically biased toward the anatomy of the population the model was trained on. A model trained mostly on young brains should shrink the ventricles of an elderly patient and thicken the cortex, and nothing in today's image-quality scores would flag it. We propose an audit that measures dozens of interpretable anatomical quantities, such as regional volumes, cortical thickness, vessel connectivity, and lung border landmarks, on paired real and generated images. It then asks which of these quantities a classifier can use to tell real from generated, and in which patient subgroups, beyond what chance allows.

The statistical engine is Boruta, an "all-relevant" feature-selection method. Boruta adds a scrambled copy (a "shadow") of every feature and keeps only the real features that beat the best shadow more often than chance across many random-forest fits. We adapt the shadows for paired, subgroup-structured data so that the chance baseline respects patient identity and population membership. A feature that survives is a statistically certified anatomical distortion. Because Boruta keeps every relevant feature rather than one representative of each correlated group, the audit reports the whole pattern of distortion, for example ventricles, hippocampus, and periventricular thickness together. That full pattern is what a clinician or model developer needs.

Core Claim

We make two claims, and each can be tested separately. The scientific claim is that generative reconstruction and translation models hallucinate anatomy that regresses toward the training population. The hallucinations are therefore concentrated in under-represented subgroups and point toward the training-set mean. The methodological claim is that a shadow-feature test with pair-swap shadows yields a calibrated, interpretable detector for such subgroup-conditional distortions. In a pair-swap shadow, each shadow feature is made by randomly swapping the real and generated values within each patient's pair, rather than by permuting across all rows. Under the null hypothesis of no distortion, the detector confirms features at no more than the nominal rate. It also has more power than per-feature univariate testing when distortions only appear in combination with subgroup membership.

The second claim is where we expect pushback. We state up front that if a simpler univariate test matches the detector's power, we will drop Boruta and keep the audit protocol.

Why This Gap Exists

The motivating evidence comes from the hallucination literature. Fast MRI Reconstruction: How Powerful Transformers Are? shows that at extreme undersampling (around 10% of raw k-space), adversarial reconstruction favours realistic texture over fidelity to the measurements. Rethinking Perceptual Metrics for Medical Image Translation shows that Inception-based scores such as Fréchet Inception Distance reward plausible fabricated texture. Together these establish that the standard scores cannot see the failure we care about.

The structure-preserving methods supply our test generators and constrain our design. SegGuidedDiff conditions image-space diffusion on segmentation masks. TPOT adds a persistent-homology loss, a penalty on broken or spurious vessel loops, to retinal enhancement. CF-Seg generates pseudo-healthy counterfactual chest X-rays. Diff-Def avoids intensity hallucination by generating deformation fields that warp a template.

Every one of these methods relies on a frozen segmentation or structure network. The research file notes that such networks fail on out-of-distribution scanners and demographic subgroups. Each method also reports a single pooled score, such as Dice, PSNR, SSIM, or radiologist preference. None asks which anatomy is distorted for whom. Our audit asks that question, and it treats the frozen extractors' own subgroup bias as a confound to be cancelled by pairing.

The Boruta literature supplies the method. Its biomedical uses select features to predict a disease label. Examples include morphometric and diffusion features for gaming disorder in MRI-Based Radiomic Machine-Learning Model…, and Cell Painting features mapped to cell-health readouts in From pixels to phenotypes. We repurpose the shadow null for a different target: telling real from generated within a patient. That target requires a different permutation scheme.

The population-bias part of the research file was truncated and gave us no citable papers. We therefore treat population structure as the evaluation setting, defined by the subgroups available in the datasets: age, sex, site, and hospital system. We do not build on a specific prior method in that area.

Research Program

The audit has three components.

Measurement. For every patient we compute a vector $f$ of interpretable anatomical measurements on both the real image and the generated image. Brain MRI uses FreeSurfer-style regional volumes and thicknesses plus a Jacobian-based local volume change map summarised by region, following the deformation view of Diff-Def. Retinal images use vessel connected-component and loop counts from a persistence diagram, as in TPOT. Chest X-rays use landmark-based heart and lung border shape descriptors.

Because the same extractor runs on both images of a pair, a subgroup-specific bias in the extractor cancels to first order. A residual concern is that the extractor may behave differently on generated texture. We check that by running two different extractors and keeping only distortions both agree on.

The test. We form a dataset in which each patient contributes one "real" row and one "generated" row, plus subgroup indicators. A random forest learns to predict which row is which. Pair-swap shadows are built by randomly exchanging the real and generated values of a feature within each patient. This preserves the feature's marginal distribution, its within-patient correlation, and its subgroup composition, while destroying any systematic real-versus-generated difference.

For unpaired synthesis, such as mask-conditioned SegGuidedDiff samples, shadows are instead permuted within subgroup strata. Without this, a feature merely correlated with subgroup would look relevant whenever the real and synthetic sets have different subgroup mixes. The Boruta binomial decision rule is then applied unchanged. Features are confirmed, rejected, or left tentative depending on how often they beat the best shadow across roughly 100 forest fits.

Localisation to subgroups. For each confirmed feature we fit the forest's partial dependence on subgroup membership and report the signed generated-minus-real difference per subgroup. This turns "ventricular volume is distorted" into "ventricular volume is under-estimated by the model in patients over 70, toward the training-set mean." That second sentence is what the scientific claim predicts.

Baselines

The naive reference point is what papers report today: pooled PSNR, SSIM, and Fréchet Inception Distance, plus pooled segmentation Dice between real and generated images. These run on exactly the same paired images. The audit beats this reference if it detects a planted or naturally arising subgroup distortion that leaves these pooled scores essentially unchanged. The Konz et al. critique predicts exactly this outcome for Fréchet Inception Distance.

The strongest simple competitor is per-feature paired testing. This means a Wilcoxon signed-rank test on generated-minus-real differences for each feature, within each subgroup and pooled, with Benjamini–Hochberg false-discovery control across all feature-by-subgroup tests. It uses the same features and pairs and has no tuning, so the comparison isolates the decision procedure. To count as beating it, our method must show higher detection power at matched empirical false-discovery rate on planted distortions. The advantage should come specifically in small subgroups and for distortions that appear only in a combination of subgroup and covariate.

The strongest representation-level competitor is a classifier two-sample test. A CNN is trained to tell real from generated images per subgroup, with held-out accuracy above 0.5 indicating a difference. It gets the same images and a matched compute budget. It may well be more sensitive than our method. Our method beats it only on localisation: naming the anatomical quantity and direction, scored against planted ground truth.

The ablation that isolates our key ingredient is standard Boruta with shadows permuted across all rows. It is run with identical forests and iteration budgets. We expect it to over-confirm features under the null whenever real and generated sets differ in subgroup mix, or whenever between-patient variance dominates. Pair-swap shadows beat it if they hold the null confirmation rate near nominal while the standard version does not.

We found no reported result on Lacuna for detection power or false-discovery rate of any anatomical hallucination audit. The only comparable published numbers are generator-quality figures: SegGuidedDiff's Dice above 0.90 on breast MRI and about 0.89 on multi-organ CT, TPOT's PSNR of 25.196 and SSIM of 0.9417 on EyeQ, and Diff-Def's Dice of 0.71 (age) and 0.75 (ventricular volume) on UK Biobank. We will use these only to confirm that our reimplemented generators reach published quality before auditing them.

Why not just…?

Why not just run univariate tests with false-discovery correction? This is the real threat. In the simple case of a single feature shifted uniformly across a large subgroup, a paired Wilcoxon test will likely match Boruta. Our bet is narrower. Splitting tests by subgroup multiplies the correction burden and starves small subgroups of power. A forest given subgroup indicators can share strength across subgroups and pick up interactions, such as distortion only in older women at one site. The planted-distortion experiment varies subgroup size and interaction order precisely to locate where, if anywhere, the forest wins. If it never wins, we will say so and recommend the univariate audit.

Why not just use a deep real-versus-generated classifier? It answers "is something wrong" but not "what anatomy, for whom, in which direction." Saliency maps on such classifiers tend to highlight texture and noise rather than structure, which is the same weakness Konz et al. document for Inception features. We will still report its sensitivity honestly. If it detects distortions our features miss, that measures the coverage gap of our feature set.

Why not just evaluate segmentation Dice per subgroup? This is what SegGuidedDiff-style evaluation would extend to. But Dice depends on a frozen segmenter whose own errors vary by population, and a regional volume can drift substantially while overlap stays high. Our design reports per-subgroup Dice alongside the audit and checks whether Dice moves at all when we plant volume shifts that the audit detects.

Proposed Experiments

Controlled study on UK Biobank brain MRI. This is the cohort Diff-Def used. We retrospectively simulate undersampling of the magnitude images, accepting that this is weaker than true k-space. We then train a diffusion reconstruction model on a deliberately age-skewed training set, mostly under 60.

On the test set we add planted distortions with known ground truth. These are Diff-Def-style deformation fields that shrink or expand chosen regions by controlled amounts, applied only within a chosen subgroup. We sweep effect size, subgroup size, and the number of jointly distorted regions.

A null condition supplies the calibration check. In it, "generated" images are fully sampled reconstructions with matched noise, so no anatomy is distorted. The decisive checks are three. First, the null confirmation rate must stay at or below the nominal level for pair-swap shadows. Second, power must be compared against the univariate and CNN baselines at matched false-discovery rate. Third, on the unplanted, naturally trained model, the regression-to-training-population hypothesis must hold. Confirmed distortions should concentrate in older patients, involve age-sensitive structures, and point toward the young-training mean. A null finding here, meaning no subgroup-concentrated distortion despite skewed training, would falsify the scientific claim.

Population shift in chest radiography. We use the pseudo-healthy counterfactual setting of CF-Seg, training on MIMIC-CXR and auditing on PadChest (and the reverse). Landmark shape features are measured on real and counterfactual images in the regions that disease should not alter. The prediction is that cross-hospital application produces confirmed shape distortions absent under within-hospital application, and that these distortions survive our two-extractor agreement filter.

Topology in retinal imaging. We audit TPOT-style enhancement on EyeQ, with DRIVE used to verify vessel extraction, using vessel component and loop counts as features. This tests whether the audit generalises from volumetric to topological measurements. It also tests whether a model trained with an explicit topology loss still shows subgroup-conditional connectivity errors, which pooled F1 and AUC would hide.

Across all three, success means a calibrated null, a documented power regime where the shadow test does or does not beat univariate testing, and at least one real generator in which the audit reveals a population-specific, directional anatomical distortion invisible to the pooled scores the field currently reports.