Reconstruction-Consistent Segmentation: Teaching MRI Models That Different Reconstructions Show the Same Anatomy

Medical image segmentation models often look excellent on curated research scans and then fail in the hospital. Bajwa et al. document models that fall to near-chance performance on everyday clinical images. One cause has received too little attention: the image a model sees is not the raw measurement. It is the output of a reconstruction algorithm, and hospitals and vendors reconstruct differently. An MRI scanner measures frequency-domain data called k-space and turns it into a picture. Sites differ in how much of k-space they skip to save time, and in whether they fill the gaps with classical methods or with neural networks. Evaluating the Impact of Medical Image Reconstruction on Downstream AI Fairness and Performance shows that reconstructors tuned only for pixel fidelity can distort structure, and that this hurts downstream models and widens gaps between demographic groups.

The proposal rests on one observation. If we hold one raw acquisition, every reconstruction of it shows the same patient, so the correct segmentation is the same for all of them. That gives an unusually clean source of "views" for consistency regularization, the practice of penalizing a model when its predictions change across transformations that should not change the answer. The views are physically real rather than hand-designed pixel noise. They need no labels, so the penalty can be applied to large unlabeled collections of raw or simulated k-space, including data from the deployment site itself. We propose to train segmentation networks with a confidence-weighted consistency loss across reconstructions of the same acquisition. We will then test whether this beats the strongest known recipe, labeled corruption augmentation, under reconstruction shift, synthetic corruption, and clinical-quality scans.

Core Claim

Reconstruction-consistency regularization on unlabeled acquisitions should give better robustness to unseen reconstruction pipelines and acquisition degradations than training on the same reconstructions with labels alone, when labeled scans are scarce. It should also cost little or nothing in clean-scan accuracy.

The claim is deliberately narrow. Its value depends on two things. First, consistency can use data that supervised augmentation cannot use, namely unlabeled acquisitions. Second, the views carry the right kind of variation, real reconstruction differences rather than mathematical noise. If labeled augmentation with the same reconstructions matches it, the claim fails. We say so up front because that is the most dangerous alternative.

Where This Sits in the Prior Work

Several papers motivate the gap. Chai et al. show that convolutional networks trained on clean neonatal brain MRI rely on texture and collapse under corruption, with Dice scores of 0–1% for some tissue classes on heavy blur. They also find that corruption augmentation, especially salt-and-pepper noise, restores robustness, but that overly aggressive corruption damages anatomical boundaries and clean accuracy. Imam et al. find that clean accuracy does not predict corruption robustness in medical vision-language models. They adapt about 1% of encoder weights with low-rank adapters (RobustMedCLIP) to cut corruption error. Both works, and the file's discussion of open problems, rely on synthetic pixel corruptions such as Gaussian noise, impulse noise, and blur. Moving to natural clinical artifacts is listed as open. Our proposal targets that gap for one important family of natural shifts.

Other work supplies methods we reuse or contrast with. AnyStar trains on fully synthetic volumes passed through a gauntlet that includes k-space spikes, Gibbs ringing, and Rician noise. It shows that acquisition-physics corruptions generalize across modalities, but it applies them as supervised augmentation on synthetic shapes and is limited to star-convex objects. We apply acquisition physics as a label-free consistency signal on real anatomy. TractSeg shows that heavy noise augmentation widens a segmentation advantage from 9 to 14 Dice points when moving from Human Connectome Project data to lower-quality clinical scans. That is evidence that robustness gains show up most where data quality drops, and we expect the same here.

For the consistency side, Fraternal Dropout enforces agreement between two dropout-perturbed copies of a network. That is a model-side perturbation. Ours is a data-side perturbation whose invariance is guaranteed by physics, which avoids the standard worry that a hand-picked augmentation silently changes the label.

One constraint shapes the design. Chai et al.'s finding that excessive corruption destroys boundaries implies that naive symmetric consistency toward a heavily undersampled view could pull the model toward confident guesses where the information is simply gone. We therefore make the loss asymmetric and confidence-gated, as described below.

Research Program

Generating views. For each acquisition we produce a set of reconstructions. These include a reference reconstruction (fully sampled, or the least accelerated available) and degraded reconstructions at several undersampling rates and sampling patterns. Each degraded rate is reconstructed several ways: zero-filled inverse Fourier transform, compressed sensing, and a neural reconstructor. We will train the neural reconstructors ourselves, with deliberately different training losses, so that they introduce different structural distortions. For datasets that store only magnitude images, such as the Developing Human Connectome Project (dHCP) neonatal T2 volumes used by Chai et al., we simulate k-space by Fourier transforming the magnitude volumes. This omits real phase and coil effects, and we treat it as a limitation. Where public multi-coil raw k-space with brain anatomy is available, we repeat the key experiments on it.

Training objective. The segmentation network $f_\theta$ is trained on a small labeled set with standard Dice and cross-entropy. On all acquisitions, labeled and unlabeled, it is also trained with

$$ \mathcal{L}_{\text{rc}} = \sum_{v} w(x_{\text{ref}})\,\mathrm{KL}\big(\,\mathrm{sg}[f_\theta(x_{\text{ref}})] \,\|\, f_\theta(x_v)\big), $$

where $x_{\text{ref}}$ is the reference reconstruction and $x_v$ a degraded reconstruction of the same acquisition. Here $\mathrm{sg}$ stops gradients so that the reference prediction acts as a fixed teacher, and $w$ is a per-voxel weight that is zero where the teacher's maximum class probability falls below a threshold. The asymmetry ensures that information can only flow from the better view to the worse one. The gate keeps the model from being pushed to agree with itself where it does not know the answer.

Domain shift by design. To create domain shift, we hold out entire reconstructor families and acceleration factors from training. At test time the model faces pipelines it never saw. In a second regime, we allow consistency, but never labels, on unlabeled acquisitions from the target pipeline. This mimics a site that can share raw data but cannot annotate it.

Analysis. Beyond accuracy, we measure whether the regularizer reduces texture reliance. We follow Chai et al.'s reasoning and probe predictions under high-frequency filtering. We also report Dice separately by subgroup where demographic metadata exists, because the reconstruction-fairness paper suggests reconstruction shift falls unevenly across groups.

Baselines

The naive reference point is a network trained only on reference-quality labeled scans with standard geometric augmentation. It is the "do nothing" model that Chai et al. show collapsing to near-zero Dice on severe blur and impulse noise. Every method shares its architecture (a 3D U-Net), training length, and labeled set.

The strongest existing method is Chai et al.'s composite corruption-augmentation recipe, which combines Gaussian noise at several strengths with salt-and-pepper noise. On their dHCP protocol it keeps Dice "often >90% for all classes," including "72%-98% on gaus05" and "88%-99% on snp20" (Chai et al.). It is the strongest competitor because it is cheap, needs no extra data, and already generalizes across unseen pixel corruptions. We will reproduce it on our splits and tune it under the same hyperparameter budget as our method, which is a fixed number of trials over corruption strengths. Beating it means two things. On held-out reconstruction pipelines, we must exceed its mean Dice with a paired, statistically reliable gain across subjects. On its own gaus05 and snp20 corruptions, we must not fall below its reported ranges. No reported result found on Lacuna for any method under held-out MRI reconstruction shift, so that comparison will be produced fresh.

A second strong competitor is AnyStar-style acquisition-physics augmentation (Dey et al.), meaning supervised training with Rician noise, Gibbs ringing, k-space spikes, and bias fields applied to labeled scans. It tests whether physics-flavored corruption alone, without consistency, is enough.

The key ablation is supervised reconstruction augmentation. Here we train on exactly our reconstructions of the labeled scans, with labels, and drop the consistency term. This isolates whether gains come from consistency and unlabeled data rather than from the reconstructions themselves. Further ablations remove the confidence gate, make the loss symmetric, and replace reconstruction views with Chai-style pixel corruptions inside the same consistency loss, which separates "consistency" from "physically real views."

For the foundation-model direction, we compare against low-rank adapter fine-tuning in the spirit of RobustMedCLIP (Imam et al.), applied to a pretrained segmentation encoder with Chai augmentation. Its reported mean corruption error of 70.1 on cell microscopy is a classification number on a different task, so it is context, not a target.

Why not just…?

Why not just augment with the reconstructions and use the labels? This is the real threat. If every acquisition is labeled, supervised augmentation over reconstructions probably captures most of the benefit, and we expect it to roughly tie our method at full label budgets. The proposal's advantage is that consistency needs no labels. We therefore run a label-budget sweep at 5, 10, 25, and 100% of labeled subjects with the unlabeled pool held fixed. If the supervised-reconstruction ablation matches our method at low budgets as well, the claim is false and we will report it as such.

Why not just use Chai et al.'s pixel-noise augmentation, since it already generalizes broadly? Salt-and-pepper training did generalize to unseen blur and filtering. However, reconstruction artifacts are spatially structured: aliasing folds, ringing, and network hallucinations that smooth or invent edges. These are unlike independent pixel noise. The reconstruction-fairness study reports structural, not merely noisy, distortions. The held-out-reconstructor test measures directly whether pixel-noise robustness transfers. If it does transfer fully, the extra machinery is unnecessary.

Why not just retrain or swap the reconstructor so it preserves structure? Downstream model developers rarely control the vendor's reconstruction, and sites change pipelines over time. A segmenter that is robust across reconstructors avoids that dependency. The two approaches are complementary, and our held-out-reconstructor results will indicate how much headroom remains for the reconstruction side.

Why not just fine-tune a foundation model with adapters? Adapters reduce corruption error in vision-language classification, but they still need labeled corrupted data and were tested on synthetic corruptions. We include the adapter baseline. Our consistency loss can also be applied on top of an adapter-tuned model, so a win for adapters would not exclude our method.

Proposed Experiments and Decisive Checks

The primary benchmark is dHCP neonatal T2 MRI with 10 tissue classes and Chai et al.'s subject splits and corruption suite. We add simulated k-space reconstructions at 4x, 8x, and 12x acceleration with three reconstructor families, holding out one family and the 12x rate entirely. We report per-class Dice, boundary distance (95th-percentile Hausdorff), clean-scan Dice, and the drop from clean to shifted. Every result is averaged over three seeds and tested with paired per-subject comparisons.

The secondary check is transfer to lower-quality clinical-style data. Following TractSeg's evidence that augmentation gains grow on clinical scans, we will evaluate any available clinical-quality neonatal or pediatric brain scans with expert labels. A raw multi-coil k-space brain collection, if obtainable, tests whether conclusions survive real phase and coil effects.

Three outcomes would settle the question. First, under the low-label regime, our method must beat the supervised-reconstruction ablation on held-out reconstructors. If it does not, consistency adds nothing. Second, it must beat Chai's composite augmentation on reconstruction shift while staying within Chai's reported Dice ranges on gaus05 and snp20. If it does not, physical views add nothing over pixel noise. Third, removing the confidence gate should hurt mostly at 12x acceleration, which would confirm that the gate prevents the boundary damage Chai et al. warn about. Target-site unlabeled consistency is a final, practically important test. If allowing consistency on unlabeled acquisitions from the deployment pipeline closes most of the remaining gap, a hospital could adapt a segmenter to its own scanner using raw data alone, with no annotation.