Adapting MRI Reconstruction Once Per Volume, Not Once Per Slice

Deep networks can rebuild an MRI image from a fraction of the usual measurements in milliseconds. That speed is what makes faster scans possible. The networks break, though, when the scanner changes: a model trained on one vendor's machines degrades on another's. The best known fix is test-time training, which fine-tunes the network on each new scan using only the scanner's own raw data. It recovers most of the lost accuracy, but it costs minutes per slice. Because every slice is adapted independently, it can also leave the stacked 3D volume with slice-to-slice jumps.

This note starts from a physical observation. The things that differ between scanners (coil arrays, receive filters, oversampling defaults, noise level, contrast settings) are the same for every slice of a scan, while the anatomy changes from slice to slice. The correction for a new scanner should therefore be estimated once per volume, from all slices at once, and applied identically to every slice. We propose to learn a small set of volume-shared correction parameters. We first fit them with a few gradient steps pooled over the whole volume. We then replace those steps with a feed-forward predictor, so that adaptation adds a fraction of a second rather than minutes. A lightweight cross-slice module lets each slice's reconstruction draw on its neighbours.

Core Claim

The claim is that, for multi-slice MRI reconstruction under scanner shift, per-volume adaptation of a small shared parameter set can match per-slice test-time training in accuracy. It should also give better through-plane continuity, and it should be one to two orders of magnitude faster. A feed-forward version that predicts those parameters from the volume's raw measurements can retain most of that benefit at near real-time latency.

What would be new is the unit of adaptation, the volume rather than the slice. The file contains no method that pools raw-measurement self-supervision across slices to fit a correction shared by the whole volume, nor any evaluation of through-plane consistency under adaptation.

The second claim addresses an open tension in the literature.

We predict the two are reconciled by ordering. Cross-slice context helps out of distribution only after the scanner-level correction has been applied. Without that correction, cross-slice context propagates scanner artifacts from neighbouring slices. This prediction is falsifiable, and the experiments are built to test it.

Research Program

Base reconstruction network. The base network is an unrolled reconstruction network in the style of E2E-VarNet. Such a network alternates learned image refinement with "data consistency" steps that force the estimate to agree with the measured raw Fourier data (k-space). It also estimates the coil sensitivity maps (how strongly each receive coil sees each point) from the fully sampled low-frequency centre of k-space. We keep that design and add two ingredients.

First ingredient: volume-shared adapter. The first is a small set of parameters $\phi$, shared by all slices of a volume. It consists of per-cascade feature scales and shifts, a low-rank update to the sensitivity-estimation sub-network, and a scalar noise-level term in the data-consistency weight. It totals a few thousand numbers.

How the adapter is fitted. We fit $\phi$ at test time with the self-supervised scheme of Darestani et al. (Test-Time Training Can Close the Natural Distribution Shift Performance Gap in Deep Learning Based Compressed Sensing). That scheme splits each slice's acquired k-space lines into a fitting set and a held-out set, and stops early when the held-out error rises. Unlike their per-slice full-weight fine-tuning, we optimize

$$\hat\phi = \arg\min_{\phi} \sum_{z=1}^{Z} \big\| M^{\text{fit}}_z \big( \mathcal{A}_z f_{\theta,\phi}(y^{\text{fit}}_{z-1:z+1}) - y_z \big) \big\|_2^2 .$$

In this objective:

  • $z$ indexes the $Z$ slices of the volume.
  • $y_z$ is slice $z$'s acquired k-space.
  • $y^{\text{fit}}_{z-1:z+1}$ are the fitting-set lines of slice $z$ and its two neighbours.
  • $\mathcal{A}_z$ is slice $z$'s measurement operator (coil maps, Fourier transform, sampling mask).
  • $M^{\text{fit}}_z$ selects slice $z$'s fitting lines.
  • $\theta$ are the frozen source-trained weights.

Early stopping uses the analogous loss on held-out lines, summed over the volume. Pooling over $Z$ slices gives roughly $Z$ times more self-supervision per free parameter than per-slice adaptation. It also rules out slice-to-slice drift, because the same correction is applied everywhere. This is the mechanism by which we expect both accuracy and continuity to improve.

Second ingredient: cross-slice module. The second ingredient is a cross-slice module inside each cascade: slice attention in the spirit of CSAM, restricted to a window of adjacent slices to keep cost linear in volume depth. It lets the refinement step use neighbouring slices' partially sampled content. When neighbouring slices use shifted undersampling masks, which is common in multi-slice protocols and easy to impose retrospectively, a neighbour often carries measured lines that the current slice lacks.

Amortized adaptation. After the optimization-based version works, we amortize it. A small encoder reads the undersampled, fully sampled-centre k-space of all slices, together with scan metadata when available. It outputs $\phi$ directly. Training uses episodes built from multiple source sites: in each episode, the encoder's predicted $\phi$ is supervised by the $\hat\phi$ obtained by optimization on that volume, plus the downstream reconstruction loss. At deployment this costs one extra forward pass per volume. The optimization-based version remains a fallback when the encoder's predicted $\phi$ yields a held-out k-space error above a threshold calibrated on source data. That held-out error is available at test time without ground truth, so it serves as a safety check.

Baselines

Unadapted 2D network. The naive reference is an E2E-VarNet trained on source data and applied unchanged to the target scanner. This defines the lower end of the performance gap. The upper end is the same architecture trained on target data, following the gap definition of Darestani et al. For context, the file reports an SSIM of 0.890 for E2E-VarNet at 8× on the fastMRI multi-coil knee test set, in-distribution (E2E-VarNet).

Per-slice test-time training. The strongest existing competitor on accuracy is per-slice test-time training on VarNet from Darestani et al. It reports closing 92% to 99% of the gap on VarNet, and 87.4% for the NYU-to-Stanford knee dataset shift with a U-Net. It costs several minutes per slice. We rerun it with the same source-trained weights, the same held-out split and the same early-stopping rule, on the same target volumes. Beating it means one of two things: matching its gap closure within two percentage points at 20× or greater lower wall-clock time per volume, or exceeding it on through-plane consistency at equal time.

Per-volume full-weight test-time training. This simpler hybrid adapts all weights once per volume with the pooled loss. It isolates whether the small shared parameter set matters, or whether pooling alone suffices.

Trajectory-level robustness training. The strongest training-only competitor for scanner shift is the k-space perturbation training of Wattad et al.

For Cartesian sampling, which is our main setting:

  • The file reports a gain of only +0.176 dB for adversarial trajectory noise with a Vision Transformer (ViT).
  • It reports up to 4.376 dB for learned trajectories, but only under radial sampling.
  • We include their adversarial trajectory noise on our backbone as a baseline.

Multi-site training. We train an equal-size E2E-VarNet on all available source sites with k-space augmentation. This is the most important check on the amortized encoder.

Ablations of the key ingredients. We remove the cross-slice module, with adaptation on and off. We also apply cross-slice context before rather than after adaptation, to test the ordering claim.

Tuning and compute. All methods receive the same source data, the same hyperparameter search budget of 20 configurations tuned on a held-out source-site split, and the same GPU. Test-time training methods get an equal cap on gradient steps per volume in a separate matched-compute comparison.

Why not just…?

Why not just run per-slice test-time training, which already closes up to 99% of the gap? It is the accuracy ceiling we must approach, and on accuracy alone it may win. Its weakness is minutes per slice, which multiplies to hours for a volume, together with independent per-slice updates that nothing ties together along the slice axis. The matched-compute comparison settles this. If per-slice adaptation truncated to the same wall-clock budget as ours matches our accuracy and continuity, the volume-sharing idea adds nothing, and we would report that.

Why not just reconstruct 2D slices and smooth or denoise along the slice axis afterwards? Post-hoc 3D filtering changes voxels without re-checking them against the measured k-space. Ongie et al. (Deep Learning Techniques for Inverse Problems in Imaging) warn that this breaks data consistency. The 2020 challenge radiologists (Results of the 2020 fastMRI Challenge) penalized over-smoothed 8× reconstructions. The approach also does nothing about the scanner shift itself. We include a 3D denoiser after the unadapted VarNet so that the continuity metric can show whether smoothing buys continuity only at the cost of detail and k-space residual.

Why not just train on many sites with physically grounded k-space augmentation and skip adaptation entirely? This is the real threat to the amortized encoder, because an encoder conditioned on the volume could collapse into a multi-site model with extra parameters. Wattad et al.'s small Cartesian gain (+0.176 dB) suggests augmentation alone leaves much of the gap. Our baseline is nonetheless trained on more sites than theirs. The decisive test is on vendors held out from all source sites. If the multi-site model matches the encoder there, we drop the encoder and keep only optimization-based per-volume adaptation.

Why not just use a full 3D network? Native 3D convolutions are memory-heavy and blur across thick slices. CSAM reports 37.0% versus 51.0% Dice for 3D nnU-Net versus 2.5D attention on anisotropic prostate data. That result is for segmentation, not reconstruction, so we include a patch-based 3D refinement variant at matched parameter count rather than assume the result transfers.

Proposed Experiments

Datasets. - The source domain is fastMRI brain, which the file describes as predominantly Siemens 1.5T and 3T scans. - Target domains are CC359, which covers GE, Philips and Siemens at 1.5T and 3T with raw multi-channel data, and M4Raw. We use the same fastMRI-brain-to-M4Raw pairing as Wattad et al., so their setting is directly comparable. - For knees, we use the NYU-to-Stanford shift studied by Darestani et al. - The fastMRI 2020 transfer-track GE/Philips test set would be ideal if accessible; since its ground truth is hidden, we treat it as optional.

Undersampling is Cartesian at 4× and 8×. We use both identical and slice-shifted masks across neighbouring slices.

Reconstruction and gap metrics. We report per-slice SSIM, PSNR and NMSE, and the fraction of the distribution-shift gap closed.

Through-plane continuity. We measure SSIM and gradient error computed on sagittal and coronal reformats against the fully sampled reference. Slice-to-slice intensity jumps show up there and not in per-slice metrics.

Latency. We report wall-clock time per volume.

Hallucination. We report held-out k-space residual as a label-free check. Following the challenge's hallucination concerns, a radiologist-blinded read of a small subset is desirable but secondary.

No reported result was found on Lacuna for any cross-scanner CC359 or M4Raw evaluation with VarNet-class models under test-time training. We will therefore establish those numbers ourselves.

Decisive check 1: accuracy versus speed. On CC359 held-out vendors at 8×, per-volume shared adaptation should reach at least 85% gap closure. It must do so in under ten seconds per volume, compared with minutes per slice for per-slice test-time training.

Decisive check 2: continuity. Reformat continuity error should be lower than per-slice test-time training, because a shared correction cannot drift between slices.

Decisive check 3: the ordering prediction. With identical masks, adding cross-slice context without adaptation should give no benefit or should hurt out of distribution. After adaptation, it should help. With shifted masks, it should help more. If cross-slice context helps equally before and after adaptation, the ordering hypothesis is wrong. If it never helps, Wattad et al.'s measurement-domain view prevails for reconstruction.

Decisive check 4: dependence on depth. Plotting gap closure against the number of slices pooled, from 1 to the full volume, should show gains saturating once the shared parameters are well determined. Failure to improve beyond a single slice would refute the central premise that scanner shift is a volume-level nuisance.