Accelerated MRI scans faster by measuring only a fraction of the frequency data, called k-space. The missing part has to be filled in by a learned model. The best models are trained on fully sampled scans, which do not exist for many anatomies, contrasts and patient groups. Self-supervised methods avoid this by splitting the measurements a scan already has into two halves: they reconstruct from one half and check the result against the other. They have a blind spot, though. Neither half contains the frequencies that were never measured, so structured aliasing in those frequencies costs the loss nothing. Aliasing here means coherent ghost copies and streaks that fold across the whole image. At eight-fold acceleration and above, this residual aliasing is exactly what these methods fail to remove.
We propose to close that blind spot with a prior borrowed from somewhere else, while still training without paired targets. A generator is a network that maps a compact code (a "latent") to an image. We train one on clean scans from a related domain where fully sampled data exist, then freeze it. On the target domain, where no clean scans are available, we train a single-pass encoder that maps an undersampled scan to a latent code for that generator. The encoder is supervised only by measurement splitting. A learned per-pixel gate decides where to trust the generator's output and where to trust a direct, measurement-faithful estimate. Aliasing is global, so the encoder must see the whole image to recognize a ghost as an artifact. We therefore build it as a hybrid, with convolutions at full resolution and attention at coarse resolution. The practical question is simple: can a borrowed manifold, reached in one forward pass, remove the null-space aliasing that splitting alone leaves behind, in milliseconds rather than the 85 to 640 seconds per slice of scan-specific methods?
Core Claim
Consider the regime with no fully sampled target data and acceleration of at least 8x. Here, a hybrid convolution–attention encoder that inverts into a frozen generative prior from a related domain, fused with a direct estimate through a learned reliability gate, will reconstruct the unmeasured k-space more accurately than the same hybrid network trained with the same splitting loss but without the prior. It will also run in one forward pass.
The gain should appear specifically in the null space of the sampling operator, meaning the k-space locations that were never measured. It should shrink as acceleration drops, because at low acceleration there is little null space left to fill. It should vanish if the encoder is restricted to local receptive fields. Each of these three predictions can fail, and each has its own experiment below.
Where This Sits
The feed-forward inversion literature supplies the mechanism. Earlier work on latent regression in GANs (Using latent space regression to analyze and leverage compositionality in GANs) showed that a regression encoder projects corrupted inputs onto the nearest valid point of a generator's image manifold. That work acts as a structural regularizer rather than fitting the corruption. GLEAN (GLEAN: Generative Latent Bank for Large-Factor Image Super-Resolution) uses this in a single pass, with a convolutional encoder that produces both latent vectors and multi-resolution feature maps to anchor a frozen StyleGAN. It reports 26.84 dB PSNR on 16x CelebA-HQ super-resolution, against 21.83 dB for iterative inversion (PULSE). We reuse GLEAN's pattern of latent codes plus spatial features. What we do not reuse is its supervision: GLEAN needs clean high-resolution pairs and has no physics operator.
BIGPrior (BIGPrior: Towards Decoupling Learned Prior Hallucination and Data Fidelity in Image Restoration) supplies our fusion rule. It mixes a GAN-inversion hallucination with a data-fidelity estimate through a learned pixel-wise map, and trains that map with a "guide-free" bias toward the data. Its inversion is iterative, however, taking minutes per image, and its authors name a feed-forward encoder as an unaddressed goal. We amortize that step and train it with k-space splitting instead of image-space objectives.
The self-supervised MRI literature supplies the training signal and the evidence for the gap. SSDU (Self-Supervised Physics-Based Deep Learning MRI Reconstruction Without Fully-Sampled Data) introduced the split into a data-consistency set $\Theta$ and a loss set $\Lambda$. Multi-mask SSDU (Multi-Mask Self-Supervised Learning for Physics-Guided Neural Networks in Highly Accelerated MRI) resamples the split several times per scan to reduce residual aliasing at 8x. ZS-SSL (Zero-Shot Self-Supervised Learning for MRI Reconstruction) trains per scan at high latency. All three use purely convolutional unrolled regularizers. None of them has any mechanism that constrains frequencies absent from both $\Theta$ and $\Lambda$.
MO-EI (Benchmarking Self-Supervised Methods for Accelerated MRI Reconstruction) attacks the same null space from a different angle. It adds virtual transforms such as rotations and warps, so that different "views" of the same anatomy reveal different frequencies. MO-EI is therefore our most serious conceptual competitor. It constrains the null space through invariance; we constrain it through a learned manifold. SSDiffRecon (Self-supervised MRI Reconstruction with Unrolled Diffusion Models) brings a generative model into self-supervised MRI. It is still a five-step sampler working in image space, however, with no explicit compact latent.
HUMUS-Net (HUMUS-Net: Hybrid Unrolled Multi-scale Network Architecture for Accelerated MRI Reconstruction) supplies the encoder design. It keeps full-resolution convolutional features and applies windowed attention with 1x1 patches only after convolutional downsampling. At a matched budget of about 2.4M parameters, its denoiser beat a U-Net (0.9378 vs 0.9348 SSIM on Stanford 3D FSE). The pure ViT of Vision Transformers Enable Fast and Robust Accelerated MRI constrains our design, together with the low-data findings of A Guide to Robust Generalization: The Impact of Architecture, Pre-training, and Optimization Strategy. Both indicate that convolution-free models depend on large-scale pre-training, which argues against a pure-attention encoder when target data are scarce. HUMUS-Net itself is trained fully supervised and never touches a latent space.
Research Program
Components. The system has three parts. The first is a generator $G$, trained on fully sampled complex-valued (two-channel) images from a source domain and then frozen. The second is a hybrid encoder $E$. The third is a fusion-and-consistency head.
Encoder design. The encoder follows the HUMUS-Block layout: convolutional stems at full resolution, then Swin-style windowed attention over 1x1 tokens at 1/4 and 1/8 resolution, then a convolutional decoder. We choose windowed attention over the criss-cross routing of CCNet: Criss-Cross Attention for Semantic Segmentation only because the MRI evidence exists for the windowed version. A criss-cross variant is a cheap secondary ablation, since its row-and-column paths match Cartesian aliasing geometry closely.
Encoder outputs. From the zero-filled image of the $\Theta$ measurements, the encoder emits three things. The first is a latent code $w$ plus coarse spatial features that condition $G$, following GLEAN. The second is a direct image estimate $h$. The third is a gate $\phi \in [0,1]$ per pixel.
Reconstruction and loss. The reconstruction applies one hard data-consistency projection $D_\Theta$ to the fused image:
$$\hat{x} = D_\Theta\big(\phi \odot h + (1-\phi) \odot G(w)\big).$$
Here $D_\Theta$ replaces the k-space values at the indices in $\Theta$ with the measured ones. Training minimizes the multi-mask splitting loss plus a BIGPrior-style fidelity bias:
$$\mathcal{L} = \frac{1}{K}\sum_{k=1}^{K} \big\|P_{\Lambda_k} A(\hat{x}_k) - y_{\Lambda_k}\big\|_1 + \lambda\, \overline{(1-\phi)}.$$
In this loss, $A$ is the coil-sensitivity-weighted Fourier operator. $P_{\Lambda_k}$ selects the held-out indices of the $k$-th split, and $\hat{x}_k$ is the reconstruction from the complementary set $\Theta_k$. The term $\overline{(1-\phi)}$ is the spatial mean reliance on the prior, and $\lambda$ sets how strongly the gate defaults to the measurement-driven estimate. We start from the reported multi-mask settings of $K=7$ and split ratio $\rho=0.4$.
What the gate buys. The gate makes the question falsifiable inside a single model. Setting $\phi \equiv 1$ recovers a self-supervised hybrid network with no prior. Setting $\phi \equiv 0$ recovers pure latent projection, which we expect to hallucinate source-domain anatomy.
Transfer settings. We study two. In the near setting, the generator is trained on fastMRI knee coronal proton-density scans, and the target is fat-suppressed knee scans with fully sampled data withheld. In the far setting, the generator is trained on fastMRI knee and the target is fastMRI brain, which matches the knee→brain protocol where ZS-SSL-TL reports its result. Images are cropped to 320x320 as in the fastMRI benchmark (fastMRI: An Open Dataset and Benchmarks for Accelerated MRI), because the generator needs a fixed size. We state this rigidity as a limitation rather than solve it.
Latency. Inference is one encoder pass, one generator pass and one data-consistency projection.
Baselines
The naive reference points are zero-filled reconstruction and classical total-variation compressed sensing. fastMRI reports that even a basic U-Net improves normalized mean squared error by 40–50% relative to total variation. Beating these points establishes only that the pipeline works.
The primary ablation, and the most important baseline, is the same hybrid encoder–decoder with the gate fixed at one and no generator. It has identical architecture, splitting loss, data, epochs and hyperparameter search budget. This is the "HUMUS-Block trained with multi-mask SSDU" system that a reviewer would propose. It has no published number because it has not been built, so we build it. Beating it means lower error on unmeasured k-space at R=8 and above in both transfer settings, with the gap larger than the seed-to-seed spread across three seeds.
A second ablation replaces the hybrid encoder with a parameter-matched purely convolutional encoder while keeping the generator and gate. This isolates the receptive-field claim.
The strongest self-supervised competitor is MO-EI, run through its public DeepInverse implementation on our exact splits and masks. Its reported number is 32.14 dB PSNR on fastMRI brain at 6x single-coil, against 33.15 dB for a supervised oracle (Benchmarking Self-Supervised Methods for Accelerated MRI Reconstruction). We treat the 1.01 dB gap to the oracle as the room available, and count closing at least half of it at matched acceleration as a clear win.
Multi-mask SSDU with its original convolutional unrolled network is run as published; its reported number is SSIM 0.81–0.83 on 8x knee. ZS-SSL-TL is the latency-unconstrained upper reference, reported at 37.1 dB PSNR on knee→brain transfer at 85 s per slice. For it, we compare quality per second of compute rather than claim to beat it on quality. SSDiffRecon is reported only relative to its baselines: +2.55 dB PSNR and +1.96% SSIM over the second-best self-supervised method on IXI. We rerun it on our splits and report both quality and wall-clock time.
Finally, a supervised hybrid network trained with target ground truth gives the oracle ceiling. All methods share preprocessing, masks, coil-sensitivity estimates and a 50-trial tuning budget.
Why not just…?
Why not just put a hybrid backbone into multi-mask SSDU and skip the generator? This is the real threat. It may already remove much of the global aliasing, because attention alone lets the network recognize coherent folds. We do not argue it away: it is our gate-fixed-at-one ablation, run under identical conditions. If it matches the full model on null-space error, the prior is unnecessary and we will report that. The learned $\phi$ maps also give direct evidence: if they sit near one everywhere after training, the prior is contributing nothing.
Why not just fine-tune a supervised source-domain network on the target with the splitting loss? This is feed-forward and uses the same source data, so it is a fair alternative way of borrowing knowledge. The difference is where the source knowledge lives. Fine-tuning lets the splitting loss overwrite the source weights, including in the null space the loss cannot see. A frozen generator keeps that knowledge fixed and lets the gate decide where it applies. We run this as an additional baseline. The prediction that separates the two is that fine-tuning drifts back toward residual aliasing as target training continues, while the frozen-prior model does not.
Why not just use MO-EI's invariance losses with the hybrid backbone? Invariance to rotations and warps reveals new frequencies only when the anatomy's distribution is actually invariant to those transforms. That is a weak assumption for knee and brain orientations under Cartesian sampling. Still, MO-EI's small gap to the oracle shows it is strong. Its loss and ours are compatible, so we also train our model with the MO-EI term added. If the combination beats both parts, the two priors are complementary; if MO-EI alone matches it, the latent path is not worth its complexity.
Why not just invert iteratively into the generator at test time? Iterative inversion fits the measurements more tightly per scan. The GLEAN comparison, however, shows that PULSE loses both fidelity (21.83 vs 26.84 dB) and identity (ArcFace 0.4047 vs 0.9678) on faces, and BIGPrior's iterative step takes minutes. We run gradient-based latent inversion under the same gate as a reference, to measure how much accuracy amortization costs.
Proposed Experiments
Test 1: where the gain appears. Our primary metric is normalized error restricted to unmeasured k-space. This is the part of the reconstruction the splitting loss never supervises, so it is where the claim lives. We also report PSNR, SSIM and NMSE against the withheld fully sampled images. We run R = 4, 8 and 12. The first decisive check is the interaction: the full model's advantage over the gate-fixed-at-one ablation should grow with R. A constant or absent gap falsifies the null-space story.
Test 2: whether it needs a global view. The convolutional-encoder variant should lose most of the prior's benefit on images with long-range folding. We measure this separately on slices sorted by the spatial extent of their aliasing in the zero-filled image.
Test 3: whether the prior invents or erases anatomy. Hallucination is the risk any borrowed prior carries. We use the pathology boxes from fastMRI+: Clinical Pathology Annotations for Knee and Brain Fully Sampled Multi-Coil MRI Data to compute error inside lesions separately from error elsewhere. We also check that $\phi$ rises toward one in lesions. A model that wins on global SSIM but loses inside lesions has failed the clinical test, whatever its average score.
Test 4: domain distance. Comparing the near and far settings shows how quickly the benefit decays as source and target diverge. In the far knee→brain setting we expect the gate to suppress the generator broadly; that outcome would itself confirm that the gate is working as intended.
Latency and throughput. We report wall-clock time per slice alongside quality, next to ZS-SSL-TL's 85 s per slice and SSDiffRecon's five-step sampler. We also report throughput in images per second, in the manner of the ViT study's 97.4 img/s figure, on the same GPU for all methods.
References
- Using latent space regression to analyze and leverage compositionality in GANs
- GLEAN: Generative Latent Bank for Large-Factor Image Super-Resolution
- BIGPrior: Toward Decoupling Learned Prior Hallucination and Data Fidelity in Image Restoration
- Self-Supervised Physics-Based Deep Learning MRI Reconstruction Without Fully-Sampled Data
- Multi‐mask self‐supervised learning for physics‐guided neural networks in highly accelerated magnetic resonance imaging
- Zero-Shot Self-Supervised Learning for MRI Reconstruction
- Benchmarking Self-Supervised Methods for Accelerated MRI Reconstruction
- Self-Supervised MRI Reconstruction with Unrolled Diffusion Models
- HUMUS-Net: Hybrid Unrolled Multi-scale Network Architecture for Accelerated MRI Reconstruction
- Vision Transformers Enable Fast and Robust Accelerated MRI
- A Guide to Robust Generalization: The Impact of Architecture, Pre-training, and Optimization Strategy
- CCNet: Criss-Cross Attention for Semantic Segmentation
- fastMRI: An Open Dataset and Benchmarks for Accelerated MRI.
- fastMRI+: Clinical Pathology Annotations for Knee and Brain Fully Sampled Multi-Coil MRI Data