Accelerated MRI rebuilds an image from far fewer frequency-space samples than the image has pixels. It works much better when the reconstruction can lean on "side information," meaning a second, already-acquired image of the same anatomy (for example, a proton-density scan guiding a fat-suppressed scan). The catch is data. Networks that learn to use side information need paired multi-contrast raw data, which is scarce. The fastMRI knee subset used to train such a network has 428 paired volumes, against roughly 1,600 single-contrast knee volumes overall. The natural fix is to synthesize the missing partner contrast and train on synthetic pairs. This proposal argues that this fix fails in a specific, predictable way. Side information synthesized from the target image agrees with the target too well, so a network trained on it learns to over-trust side information and then does worse than using no side information at all when the real partner scan is misregistered or differs in content.
We propose to measure this failure in a unit that matters for scanning: how many measurements the side information saves. We also propose to fix it with two ingredients. The first is a simulator that deliberately injects controlled disagreement between the side image and the target. The second is joint (multi-task) training that keeps a working no-side-information reconstruction path alive inside the same network. Put in two sentences: synthetic partner scans are worth something only if they reproduce the real rate of disagreement with the target, and their worth should be priced as saved k-space samples, not as synthetic images per real image. We test this first in a sparse-recovery setting where the price can be predicted from geometry, then on fastMRI and fastMRI+ with pathology-level checks.
Core Claim
Compressed sensing with side information has a clean theory for when a reference signal helps. Compressed sensing with side information: Geometrical interpretation and performance bounds studies the objective
$$\min_x \|x\|_1 + \beta\|x - w\|_1 \quad \text{subject to} \quad Ax = y,$$
where $x$ is the unknown sparse signal, $w$ the side information, $A$ the measurement matrix and $y$ the measurements. It shows that the number of measurements needed depends on the count $h$ of "bad" components, the entries where $w$ disagrees with $x$. It also shows that bad components never push the requirement above that of plain compressed sensing: 136 measurements instead of 472 at $n=1000$, $s=70$, while a squared-distance penalty needs 666. This "no worse than no prior" property is exactly what learned side-information networks do not obviously have.
Our claim has three parts. First, synthetic side information generated from the target has a systematically smaller $h$, and differently structured bad components, than real side information. Second, a learned reconstructor calibrated to small $h$ loses the no-worse-than-nothing property at real $h$. Third, both the value and the risk of synthetic side information are best reported as a measurement exchange rate. We define that rate as
$$\rho = \frac{R_{\mathrm{SI}}}{R_{0}},$$
where $R_0$ is the acceleration factor at which a no-side-information reconstructor reaches a reference quality $Q$, and $R_{\mathrm{SI}}$ is the acceleration at which the side-information reconstructor reaches the same $Q$ on real test pairs. A value $\rho > 1$ means the side information buys scan time. A value $\rho < 1$ means it costs scan time.
This gives the empirical "exchange rate" idea of On the Equivalency, Substitutability, and Flexibility of Synthetic Data a physical unit. That paper measured 30 synthetic tracking frames per real frame and 60–80% replaceable real data, with no analytic grounding. In the sparse testbed, the Gaussian-width theory of Mota et al. predicts $\rho$ from $h$, so the exchange rate stops being purely empirical there.
Relation to Prior Work
Two strands supply the reconstruction machinery. A Trust-Guided Approach to MR Image Reconstruction with Side Information (TGVN) builds an unrolled variational network, a network whose layers mimic iterations of an optimization algorithm. It restricts side information to the "ambiguous space" of the scanner's forward operator, the image content the measurements cannot determine, and reaches 84.92% SSIM at 20x on fastMRI knee against 81.33% for E2E-VarNet. TGVN trains and tests only on real pairs. We reuse its architecture and ask what happens when most pairs are synthetic.
That restriction also sharpens our concern rather than removing it. The ambiguous space is precisely where the measurements cannot veto wrong side information, so over-trusted synthetic side information can insert structure there unopposed. MRI Reconstruction with Side Information using Diffusion Models learns a joint prior over two echoes and reaches SSIM 0.878 at 3x on SKM-TEA, but it runs about 400x slower than TGVN. We treat it as a reference point, not a main baseline.
A second strand motivates the gap. Multi-Contrast MRI Segmentation Trained on Synthetic Images synthesizes missing contrasts with a conditional generative adversarial network (GAN) and finds under 1–2% Dice loss for segmentation. That is evidence synthetic contrasts can be useful, but downstream of reconstruction rather than inside it. Results of the 2020 fastMRI Challenge and fastMRI+ show that high SSIM (around 0.964) coexists with hallucinated vessels and erased meniscus tears. That constrains our evaluation: pixel metrics alone cannot tell us whether synthetic side information is safe.
The classical side-information literature constrains the design of the fix. Modified-CS and Regularized Modified BPDN show that erroneous support or value priors cause inexact recovery, and they model prior errors as static and additive. Performance analysis of joint-sparse recovery with prior information and Exact Recovery Conditions with Partial Support Information show that support knowledge matters more than value knowledge. That tells us to parameterize the simulator's errors by where they fall, not only by how large they are.
Multi-task learning supplies the training mechanism. Assisting Main Task Learning by Heterogeneous Auxiliary Tasks and Deep Multitask Representation Learning argue for concurrent over sequential training. RotoGrad shows that conflicting task gradients cause negative transfer and can be reduced by rotating shared features. RotoGrad has never been tested on physics-constrained reconstruction; we use it only as an ablation. Multi-task compressive sensing with Dirichlet process priors is the closest existing bridge between multi-task learning and compressed sensing. It clusters tasks by shared sparsity support, and on its synthetic signals global sharing errors exceed 50% while clustered sharing stays near 10%. It shares support across exchangeable real tasks and never asks what a synthetic prior is worth, so it addresses a different question.
Research Program
The work has three pieces.
Sparse testbed. We first build a controlled testbed where the answer can be predicted. Using $n=1000$, $s=70$ and Gaussian $A$, we reproduce the 472/136/666 measurement counts of Mota et al. We then generate side information from a "simulator" whose bad components number $h_{\mathrm{sim}}$ and follow a chosen pattern: random entries, or clustered support errors. The deployment side information has $h_{\mathrm{real}} > h_{\mathrm{sim}}$.
On this testbed we train an unrolled $\ell_1$-$\ell_1$ network, a learned iterative thresholding solver with learned $\beta$ and step sizes, using simulator pairs. We compare its measured phase transition against the Gaussian-width prediction at $h_{\mathrm{real}}$ and against classical $\ell_1$-$\ell_1$, which keeps the no-worse-than-plain-CS guarantee. This tells us whether learned solvers lose the guarantee, how $\rho$ scales with the mismatch between $h_{\mathrm{sim}}$ and $h_{\mathrm{real}}$, and whether our fixes restore it.
Knee MRI with real targets and synthetic partners. We then move to fastMRI knee. The target is always a real fat-suppressed volume with real raw k-space, so the measurement physics is never simulated. Only the proton-density partner image is synthetic. We use two synthesizers. One is procedural: tissue-class intensity remapping with randomized contrast parameters, in the spirit of the parameterized simulation of Chang et al. The other is a pix2pix-style generator trained on a small held-out set of real pairs, following Irmakci et al.
Each synthesizer runs in two modes. In the "faithful" mode, the partner image is derived directly from the target. In the "disagreement-calibrated" mode, the simulator injects misregistration, through-plane offset, local contrast errors and occasional missing structures. Their rates are fitted so that the distribution of target-versus-side-image disagreement, measured in wavelet coefficients as a stand-in for $h$, matches that measured on real training pairs. This calibration is a small version of the "simulation Turing test" of Is this Model for Real?: we check that disagreement statistics, not image appearance, are indistinguishable from real data.
Joint training with a fallback task. The shared TGVN-style network is trained on three streams at once. The first is real single-contrast reconstruction with the side-information input blanked, using all single-contrast volumes. The second is side-information reconstruction on real-target/synthetic-partner pairs. The third is side-information reconstruction on whatever real pairs are available. The first stream is the multi-task ingredient that matters. It forces the shared layers to keep a competent no-side-information path, so that unreliable side information can be ignored rather than followed. We log the cosine between stream gradients in the shared layers to see whether the streams conflict. RotoGrad is applied only if they do.
Baselines
The do-nothing reference is E2E-VarNet trained on all real single-contrast fastMRI knee data with no side information. It defines $R_0$ in the exchange rate, and TGVN's paper reports 81.33% SSIM for it at 20x. Every method is scored against it on the same real test pairs, at the same accelerations and sampling masks.
The strongest existing method is TGVN trained on real pairs only, reported at 84.92% SSIM at 20x in A Trust-Guided Approach to MR Image Reconstruction with Side Information. We retrain it on our split of the 428-volume paired set, so that our number and theirs share a protocol, and we report our reproduced value next to theirs. Our method beats it if, with the same architecture, epochs and hyperparameter search budget, it matches TGVN's SSIM using a markedly smaller fraction of real pairs. It also beats it if it exceeds TGVN at full real data while holding the safety floor described below.
We run two strong training alternatives with identical compute. The first pretrains on synthetic pairs and then fine-tunes on real pairs. The second mixes real and synthetic pairs in a single stream without the no-side-information task. Two ablations isolate our ingredients: removing disagreement calibration (faithful synthetic partners), and removing the fallback task.
For the classical reference we run $\ell_1$-$\ell_1$ reconstruction in the wavelet domain with the synthetic partner as $w$. It keeps a worst-case guarantee in the Gaussian setting, but it may not keep it under Fourier sampling. The file's only related evidence is indirect: a U-Net improved NMSE 40–50% over total variation in fastMRI. On the testbed, the Gaussian-width prediction serves as a baseline in its own right. No reported result was found on Lacuna for any side-information reconstructor trained on synthetic partner scans, so those numbers will be new.
Why not just…?
Why not just train TGVN on the real pairs you have? That is the right default whenever paired data is plentiful. Our substitutability curve tests exactly this. We sweep real pairs from 0% to 100% of the training split, with and without synthetic pairs. If real-only TGVN already saturates at small fractions, the synthetic pipeline is unnecessary, and we will report that.
Why not just pretrain on synthetic pairs and fine-tune on real ones? This is the most serious threat. The file's evidence against it comes from other domains (Supervised Pretraining Can Hurt Generalization, Situ et al.) and does not settle it for reconstruction. We run it with the same compute. If it matches joint training on SSIM, on the safety floor and on fastMRI+ lesion fidelity, then our contribution narrows to disagreement calibration and the exchange-rate measurement. We would say so.
Why not just augment the real pairs with random misregistration instead of synthesizing new ones? Augmentation teaches robustness to disagreement but adds no new anatomy. We include it as a baseline. If augmented real-only training matches our method, synthetic partners add nothing beyond disagreement modeling, and that would itself be a useful negative result.
Why not just use classical $\ell_1$-$\ell_1$, which already cannot be hurt by bad side information? Its guarantee is proved for Gaussian measurements, while MRI samples frequencies. Its reconstructions are also likely far weaker than learned ones at 20x. We report it anyway, because if its safety floor holds on Fourier data where learned models' floors fail, that is an informative finding.
Proposed Experiments
Testbed experiment. We plot measured versus predicted measurement counts as $h_{\mathrm{real}}-h_{\mathrm{sim}}$ grows, for classical, learned, and learned-with-fallback solvers. The expected, falsifiable result has three parts. Classical $\ell_1$-$\ell_1$ never exceeds 472 measurements. The learned solver trained at small $h_{\mathrm{sim}}$ crosses above 472 at some mismatch. The fallback task keeps it at or below 472.
MRI experiment. On fastMRI knee at 8x, 12x, 20x and 30x we report SSIM, NMSE, the exchange rate $\rho$, and the substitutability fraction, defined as the smallest real-pair fraction that matches full real-pair TGVN.
Safety-floor check. At test time we feed real side images that are deliberately wrong: shifted, taken from the wrong slice, or taken from a different patient. The margin by which performance drops below E2E-VarNet with no side information is the safety-floor violation. Faithful-synthetic training should violate it most, and calibrated joint training should violate it least.
Pathology check. This addresses the concern raised by Muckley et al. and Zhao et al. We run a pathology detector, trained on fully sampled images, on the reconstructions of fastMRI+ annotated knee studies that fall in our test set, and we report the size of that intersection. We measure detection recall inside the expert boxes relative to fully sampled images, and false detections outside them.
Replication. We repeat the core comparison on M4Raw low-field brain data, with T2-weighted scans guiding FLAIR, where no per-contrast SSIM figure was found on Lacuna.
Decisive outcome. The proposal succeeds if disagreement-calibrated synthetic partners with joint training reach $\rho > 1$ on real test pairs using well under the full real paired set, while faithful synthetic partners produce $\rho < 1$ on corrupted side images. It fails if the faithful and calibrated simulators are indistinguishable on the safety floor, or if sequential fine-tuning matches joint training everywhere.
References
- Compressed sensing with side information: Geometrical interpretation and performance bounds
- On the Equivalency, Substitutability, and Flexibility of Synthetic Data
- A Trust-Guided Approach to MR Image Reconstruction with Side Information
- MRI Reconstruction with Side Information using Diffusion Models
- Multi-Contrast MRI Segmentation Trained on Synthetic Images
- Results of the 2020 fastMRI Challenge for Machine Learning MR Image Reconstruction
- fastMRI+: Clinical Pathology Annotations for Knee and Brain Fully Sampled Multi-Coil MRI Data
- Modified-CS: modifying compressive sensing for problems with partially known support
- Regularized Modified BPDN for Noisy Sparse Reconstruction With Partial Erroneous Support and Signal Value Knowledge
- Performance analysis of joint-sparse recovery from multiple measurement vectors with prior information via convex optimization
- Exact Recovery Conditions for Sparse Representations With Partial Support Information
- Assisting Main Task Learning by Heterogeneous Auxiliary Tasks with Applications to Skin Cancer Screening
- Transfer Learning for Sequence Tagging with Hierarchical Recurrent Networks
- RotoGrad: Gradient Homogenization in Multi-Task Learning
- Multi-task compressive sensing with Dirichlet process priors
- Is this Model for Real? Simulating Data to Reveal the Proximity of a Model to Reality
- fastMRI: An Open Dataset and Benchmarks for Accelerated MRI.
- Can You Tell Me How to Get Past Sesame Street? Sentence-Level Pretraining Beyond Language Modeling