Held-Out Measurements as a Verifier for Scaling Test-Time Compute in MRI Reconstruction

Generative image models can now reconstruct an MRI scan from a fraction of the usual measurements. They are also known to invent things. At 8× acceleration, the 2020 fastMRI challenge found reconstructions that turned post-surgical resections into plausible-looking vessels and smoothed away meniscus tears (Results of the 2020 fastMRI Challenge). In language models, the standard response to hard problems is to spend more compute at inference time: sample many candidates and let a verifier choose. MRI seems to offer an ideal verifier for free, the physics. A candidate image can be pushed through the exact scanner model (Fourier transform, coil sensitivities, sampling mask) and compared to the measured data. Our starting observation is that this verifier is nearly blind exactly where hallucinations live. Modern samplers enforce agreement with the measured frequencies at every step, so every candidate matches the measurements almost perfectly. The differences between candidates, including invented anatomy, sit in the unmeasured frequencies, which the physics check never sees.

We propose to fix this by hiding part of the measurements from the generator and using them only to grade it. Each candidate is generated from a random subset of the acquired k-space lines, meaning the rows of spatial-frequency data the scanner actually recorded. It is then scored by how well it predicts the remaining acquired lines, which it never saw. This is cross-validation performed inside a single scan. It turns the physics operator from a constraint that every candidate trivially satisfies into an unbiased judge of what the generator filled in. We then ask whether spending more samples, more partitions, and more tournament comparisons under this judge gives monotone gains in fidelity and pathology preservation. The alternative is that it amplifies hallucination, as extended reasoning does in multimodal models.

Core Claim

Write the acquired measurements as $y = A_\Omega x + \epsilon$, where $x$ is the true image, $A_\Omega$ is the multi-coil Fourier operator restricted to the acquired line set $\Omega$, and $\epsilon$ is noise with variance $\sigma^2$. We split $\Omega$ into a generation set $\Gamma$ and a held-out set $\Lambda$. A candidate $\hat x$ is sampled from a diffusion model, a generator that turns random noise into an image by gradual denoising, with data consistency enforced only on $\Gamma$. The candidate is scored by

$$ s(\hat x) = \| A_\Lambda \hat x - y_\Lambda \|_2^2 , $$

whose expectation equals $\|A_\Lambda(\hat x - x)\|_2^2 + \sigma^2 |\Lambda|$. The expectation takes this form because $\hat x$ is statistically independent of the noise in $y_\Lambda$. The score is therefore an unbiased estimate of the candidate's error on real frequencies it had to guess.

The in-sample residual $\|A_\Omega \hat x - y_\Omega\|^2$ is different. Its value is set mostly by how hard data consistency was enforced, not by whether the unmeasured content is correct.

Our central claim has two parts. First, the held-out score is a pairwise judge with accuracy clearly above chance at ranking candidates by true error, while the in-sample residual is close to chance. Second, this difference decides whether test-time compute scaling works. Provable Scaling Laws for the Test-Time Compute of Large Language Models proves that knockout and league tournaments drive failure probability down with compute only when the pairwise judge beats random guessing ($p > 0.5$), and fail otherwise. We claim the in-sample physics residual is an instance of the $p \approx 0.5$ regime and held-out measurements move MRI into the $p > 0.5$ regime.

Relation to Prior Work

The nearest direct precedent is Inference-time Physics Alignment of Video Generative Models with Latent World Models. It scales best-of-N and gradient guidance for video generators using a learned world model (VJEPA-2) as the physics reward, and reports monotone gains. We share its framing of physical correctness as an inference-time selection problem. We differ in two ways. Our reward comes from real measurements of the specific object being imaged, not from a learned model with its own blind spots. We also study an ill-posed inverse problem where the ground truth is a particular patient, not general plausibility.

World Action Verifier motivates the design. It argues that checking a physical prediction is easier than generating it and uses inverse models to prune impossible robot rollouts. Our held-out check is the MRI version of that asymmetry, and it needs no extra learned models.

Two papers on the language side supply the scaling theory and the selection methods. Scaling Test-Time Compute Without Verification or RL is Suboptimal argues that verifier-based search exploits the chance of sampling rare, highly accurate candidates, and that imitation without verification cannot exploit it. The tournament paper above gives the judge-accuracy condition.

Two papers constrain the design and are central to it. Inverse Scaling in Test-Time Compute shows that more reasoning tokens can monotonically hurt accuracy. More Thinking, Less Seeing? shows longer reasoning shifting attention from visual evidence to learned priors, which increases hallucination. Both suggest that more sampling compute with a weak verifier could make MRI hallucination worse. This proposal is designed to test that directly.

On the reconstruction side, three lines of work spend inference compute in fixed amounts:

  • SPIRiT-Diffusion embeds multi-coil self-consistency into diffusion sampling.
  • i-RIM unrolls eight data-consistency iterations.
  • ELDA unrolls a provably convergent descent for CT.

None of them measures accuracy as a function of a variable compute budget, and none selects among candidates. We reuse SPIRiT-Diffusion as one of our generators.

The idea of splitting acquired k-space comes from self-supervised MRI training, such as Joint Supervised and Self-supervised Learning for MRI Reconstruction, which enforces consistency between acquisition subsets during training. We move the split from training to test time and use it as a verifier. As far as the material shows, that use is new.

Research Program

We will build a test-time search procedure around a fixed pretrained generator. Two generators will be used: a score-based diffusion prior trained on fully sampled fastMRI multi-coil knee and brain data (fastMRI), and SPIRiT-Diffusion. Generator weights are never updated at test time.

For each scan the procedure works as follows:

  1. Draw K random partitions of the acquired lines. The center of k-space always stays in the generation set, and roughly 10–20% of the acquired high-frequency lines go to the held-out set.
  2. For each partition, sample N candidates.
  3. Score every candidate on its partition's held-out lines.

We then compare four ways to turn the scored candidates into an image:

  • best-of-N selection;
  • a league tournament, in which each candidate is compared with a few others. Because partitions differ, comparisons use held-out lines common to both candidates' held-out sets.
  • a per-pixel weighted average whose weights decay with held-out score;
  • a final data-consistency projection onto all of $\Omega$, applied to any of the above, so that no measured line is discarded in the output.

The held-out scores also give a per-scan estimate of reconstruction error. The spread of the retained candidates gives a map of where the generator is unsure.

The analysis has three parts.

We first measure judge accuracy $p$ directly. We sample candidate pairs, record whether the held-out score and the in-sample residual rank them the same way as true error against the fully sampled reference, and plot $p$ against acceleration factor, held-out fraction, and anatomy.

We then trace scaling curves. Error, measured as normalized mean squared error (NMSE), peak signal-to-noise ratio (PSNR), and structural similarity (SSIM), is plotted against total denoiser evaluations. Compute is spent along three axes, and we report each one alone and in combination:

  • more candidates N;
  • more partitions K;
  • more reverse-diffusion steps T. This is the "think longer" axis, which could plausibly show inverse scaling.

Finally, we quantify one honest cost. A candidate generated from $\Gamma$ sees fewer lines than the full data, so it is effectively reconstructed at higher acceleration. We will measure how much this penalty is recovered by the partition ensemble and the final full-data projection.

Baselines

The naive reference point is a single posterior sample from the same diffusion generator, using all acquired lines and the same number of sampling steps. It shows what the generator does with no test-time search.

The strongest single-pass competitor is a trained unrolled network, i-RIM or an equivalent. It is the method designed for this benchmark and spends a fixed, modest amount of inference compute. i-RIM reports multi-coil 8× test-set NMSE 0.0093, PSNR 36.53, and SSIM 0.887, and multi-coil 4× NMSE 0.0052, PSNR 39.52, and SSIM 0.928. The fastMRI challenge report gives the top 4× SSIM as about 0.964 (Results of the 2020 fastMRI Challenge). The challenge test sets are hidden, so we will retrain or run an available i-RIM-class model on our own validation-derived split and compare on that split. We will not compare against the published numbers directly.

On the high-acceleration vessel-wall data, SPIRiT-Diffusion reports 41.30 dB PSNR at R=7.6, against 37.58 dB for SPIRiT and 39.37 dB for DDS (SPIRiT-Diffusion). Where that data is available, our search wrapped around SPIRiT-Diffusion must beat its single-run output at matched step count.

The ablation that isolates our key ingredient is best-of-N and league selection scored by the in-sample residual on all of $\Omega$, run with the same generator, the same N, and the same number of denoiser evaluations. If held-out scoring does not beat this, the proposal's central claim is false.

The uniform posterior mean of the same candidates is the ablation for the averaging variant. It uses identical compute and no scoring.

Total variation reconstruction is included as a physics-only reference. fastMRI reports that a U-Net gave a 40–50% relative NMSE improvement over it.

To keep comparisons fair, all search variants share the generator, test scans, masks, and a fixed budget of denoiser evaluations. Held-out fraction and weighting temperature are tuned on the validation split only. Beating a baseline means lower NMSE, and lower NMSE inside pathology boxes, at equal compute, with paired-bootstrap confidence intervals over volumes that exclude zero. For posterior-sampling methods, no reported result was found on Lacuna for best-of-N or tournament selection on fastMRI, so that curve has no published reference.

Why not just…?

Why not just average many posterior samples? This is the most serious threat. The posterior mean minimizes expected squared error, so on PSNR and SSIM a uniform average may match or beat any selection rule. The tournament paper's finding that majority voting fails under systematic bias (Provable Scaling Laws) suggests averaging should suffer when all candidates share the same prior-driven hallucination. It should also blur small focal lesions that only some candidates reconstruct. That is a hypothesis, not evidence for MRI. The uniform-mean ablation settles it in two ways: whole-image NMSE, and lesion-region metrics using fastMRI+ boxes and inserted synthetic lesions. If the uniform mean wins on both, the contribution shrinks to held-out scores as an error estimate and hallucination flag, and we will report it that way.

Why not just use the physics residual on all measured data as the verifier? It is the obvious physics verifier, and it is exactly our main ablation. Our argument that it is nearly uninformative rests on data-consistency steps forcing it toward the noise floor for every candidate. The judge-accuracy measurement tests this directly. If the in-sample residual turns out to rank candidates well, for instance at low acceleration where consistency is enforced loosely, the held-out split adds nothing there, and we will say so.

Why not just train a single better network, or a hypernetwork over regularization settings? HyperRecon amortizes the search over regularization strength into one forward pass taking about a quarter second. Unrolled networks like ELDA are fast and accurate. However, amortized models commit to one answer per input and give no per-scan evidence that the answer is right. The fastMRI challenge shows exactly this failure: fixed networks degrade on scanners from other vendors. The comparison against i-RIM-class networks answers whether search pays for its extra compute in the training distribution. The shifted-distribution experiment below answers whether it pays outside it.

Why not just train a learned quality predictor as the verifier? A learned verifier inherits the training distribution. Under the same vendor shift that breaks reconstruction networks, it is unlikely to stay calibrated. The world-model reward in Inference-time Physics Alignment has the same limitation, which its authors leave open. Held-out measurements come from the test scan itself and require no training. We will still include a small learned quality regressor, trained on in-distribution validation scans, as a verifier baseline in the shifted-distribution experiment.

Proposed Experiments and Decisive Checks

The main experiment is on fastMRI multi-coil knee and brain at 4× and 8× with equispaced masks. For each scan we report NMSE, PSNR, and SSIM against the fully sampled reference, as functions of denoiser evaluations, for every search variant and baseline.

Pathology preservation is measured in two complementary ways. First, we restrict error metrics to the lesion bounding boxes of fastMRI+, over 16,000 knee and 7,500 brain annotations. Second, we insert small synthetic lesions into fully sampled test images before undersampling and record whether each method preserves their contrast and extent. This mirrors the unseen-pathology test that made QSMnet convincing. As a hallucination proxy, we count structures present in a reconstruction but absent from the reference, using a thresholded difference map restricted to high-uncertainty regions.

Three checks decide the proposal.

The first is judge accuracy. Held-out scoring must rank candidate pairs correctly well above 50% at 8×, and the in-sample residual must not. If neither is above chance, the premise fails.

The second is the scaling curve. Under held-out selection, lesion-region error at 8× should fall monotonically with N and K. Increasing sampling steps T alone, with no selection, may flatten or reverse. That would be the MRI counterpart of Inverse Scaling in Test-Time Compute, and observing it would itself be a result.

The third is distribution shift. We train on knee data and test on brain, and train at 1.5T-like contrasts and test elsewhere where data permits, following the out-of-distribution protocol of SPIRiT-Diffusion. The file does not indicate that the multi-vendor challenge transfer set is publicly available. On these shifts we ask whether the gap between held-out search and the fixed unrolled network widens relative to in-distribution. That widening is where a verifier grounded in the test scan's own measurements should matter most.

A negative outcome on the third check would still be useful. It would show that test-time physics verification, like the learned reward it replaces, cannot get past the generator's prior when that prior is wrong.