Auditable Scribble Guidance for Joint MRI Reconstruction and Segmentation

When an MRI scan is accelerated by measuring only a fraction of its frequency data (called k-space), the image has to be filled in. A network trained jointly to reconstruct and segment the scan will fill the gaps partly with whatever helps the segmentation. That can help: anatomy knowledge sharpens a blurry boundary. It can also hurt, because the network may paint an edge that the scanner never saw, simply to satisfy the segmentation loss. The risk is worst when the segmentation labels are a few rough scribbles, which is the realistic regime once you accept that radiologists will not trace every voxel. This proposal builds a joint reconstruction-and-segmentation model that is trained only from raw undersampled k-space and sparse scribbles. The model is structured so that every change the segmentation makes to the image is held in a separate, inspectable correction map. By construction, that map can occupy only the frequencies the scanner did not measure.

The core claim is that this split can be made exact for Cartesian sampling, and that it is testable without any fully sampled ground truth. The correction lives on unmeasured k-space locations, so we can withhold some measured lines from training and ask a direct question: does the scribble-driven correction predict those held-out lines better or worse than the reconstruction without it? If better, the anatomy prior injected real information. If worse, it hallucinated. The model gives one exact image decomposition and a built-in hallucination test. It needs neither dense masks nor clean reference images for training.

Core Claim

Write the per-coil acquisition as $y = M F x + \epsilon$. Here $x$ is the per-coil image, $F$ the Fourier transform, $M$ the binary Cartesian sampling mask, and $\epsilon$ noise. Working per coil sidesteps sensitivity calibration, in the spirit of Joint Calibrationless Reconstruction and Segmentation of Parallel MRI. The part of the image the scanner cannot constrain is then exactly the content on unsampled frequencies, $P_N = F^{-1}(I-M)F$. The model outputs

$$\hat{x} = x_{\mathrm{rec}} + P_N\, c_\phi(z),$$

where $x_{\mathrm{rec}}$ is an unrolled, data-consistent reconstruction trained only with a k-space self-supervision loss, $z$ are features shared with the segmentation head, and $c_\phi$ is a semantic correction head. The correction head receives gradients only from the scribble segmentation loss and a sparsity penalty $\lambda \|P_N c_\phi\|_1$.

The segmenter reads $\hat{x}$. The returned "semantic correction map" $P_N c_\phi$ is therefore an exact additive account of what the labels changed. Because it is projected into the null space of the measurement, it can never contradict a measured sample.

This borrows its logic from Orthogonal projection-based regularization for efficient model augmentation. That work penalizes the component of a neural correction that lies in the span of a known physical model's Jacobian, so the network cannot "steal" dynamics the physics already explains. It was shown only on low-dimensional vehicle simulations. We use a hard projection rather than a penalty, because for Cartesian MRI the measured subspace is known exactly. Our claim also differs in kind: the separation is not mainly there to keep parameters identifiable. It turns the label-driven part of the image into an object that can be audited and falsified.

Research Program

The work builds on three prior lines, each in a different role.

Joint reconstruction-segmentation networks supply the architecture and motivate coupling the two tasks. Joint CS-MRI Reconstruction and Segmentation with a Unified Deep Network (SegNetMRI) shares encoders between a data-fidelity cascade and a U-Net segmenter. It reports gray-matter Dice of 86.38% and 34.27 dB PSNR on MRBrainS at 20% Cartesian sampling, against 75.55% for a traditional pipeline. Pramanik and Jacob show that reconstruction acts as a pretext task, so segmentation trains well with 10% of dense masks. Both still need dense masks, and both let segmentation gradients reshape the shared features with no record of what changed.

Scribble supervision constrains the design. Learning to Segment Medical Images with Scribble-Supervision Alone reaches 88.0% Dice on ACDC using random-walk seeds, conditional random fields and dropout uncertainty, but it assumes clean, fully reconstructed images. Revisiting 3D Medical Scribble Supervision: Benchmarking Beyond Cardiac Segmentation shows that elaborate scribble heuristics collapse outside cardiac data, while plain nnU-Net with a partial cross-entropy loss holds up (0.813 versus 0.369 mean Dice for ShapePU). We take that negative result seriously. The segmentation branch is an nnU-Net-style network with partial loss only, with no graph-cut or CRF machinery. Any gain must then come from the reconstruction coupling, not from heuristics.

The variational multi-task MRI work, Variational Multi-Task MRI Reconstruction: Joint Reconstruction, Registration and Super-Resolution, motivates the joint formulation. Its multi-reader study found radiologists preferred the joint reconstruction over sequential pipelines in more than 44% of cases. It does not use learning or labels.

Training proceeds in one stage. The reconstruction branch is trained by k-space splitting: a random subset of measured lines is hidden, and the network must predict it, so no fully sampled targets are needed. The segmentation branch is trained by partial loss on scribbles. The correction head sits between them.

We will study three gradient routings. In the first, segmentation gradients enter only the correction head. In the second, they also enter the shared trunk. The third has no correction head, which is plain joint training with hard data consistency. Separately, a fixed set of measured lines per scan is permanently held out, never used for any loss, as the audit set. The central analysis is the held-out-line error with and without the correction map. We will break it down by region (inside scribbled structures, at their boundaries, in background) and by acceleration factor.

Baselines

The naive reference point is the zero-filled inverse Fourier image, segmented by nnU-Net with partial loss on the same scribbles. It shows how much any reconstruction buys at all.

The strongest competitor is the cascade. It trains the same self-supervised unrolled reconstructor alone, then trains nnU-Net with partial loss on its outputs. This pairs the best label-free reconstruction we can build with the method the ScribbleBench paper found most robust. The reported number for the segmentation component is 0.813 mean Dice on ScribbleBench for nnU-Net+pL, from Revisiting 3D Medical Scribble Supervision, but on fully sampled images. No reported result was found on Lacuna for scribble-supervised segmentation from undersampled k-space. For reconstruction, the reference is the 22.79 dB SNR at 6× acceleration on Calgary Campinas reported by Pramanik and Jacob, though that figure used supervised training and 10% dense masks.

The second strong baseline is an unconstrained joint model: a SegNetMRI-style shared encoder with hard data consistency, trained with the same self-supervised k-space loss plus partial scribble loss.

All systems share one backbone, identical scribbles, identical masks and acceleration factors, and the same GPU-hour and hyperparameter-search budget. The search is tuned on a validation split using held-out-line error and scribble-pixel accuracy only, never dense masks.

Beating the cascade means higher dense-mask Dice at 6× acceleration and above, with held-out-line error no worse than the cascade's reconstruction. Beating the unconstrained joint model means matching its Dice while showing lower held-out-line error and, on scans with a fully sampled reference, fewer spurious edges.

The ablation that isolates the key ingredient keeps the correction head but removes the null-space projection $P_N$. If that ablation performs identically, the projection is doing nothing and the claim fails.

Why not just…?

Why not just reconstruct first and then segment with nnU-Net plus partial loss? This is the real threat. The ScribbleBench result shows simple segmenters are hard to beat, and at low acceleration the cascade may match us on Dice. Our bet is that the advantage appears only where the measurement genuinely leaves boundaries ambiguous. SegNetMRI's gap over sequential processing suggests coupling matters under undersampling, but that evidence used dense labels. The acceleration sweep settles this. If the cascade matches us at every factor, the joint coupling is not worth it. The remaining contribution would then be the audit protocol, which we would report as such.

Why not just rely on hard data consistency, which already confines network changes to unmeasured frequencies? Hard data consistency fixes the measured samples, but it mixes the generic image prior and the label-driven change into one output. Nothing says which unmeasured content came from the scribbles. The unconstrained joint baseline is exactly this system. Comparing its held-out-line error with that of our split model will show whether the separate head changes behavior or only bookkeeping.

Why not just audit post hoc by subtracting a reconstruction-only model's output from the joint model's output? This is cheap and plausible. However, the difference also contains seed-to-seed training variation and trunk changes, so it is not an exact attribution. We will test it directly. We inject synthetic false structures into the scribbles of a subset of scans, for example scribbling a nonexistent lesion. We then compare how well the difference-image audit and our correction map each localize the resulting hallucination, measured by area under the precision-recall curve for flagging corrupted pixels.

Proposed Experiments

The primary dataset is Calgary Campinas. It provides raw 12-channel k-space, so per-coil Cartesian undersampling at 4×, 6× and 8× is physically faithful. Secondary checks use MRBrainS and ACDC with retrospective undersampling of magnitude data, acknowledged as less realistic. Scribbles are generated from the dense masks with the ScribbleBench interior-and-border protocol. Dense masks and fully sampled images are used only for evaluation. Metrics are Dice per tissue or structure, and PSNR/SSIM against fully sampled references where available. The label-free audit metric is normalized error on permanently held-out measured k-space lines.

The decisive checks are three.

First, at 6× and 8× acceleration, our model should beat the cascade on Dice by a margin exceeding seed variance, while keeping held-out-line error no worse.

Second, adding the correction map should lower held-out-line error near true boundaries. If it raises that error, the scribbles are inducing hallucination and the method is reporting it honestly, which is itself a result. This check needs no ground truth, so it can be run on any clinical archive of raw k-space.

Third, in the injected-false-scribble experiment, the correction map should localize the fabricated structure, and its energy there should raise held-out-line error. Both the unconstrained joint model and the no-projection ablation should hide the same fabrication inside the image with no signal.

If the second and third checks pass while the first fails, the work still delivers a usable hallucination audit for label-guided reconstruction. If all three fail, null-space separation does not buy anything over a cascade.