Reconstructing for a Reader Who Adapts: Stackelberg Training of Medical Image Reconstructors

A CT or MRI reconstruction algorithm turns incomplete scanner measurements into an image. Its output is then read by a radiologist or a diagnostic AI model. Reconstructors are usually trained and ranked by pixel-matching scores such as PSNR (peak signal-to-noise ratio) and SSIM (structural similarity). These scores are known to disagree with diagnosis. The sharpest case comes from Tivnan et al.: on 72-view sparse head CT, iterative model-based reconstruction (MBIR) beat filtered backprojection (FBP) on PSNR and SSIM, yet an AI hemorrhage detector was less accurate on the MBIR images (65.6% vs 69.7%). The usual fix is to add a downstream task loss computed by a fixed, pretrained observer. That observer was trained on other images, though, and in deployment readers adapt: third-party AI tools get fine-tuned on a vendor's images, and radiologists learn a reconstruction's look. So "accuracy of a frozen observer" mixes two different things. One is how much diagnostic information the reconstruction keeps. The other is how familiar the images look to that particular observer.

We propose treating reconstruction as a Stackelberg game, a two-player game in which a leader commits to a strategy first and a follower then picks its best reply. Here the reconstructor is the leader. The diagnostic observer is the follower, and it is retrained on whatever images the leader produces. The leader is trained to maximize the follower's accuracy after that retraining, subject to staying faithful to the measurements. The claim is concrete and falsifiable. Optimizing against a best-responding observer should yield reconstructions whose diagnostic gains survive when new observers are trained on them. Optimizing against a frozen observer should yield gains that mostly disappear once the observer is retrained or swapped.

Core Claim

Write $R_\theta$ for a reconstructor with parameters $\theta$ and $h$ for an observer (a classifier or detector). The Stackelberg objective is

$$\max_\theta \; U\big(h^*(\theta), R_\theta\big) \quad \text{s.t.} \quad h^*(\theta) \in \arg\min_h \mathcal{L}_{\text{task}}\big(h;\, R_\theta(\mathcal{D}_{\text{train}})\big), \quad \mathcal{L}_{\text{fid}}(R_\theta) \le \tau .$$

In this objective, $U$ is held-out diagnostic utility (area under the ROC curve, or AUC, for hemorrhage detection). $h^*(\theta)$ is the observer retrained on reconstructions of the training scans $\mathcal{D}_{\text{train}}$. $\mathcal{L}_{\text{fid}}$ is a data-consistency and pixel-fidelity loss. $\tau$ is a fidelity budget, set so the leader cannot win by producing images that are no longer anatomically faithful.

The frozen-observer task loss fixes $h$ at a pretrained network. Joint multi-task training updates $h$ and $\theta$ at the same time. Both are different games from this one, and the difference is measurable.

The same lens reframes the MBIR paradox as an open question. It may reflect information MBIR actually destroyed. It may instead reflect a mismatch with an observer that, as the research file describes it, was trained on full-view images. Under best-response evaluation, the two explanations predict different outcomes. We do not claim formal Price-of-Anarchy bounds of the kind proved for stylized one-way games by Abeliuk et al.. We borrow only their framing: an upstream agent with a selfish objective can impose large losses on a downstream agent it never coordinates with. We measure that loss empirically, as the drop in best-response utility for a reconstructor trained only on pixel fidelity.

Relation to Prior Work

Three works motivate the gap. Tivnan et al. use an AI observer purely for passive evaluation. Lin et al. show that PSNR and SSIM ignore small organs and thin vessels. They propose anatomy metrics, NSD (normalized surface Dice) and clDice (centerline Dice), and a post-hoc diffusion enhancer, CARE, guided by a frozen nnU-Net segmenter. Kofler et al. show that Dice fails to track expert judgment.

Two works supply competing methods. SemuNet trains sampling, reconstruction and segmentation jointly with a fixed weighted sum of losses. In game terms that is simultaneous cooperative play, not leader–follower play. CARE optimizes against a follower that never moves.

The bilevel survey of Bilevel Methods for Image Reconstruction supplies the machinery we reuse: implicit differentiation and unrolled inner loops. It applies that machinery only with pixel losses at the upper level, and it notes that its guarantees need a strictly convex lower level. A deep observer violates that condition, which is a real constraint on our design (see below).

The performative-prediction work of Narang et al. gives a related warning. Learners that ignore how downstream data reacts to their deployment end up worse off. Our follower's reaction is retraining rather than a shift in the data distribution, but the lesson carries over.

Research Program

The leader is a learned reconstructor for sparse-view CT and accelerated MRI. In the MRI variant, a learned sampling mask in the style of SemuNet can optionally be added. The follower is a diagnostic observer split into a pretrained backbone and a trainable head.

Computing an exact best response for a full deep network inside every leader step is intractable, so we use three approximations of increasing fidelity. In the first, the follower is a linear or ridge head on frozen backbone features. Its best response then has a closed form, and implicit differentiation is exact. In the second, the follower fine-tunes its last blocks for $K$ unrolled steps and we backpropagate through them, as in the unrolled approach of the bilevel survey. In the third, the follower is retrained from scratch every few leader epochs and the leader trains against the latest copy. This is an outer loop that checks whether the cheap approximations track the true best response. Because the lower level is non-convex, we treat the hypergradients as heuristics. We report how far the cheaper approximations drift from the full-retraining loop, rather than claiming convergence guarantees.

The analysis has two parts. First, we re-evaluate FBP, MBIR and deep reconstruction under best-response observers to test whether the MBIR paradox survives. Second, we train the same reconstructor architecture under four leader objectives: pixel-only, frozen-observer, joint, and Stackelberg. We then measure transfer to observers the leader never saw.

Baselines

The naive reference point is a reconstructor trained only on pixel or data-consistency loss, alongside classical FBP and MBIR. For hemorrhage detection, the strongest reported numbers come from Tivnan et al.. They report overall accuracy of 71.7% for deep learning reconstruction, 69.7% for FBP, 65.6% for MBIR and 73.2% for full-view images, all under a fixed observer. We will reproduce this frozen-observer protocol first so that our numbers sit on the same scale.

The strongest existing method for aligning a reconstructor with a downstream task is the frozen-observer task loss. It is the core of CARE's anatomy guidance and of Tivnan's observer. We run it as a reconstructor trained with fidelity loss plus the loss of a fixed pretrained detector, sweeping the weight. The reconstructor architecture, data split, GPU-hours and number of hyperparameter trials are the same as for our method.

On JHH-style anatomy completeness, Lin et al. report CARE gains of "up to +32%" for large organs, "+22%" for small organs and "+36%" for vessels. No reported result on Lacuna was found for task-trained reconstructors on RSNA ICH or fastMRI+ under retrained observers.

The second competitor is SemuNet-style joint training, where reconstructor and observer update together under a weighted loss. On MRB at 10% sampling it reports Dice 75.08% against 72.91% for LOUPE plus a segmentation network. We reimplement it with our reconstructor at an equal budget.

The key ablation swaps only the follower model inside an otherwise identical pipeline: frozen versus best-responding. That isolates the Stackelberg ingredient.

Beating the frozen-observer baseline means three things. The Stackelberg reconstructor must give higher held-out AUC for observers retrained from scratch, including architectures absent during leader training. It must keep equal PSNR and SSIM within the fidelity budget $\tau$. And it must not lose on the in-loop observer by more than the gain it shows on held-out observers.

Why not just…?

Why not just add a frozen observer's task loss? This is the real threat. If the frozen-observer gains carry over to retrained and held-out observers, the game adds nothing beyond cost. Our prediction rests on a specific mechanism. A frozen network rewards images that match what it learned on full-view data, and a retrained observer does not need that match. The decisive experiment trains both leaders and then retrains fresh observers of three architectures on each leader's outputs. If the two leaders tie under retraining, we concede the point and report best-response evaluation as the contribution.

Why not just fine-tune the downstream detector on each reconstruction and skip training the leader? This may be all that is needed. Best-response evaluation is cheap, and it might show that a retrained detector on ordinary deep reconstructions already closes most of the 71.7%-to-73.2% gap to full-view images. If so, there is no headroom for leader training. We treat this as a go/no-go measurement before any bilevel training, and pursue leader training only if the retrained-observer gap stays meaningful at harder sparsity (fewer than 72 views) or higher MRI acceleration.

Why not just train reconstructor and observer jointly, as SemuNet does? Simultaneous co-training lets the pair settle on cues that only the co-trained observer reads. That is a form of collusion: the images keep task information in a form an independently trained reader cannot use. The held-out-observer test detects this directly, and Lin et al.'s anatomy metrics check that the images remain anatomically faithful.

Why not just post-process with CARE? A post-processor is a deterministic function of the reconstruction, so it cannot add information the reconstruction lacks for an observer that is allowed to retrain. Under best-response evaluation, any CARE gain must come from making existing information easier to learn. That would show up as gains that shrink as observer training data grows, which we test by varying the training-set size.

Proposed Experiments

The primary benchmark is the RSNA intracranial hemorrhage dataset with simulated sparse-view sinograms at 72, 48 and 24 views, following Tivnan et al. The secondary benchmark is fastMRI brain and knee at 4x and 8x acceleration, with fastMRI+ pathology boxes as the detection task. The vendor-transfer split from the fastMRI challenge tests whether gains hold on GE and Philips scanners. Metrics are AUC and accuracy under frozen, in-loop and held-out retrained observers, plus PSNR and SSIM. For CT anatomy we add NSD and clDice where segmentations exist.

The first decisive check is whether MBIR still underperforms FBP once each method gets its own retrained observer. The second is the headroom measurement described above. The third, which carries the main claim, is whether Stackelberg-trained reconstructors beat frozen-observer-trained ones on held-out retrained observers at matched fidelity. A secondary check is how closely the closed-form, unrolled and full-retraining followers agree. That agreement tells practitioners which approximation is good enough. A small blinded reader study on a subset of scans is a useful complement but is not required for the central claim.