Contracts with Hidden Auditors for Outsourced Medical Image Reconstruction

Hospitals increasingly buy image-reconstruction software, the models that turn undersampled CT or MRI measurements into diagnostic images, from outside vendors. They judge and pay those vendors using cheap pixel scores such as peak signal-to-noise ratio (PSNR) and structural similarity (SSIM). These scores can be pushed up by methods that smooth away the details radiologists need. The clearest case is sparse-view head CT. There, a smoothing iterative reconstruction beats plain filtered backprojection on PSNR and SSIM, yet it lowers intracranial hemorrhage detection accuracy from 69.7% to 65.6% (Tivnan et al.). A vendor paid on PSNR is therefore paid to hurt diagnosis.

This note proposes treating that situation as a moral hazard problem: a contract-design setting in which a principal pays an agent whose effort it cannot observe. We then design payments that combine the cheap pixel score with occasional, budgeted audits by downstream "observer" models. These are classifiers or segmenters that measure whether the diagnostic content survived. The key change is that the auditor is drawn secretly from a held-out pool, so the vendor cannot tune its outputs to the auditor itself.

Core Claim

A contract that pays a small share of the proxy score, plus a bonus paid with low probability based on an audit by a randomly chosen hidden observer, can steer a cost-minimizing vendor toward clinically faithful reconstructions. We expect it to spend far fewer labeled clinical cases than auditing everything. Two naive contracts should fail. Paying on the proxy alone rewards smoothing. Paying on a single fixed, known observer rewards observer gaming: the vendor gets gains that show up only on that observer and vanish on independent ones.

The claim can be falsified in both directions. If fixed-observer contracts are not gamed in practice, the hidden pool is unnecessary. If a well-chosen fixed proxy already aligns incentives, the audit machinery is unnecessary. The experiments below are built to detect either outcome.

Relation to Prior Work

Three kinds of prior work shape this proposal: work that motivates it, work that supplies its method, and work that constrains it.

Work that motivates the proposal. Three papers show the gap between pixel scores and clinical value. Tivnan et al. compare three fixed, non-strategic reconstruction methods on RSNA hemorrhage CT with a ResNet-50 AI observer. Lin et al. show that PSNR and SSIM miss whether organs, vessels and intestines survive reconstruction. They measure this with nnU-Net boundary scores (normalized surface Dice) and a topology score for tubular structures (clDice). Kofler et al. find that Dice correlates only about 0.38 with radiologist judgment for enhancing tumor, and that radiologists prefer algorithmic contours to human reference masks. All three treat evaluation as passive and offline. None asks what a paid developer would do if the metric became the paycheck.

Work that motivates the contract setting. Delegating Data Collection in Decentralized Machine Learning is the closest contract-theoretic paper. It shows that linear contracts on test accuracy capture a $1-1/e$ fraction of first-best utility. It assumes, however, that the paid metric faithfully reflects the principal's utility, which is exactly the assumption the imaging papers break.

Work that supplies the method. Contract Design With Safety Inspections pairs linear output sharing with randomized inspections under a budget. It computes optimal sharing and inspection rates in $O(n \log n)$ time. In its model, an inspection reveals compliance exactly. We keep its structure but replace the exact inspection with a noisy, learned auditor that the vendor could adapt to. That changes both the incentive constraint and the failure modes. Deep Contract Design via Discontinuous Networks supplies a tool for optimizing contracts when the agent's best response jumps between discrete actions. We use it only if the vendor menu becomes too large for exact enumeration.

Work that constrains the proposal. Hardness results for combined hidden actions and hidden types (Contracts under Moral Hazard and Adverse Selection) and for multi-dimensional effort (Optimal Scoring Rules for Multi-dimensional Effort) warn against seeking exact optimal contracts in general. We therefore restrict attention to a two-parameter contract family evaluated against an empirically measured vendor menu.

A result that cuts against us. Tivnan et al. also find that a deep-learning reconstruction improves PSNR, SSIM and accuracy together: 12.59 dB PSNR, 0.81 SSIM and 71.7% accuracy, against a 73.2% full-view ceiling. Misalignment is therefore not universal. Contracts matter only when the vendor's cheapest proxy-maximizing option differs from its clinically best option. Our vendor menus are built to measure how often that happens rather than to assume it.

The Model

The vendor chooses a reconstruction pipeline $a$ from a finite menu. Each pipeline has a compute and engineering cost $c(a)$, a proxy score $q(a)$ (PSNR or SSIM on the public validation set), and a pass probability $\pi_o(a)$. The pass probability is the chance that a randomly sampled case reconstructed by $a$ is judged correctly by observer model $o$.

The principal pays the vendor $\alpha q(a)$ in every round. With probability $\beta$ it also runs an audit: it draws an observer $o$ from a hidden pool $\mathcal{O}$ and a labeled case, and pays a bonus $b$ if the audit passes. Limited liability caps the bonus at $b \le \bar b$. The vendor prefers the clinically faithful pipeline $a^*$ over a proxy-gaming alternative $a'$ when

$$\alpha\,[q(a^*) - q(a')] + \beta\,\bar b\,[\bar\pi(a^*) - \bar\pi(a')] \ge c(a^*) - c(a'),$$

where $\bar\pi$ is the pass probability averaged over the pool.

Two things follow directly from this inequality. First, the audit rate needed grows as the audit's discriminating gap $\bar\pi(a^*)-\bar\pi(a')$ shrinks. Noisy observers therefore cost audit budget in proportion to how poorly they separate faithful from smoothed reconstructions. Second, if the pool contains a single known observer, the vendor can add a pipeline $a''$ that raises $\pi_o$ specifically. It could, for example, fine-tune the reconstruction through that observer's gradients, much as the anatomy loss in Lin et al. does for nnU-Net. Such an $a''$ earns the bonus without clinical gain. Averaging over a hidden pool should dilute that gain, because features that fool one observer need not fool others. How much it dilutes is the central empirical question.

The principal's utility is clinical accuracy measured by a reference observer that is never used in audits, minus expected payments, minus audit cost. We count audit cost in labeled clinical cases consumed, not in GPU time. Running a ResNet-50 is cheap. What is scarce is labeled pathology and radiologist reads, and reusing them risks leaking the audit set to the vendor.

Research Program

Building vendor menus. We construct menus for two tasks. The first is sparse-view head CT on the RSNA Intracranial Hemorrhage data, simulated at 72 views following Tivnan et al. and also at 20 to 40 views. The second is accelerated knee and brain MRI on fastMRI, where fastMRI+ pathology bounding boxes supply the audit labels.

Each menu contains filtered backprojection or zero-filled inverse Fourier reconstruction, total-variation or MBIR reconstruction swept over regularization strength, a U-Net, a variational-network cascade (VarNet with feature-domain learning), and diffusion reconstructions with Gaussian and with Poisson noise models. Including both noise models deliberately introduces the hallucination failure mode reported in DM4CT. The menu also includes gaming pipelines fine-tuned to maximize the pass rate of one specific observer. Each pipeline's cost is its measured training and inference GPU-hours.

Building observer pools. Each pool contains observers that differ in architecture (ResNet, ConvNeXt-style, vision transformer), training seed and training data split, and also in task. For CT, the tasks are hemorrhage presence and hemorrhage subtype. For MRI, the tasks are detection of the fastMRI+ pathology classes. A separately trained reference observer, never exposed to the vendor, measures principal utility. Where budget allows, a small radiologist read on a stratified subset checks whether the reference observer's rankings agree with human judgment.

Measuring gaming. For every pipeline, we record the gap between its pass rate on the observer it was tuned against and its pass rate on held-out observers. This transfer gap is the quantity that decides whether hidden pools are needed at all.

Solving contracts. Given the measured menu, we compute the vendor's best response to every $(\alpha, \beta, \bar b)$ on a grid. We report the principal-optimal contract for each audit budget, meaning each number of labeled cases consumed. The menus are small enough for exact enumeration. We invoke the discontinuous-network optimizer of Wang et al. only if we extend to continuous regularization weights.

Two extensions. The first adds hidden vendor types (vendors with different cost structures) to test robustness to adverse selection. The second is a repeated game in which the vendor observes which observer audited it and adapts. Both are secondary to the static study.

Baselines

The do-nothing reference is a linear contract on PSNR alone ($\beta = 0$), the regime implied by leaderboard-based procurement and by linear contracts on accuracy in Delegating Data Collection. It uses the same vendor menu and is swept over $\alpha$ with the same grid resolution as every other contract. We expect the vendor to select smoothing pipelines. Any audit contract must beat it on reference-observer accuracy at matched expected payment.

The strongest fixed-metric competitor is a linear contract on an anatomy- and task-aware composite: normalized surface Dice and clDice from nnU-Net, following Lin et al., weighted according to the problem-fingerprint guidance of Metrics Reloaded, plus hemorrhage accuracy from one known observer. This is the real threat, because it removes much of the misalignment without any randomization. It gets the same payment budget and the same weight-tuning grid as our contract.

The ceiling on attainable accuracy is the deep-learning reconstruction of Tivnan et al. at 72 views, with a reported 71.7% hemorrhage accuracy against a 73.2% full-view ceiling. Beating the fixed-metric competitor means two things at the same payment. First, the chosen pipeline's reference-observer accuracy must be statistically closer to this ceiling. Second, the transfer gap must shrink. No reported result was found on Lacuna for any contract or incentive mechanism evaluated on these benchmarks.

The inspection baseline is the Fallah–Jordan design from Contract Design With Safety Inspections. It is implemented with a single fixed observer and with $\beta$ and $\alpha$ set as though audits were exact. This isolates the cost of ignoring audit noise.

The key ablation holds everything fixed, including audit budget, $\alpha$, $\beta$ and $\bar b$, and changes only whether the audited observer is fixed and disclosed or drawn from the hidden pool. A second ablation varies pool diversity: seeds only, then architectures, then tasks. This shows which kind of diversity actually suppresses gaming.

Why not just…?

Why not just train the reconstruction end-to-end on the clinical task? Joint training works when one party controls the loss, as in One Network to Solve Them All and CARE. Our setting is procurement: the hospital cannot set a vendor's loss, compute budget or regularizer. Moreover, "train end-to-end on the observer" is precisely the gaming pipeline in our menu. If its gains on held-out observers match its gains on the target observer, this alternative wins and our hidden pool is unnecessary. The transfer-gap measurement settles this directly.

Why not just pay on a better metric such as normalized surface Dice and clDice? This is the most serious alternative. It could plausibly work whenever the menu contains no pipeline that inflates nnU-Net-based scores without diagnostic gain. Our case against it is conditional. These scores are themselves outputs of a learned model evaluated against noisy tubular labels, as Lin et al. acknowledge. A disclosed fixed scorer can be optimized against in the same way a disclosed observer can. The fixed-metric baseline above tests this head-on. If it matches our contract on reference accuracy and shows no transfer gap, we will report that a well-chosen public metric suffices and that randomized auditing is not worth its complexity.

Why not just audit every case? Tivnan et al. show that observer inference is computationally cheap, so compute is not the objection. The objection is that each audit consumes labeled clinical cases, and full audit reuses a fixed labeled set every round. Reuse lets the vendor adapt to the audit set, and the Fallah–Jordan analysis shows that randomized inspection can preserve incentives at a fraction of the budget. We plot principal utility against labeled cases consumed. Full audit is the right-hand endpoint of that curve. If the curve is flat only near full audit, the savings are illusory and we will say so.

Why not just pay against radiologist reference masks? Kofler et al. find radiologists rate algorithmic contours above human references (4.79 versus 4.47 stars). Paying for agreement with human masks would therefore penalize cleaner boundaries. We use radiologist reads only as a small, sequestered validation of the reference observer, never as the payment signal.

Proposed Experiments

The first decisive check is whether gaming is real. On RSNA CT at 72 and 32 views, and on fastMRI knee at 4× and 8× acceleration, we measure the transfer gap for pipelines tuned against a single observer. We report the target-observer gain, the held-out-observer gain, and the reference-observer accuracy, with bootstrap confidence intervals over patients. If held-out gains track target gains within noise, we shrink the proposal to a study of noisy-audit rates without hidden pools.

The second check is the utility–budget frontier. For each of the do-nothing contract, the fixed-metric composite, the exact-inspection contract, the fixed-observer audit and the hidden-pool audit, we plot reference-observer accuracy and principal net utility against labeled audit cases per round. Our claim holds if the hidden-pool contract reaches within the confidence interval of the 71.7% ceiling configuration at a small fraction of the full-audit budget, while the proxy-only contract selects a smoothing pipeline.

The third check tests whether the incentive inequality predicts behavior. We compare the audit rate at which the vendor's best response switches from $a'$ to $a^*$ against the rate predicted from the measured pass-probability gap. Agreement would mean audit noise can be priced in advance from a small pilot.

The final check addresses dense regimes. Lin et al. report that anatomy-aware corrections matter little at high view counts, so we repeat the study at full or near-full sampling. There we expect every contract to converge to the same choice, and this negative control confirms the effect is driven by misalignment rather than by the audit machinery itself.