Catching Cheap Shortcuts in Decentralized 3D Medical Model Evaluation

Medical imaging benchmarks increasingly cannot move patient data, so they move the model to the data. Each hospital runs a submitted model on its own private scans and reports the scores back. MEDS-DEV builds this kind of decentralized benchmark for health records, using standardized task configurations and score submissions by pull request. It trusts every site to have actually run what it says it ran. For 3D imaging that trust is costly to honor. Full volumetric models such as 3D nnU-Net are expensive enough that nnU-Net for Brain Tumor Segmentation trained at batch size 2. A site under compute pressure can quietly substitute a cheaper model and still report plausible numbers. The cheaper model might work on 2D slices, use only adjacent slices, run at lower resolution, or use one fold instead of a five-fold ensemble. Blockchain systems call this laziness the verifier's dilemma, and they handle it by having participants post a deposit that is confiscated ("slashed") when misbehavior is proven (Proof of Diligence).

This note proposes the missing piece that makes any such penalty usable for medical models. The piece is an audit test that decides, as a yes/no verdict, whether a site ran the claimed 3D model. The test is built so that honest sites are almost never accused, even though GPU arithmetic is not bit-reproducible. The audit hides a few "probe" scans, whose correct outputs the auditor can recompute, inside each site's evaluation queue. It then compares the site's committed output probability maps against a reference re-execution. The probes are chosen to be the cases where through-plane anatomical context changes the model's answer most. Those are the cases where models that discard volumetric context diverge from the true 3D model by far more than honest hardware noise does. The economic layer then reduces to one inequality that sets how many probes and how large a bond are needed.

Core Claim

The claim is that execution-diligence verification for volumetric medical models is mainly a problem of separating two signals. The first is honest nondeterminism: the same model on different GPUs, library versions and kernel choices. The second is the output gap of the cheapest plausible substitute models. Context-poor substitutes are exactly what volumetric context loss predicts a lazy site would use. Their gap from the true 3D model is concentrated on a small, identifiable set of cases, such as small lesions spanning few slices, structures crossing slice boundaries, and ambiguous enhancing-tumor margins. If that holds, a handful of context-sensitive probes per evaluation round gives near-certain detection at a false-accusation rate the protocol can state in advance.

The binary-classification lens matters here because a slashing contract is itself a binary classifier on noisy evidence. Following the fault-differentiation principle of Cryptoeconomic Security for Data Availability Committees, we do not force a single threshold. The verdict has three outcomes: match, benign deviation that triggers re-audit at no penalty, and shortcut that triggers the penalty. Isolated hardware oddities therefore never cost an honest site its bond.

Research Program

The site commits, by hash, to the per-voxel probability maps for every case before any probe identities are revealed. This prevents it from answering probes differently after the fact.

Let $\delta(x)$ be a distance between the site's map and the reference map on probe $x$; the primary candidate is one minus soft Dice over all three BraTS sub-regions. Let $F_0$ be the distribution of $\delta$ under honest cross-hardware re-execution, and $F_1$ its distribution under a substitute model. We set the shortcut threshold $\tau$ at the $1-\alpha$ quantile of $F_0$, estimated across a deliberately diverse hardware pool. A site caught on any one of $m$ probes is then falsely accused with probability at most $m\alpha$.

Probe selection ranks candidate public cases by the expected gap between the claimed 3D model and a family of context-poor substitutes. The ranking is computed once by the auditor. We then test whether that ranking transfers to substitutes the auditor never saw.

The economic condition is simple. Let $c$ be the compute saved per case by shortcutting, $n$ the cases in a round, $d$ the per-probe detection probability, and $S$ the bond. Shortcutting is unprofitable when

$$S\left(1-(1-d)^{m}\right) \;\ge\; c\,n,$$

which follows from the cheater's expected loss being the bond times the chance at least one probe fires. Following the data-availability committee analysis, the penalty is burned or neutrally redistributed rather than paid to the challenger, which removes any reward for engineered false accusations.

The bond need not be cryptocurrency. It can be an institutional deposit or a reputational stake, and the detection test is the same in every case. That is deliberate: the test, not the settlement layer, is the contribution.

There is one honest limit on scope. The audit proves the claimed model was executed on cases the site could not tell apart from probes. It does not prove that the site's private labels or metric arithmetic are correct, because no one outside the site can see those labels.

Baselines

The naive reference point is MEDS-DEV-style unverified reporting, which detects nothing by construction. We run it only to report the score inflation or cost saving a lazy site obtains undetected, which sets the stakes.

The second baseline is score-level anomaly detection: flag a site whose reported Dice is an outlier relative to other sites. It uses only the reports the status quo already collects, so it is the cheapest possible defense and the natural competitor. It gets the same simulated sites and the same report history. It beats us if it flags shortcut sites at equal false-accusation rate.

The third baseline is random-probe auditing with probes drawn uniformly from public cases, a fixed Dice tolerance, and no hardware calibration. It is the strongest simple competitor, because it is our protocol minus its two key ingredients. Running it on the same probe budget isolates whether context-sensitive selection and calibrated thresholds earn their keep. We count our method as beating it if it reaches the same detection power with at least several times fewer probes at matched false-accusation rate.

Two ablations complete the comparison: calibrated thresholds with random probes, and selected probes with uncalibrated thresholds.

No reported result for detection rate or false-accusation rate of any verification scheme on medical segmentation was found on Lacuna. The reported numbers we can anchor to describe the model being protected and the cost of settlement. The claimed model is the nnU-Net BraTS 2020 ensemble, at test Dice 88.95 whole tumor, 85.06 tumor core and 82.03 enhancing tumor (nnU-Net for Brain Tumor Segmentation). Each on-chain bounty submission in Proof of Diligence costs about 380,000 gas with roughly 3-minute finality (Proof of Diligence), which bounds settlement cost.

Why not just…?

Why not just use a trusted central challenge server, as BraTS has for a decade? Central servers work when data can be shipped or when sites trust the host. The setting here is the one MEDS-DEV targets, where private cohorts stay local. Our naive baseline measures exactly what is lost without verification in that setting.

Why not just require a trusted execution environment or a zero-knowledge proof? This is the real threat. If every hospital had confidential-computing GPUs, attestation would prove the claimed code ran, and our audit would be redundant. Zero-knowledge proofs for convolutional networks exist (VerfCNN), but proving inference over full 3D MRI volumes is plausibly far beyond practical prover cost. Agora shows that even attested enclaves need lightweight external checking to keep the trusted code small. We position the probe audit as the hardware-free option and report its per-round auditor compute against a measured enclave-overhead estimate. Where enclaves are available, the audit is a complementary check.

Why not just audit with random public cases and skip the economics? This is a serious objection, and it reshaped the proposal. Detection is the core; the bond is a tunable consequence. The random-probe baseline settles whether context-aware selection matters. If random probes match selected ones, the contribution shrinks to the calibration study and the deterrence formula, and we will report that.

Why not just flag outlier scores? A 2.5D or single-fold substitute may lose only a few Dice points. Real site heterogeneity can easily produce gaps that large. The score-anomaly baseline quantifies this directly.

Proposed Experiments

We simulate a consortium on BraTS 2021 (RSNA-ASNR-MICCAI BraTS 2021, 2,040 patients) by partitioning cases into sites. We will partition by contributing institution where that information is available, so that site distributions genuinely differ. The claimed model is a 3D nnU-Net ensemble trained on BraTS 2020 data.

The substitutes form the cheapest-shortcut family. They are a 2D nnU-Net, a 2.5D adjacent-slice model in the style of 3D-CAM, a lower-resolution 3D model, a single fold of the ensemble, and the model without test-time augmentation.

Honest variation is measured by re-running the claimed model across multiple GPU types, deterministic and non-deterministic kernel settings, and mixed versus full precision. These runs estimate $F_0$ and the threshold $\tau$.

A second domain tests generality: AbdomenMRI with Swin-UMamba as the claimed model and its 28M-parameter variant Swin-UMamba† as the substitute. This tests whether the separation holds when the shortcut is a smaller model rather than a context-poor one.

The metrics are the per-probe detection probability $d$ for each substitute, the empirical false-accusation rate on held-out hardware configurations, probes needed for 99% detection, and auditor compute as a fraction of the site's evaluation cost. The bond implied by the inequality above, at measured compute savings $c$, is reported alongside.

The first decisive check is separation. The honest-noise distribution must sit well below the gap of every substitute, including the hardest one (single fold). If single-fold outputs fall inside honest noise, the three-outcome verdict cannot separate them, and we will say so.

The second decisive check is transfer. Probes selected against the auditor's substitute family must still detect a held-out substitute, for example a Mamba-based 2D model the auditor never used.

The third decisive check is indistinguishability. We train a classifier to tell probes from a site's own cases. Probes would be drawn from public BraTS data or from other consortium sites. If the classifier succeeds, an adaptive site could run the true model only on suspected probes. We would then measure how much detection survives when probes are domain-matched, for example contributed by a site with similar scanners. Failure here would mean the scheme needs probe donation from same-domain sites, and that requirement becomes part of the result.