Committed Audit Sampling for Certified Medical Image Quality Control

Large imaging studies such as the UK Biobank and the German National Cohort now segment tens of thousands of scans automatically, and nobody can inspect them all. Current practice uses an automated quality score to decide which cases a clinician looks at. That score is itself an unvalidated model, so whatever it wrongly accepts is never seen again. The resulting "clean" dataset carries no stated error rate, and no outsider can check how it was curated. We propose a different use of the same score. It should decide how often a case is spot-checked, not whether a human ever sees it. And before any spot-check is drawn, every automated decision should be written into a tamper-evident log, so the check cannot be steered after the fact.

In two sentences: first commit the automated quality decisions for the whole cohort to an append-only, hash-linked record. Then draw a small random audit sample, with probabilities weighted by the automated quality score and randomness fixed only after the commitment. The clinicians' verdicts on that sample give a statistically valid upper bound on the failure rate among accepted scans, overall and per demographic group, with far fewer reviews than exhaustive inspection. The bound holds even when the quality model is wrong, and a third party can re-verify it.

Core Claim

Immutable audit trails and automated quality control have grown up apart. FG-Trac anchors sample-level lineage for fMRI pipelines on ABIDE and ADHD-200 into Merkle trees, which are hash trees that let anyone prove a single record belongs to a committed batch. It logs data passively, so a corrupt scan is certified just as faithfully as a good one. The Ethical Black Box pattern proposes immutable decision logs as a software-engineering design, without concrete quality-control machinery. On the other side, reference-free quality estimators are now strong but live in mutable stores. The shape-prior alarm of Liu et al., the Dice regressor of Robinson et al., and Quality Sentinel all fall in this group, and all are used as hard gatekeepers.

Bolting a blockchain onto a quality-control pipeline would just be FG-Trac with a score field added. It would also invite the obvious objection that a signed database does the same job. Our claim is narrower. Committing to decisions before sampling is exactly what turns a cheap spot-check into a certificate. The certificate is valid under honest operation because the sample is random with known inclusion probabilities. It is valid under dishonest operation because nobody can re-score, re-threshold, or choose cases after seeing which ones will be reviewed. The quality model only has to rank cases usefully to save reviewer effort. It never has to be correct for the guarantee to hold.

We state plainly which ingredient does what. Risk-weighted sampling saves the labor. The commitment protects the guarantee against manipulation. A distributed ledger is optional, and adds something only when several institutions distrust one another.

Research Program

The pipeline has four stages.

First, every case receives an automated segmentation, a quality score $s_i$, an accept or flag decision, and a coarse stratum (sex and age band where available). The quality score comes from a Quality Sentinel–style regressor or a shape-prior fake-Dice estimator.

Second, these records, together with a hash of each mask and the full sampling protocol, are sealed into a Merkle root. The protocol covers the map from $s_i$ to an inclusion probability $\pi_i$, the failure definition, and the target error $\epsilon$. The root is appended to a transparency log, meaning an append-only, hash-chained public record whose history cannot be rewritten without detection.

Third, audit randomness is derived by hashing the root together with a public randomness source published after the commitment. Each accepted case enters the audit independently with probability $\pi_i$.

Fourth, clinicians review the drawn cases, and their signed verdicts $y_i \in \{0,1\}$ (1 = failure) are appended to the same log.

The number of failures hidden among accepted cases is then estimated without bias by the Horvitz–Thompson estimator

$$\hat F = \sum_{i \in \mathcal{S}} \frac{y_i}{\pi_i},$$

where $\mathcal{S}$ is the audited set. It is unbiased because each case's contribution is reweighted by the inverse of its known inclusion probability.

We will build upper confidence bounds on the accepted-set failure rate using betting-style confidence sequences. These allow clinicians to stop reviewing as soon as the bound falls below $\epsilon$, which is a rigorous generalization of the "58 consecutive clean samples" rule used by Gröger et al..

The variance of $\hat F$ shrinks when $\pi_i$ is roughly proportional to the true failure probability. So a better quality model means fewer reviews, and a worse one means more, but coverage is never lost. We will study how to choose $\pi_i$, including a floor probability for every case. The floor is what catches confident silent failures, which Liu et al. show uncertainty-based scores miss.

We will also study per-stratum certificates. These respond directly to the finding that TotalSegmentator label quality is worse for female and older subjects (Quality Sentinel), and to the risk that gatekeepers silently remove atypical phenotypes, which Full et al. warn about.

"Failure" will be defined two ways. The first is Dice below 0.7, following Robinson et al. The second is relative volume error above 10%, because Zhang and Bağcı show that Dice alone does not bound volume error. This matters because downstream biomarkers such as ejection fraction depend on volume, not overlap.

Baselines

The naive reference is to accept every automated segmentation with zero manual review and report its true failure rate. This sets the scale of the problem, and the certificate should show it has been reduced and bounded.

The first real competitor is score-thresholded gating with no review of accepted cases, in the manner of the MRI Filter of González et al.. On the white-matter lesion cohort that filter let 84.5% of scans bypass review and raised mean Dice from 0.84 to 0.87.

The strongest existing method is threshold gating with Quality Sentinel. It reports a Pearson correlation of 0.902 between predicted and true Dice on external sets, and a 33% reduction in manual annotation cost in human-in-the-loop use. We will run it as released, using the same segmentations, the same review budget, and a threshold tuned on the same validation split as our sampling map.

We also include flag-and-review, as in the NAKO pipeline. There, 17.0% of the cohort was flagged and adjudicated, and the combined outlier detector reached an AUC of 0.748. We will match its review budget.

Uniform sequential review with the Gröger stopping rule is included as the non-weighted statistical reference.

To beat the gating baselines, we must meet the same review budget while delivering a bound whose empirical coverage stays at or above 95%, including on held-out sites and subgroups, where their implicit error claims fail. To beat uniform review, we must reach the same certified $\epsilon$ with meaningfully fewer reviews.

Two ablations isolate the ingredients. Uniform sampling with commitment isolates the value of risk weighting. Risk-weighted sampling without commitment is statistically identical to our method under honest operation, so it isolates the value of the commitment under the manipulation experiments below.

Why not just…?

Why not just use a signed, append-only database instead of any ledger? For a single institution, this is correct, and it is our default. The mechanism needs a commitment made before sampling and a log nobody can quietly rewrite, which a hash-chained transparency log provides. Distributed consensus becomes worth its cost only in multi-site consortia, where no site is trusted to hold the record. We do not claim otherwise, and the system is designed so the anchor can be swapped out.

Why not just threshold on Quality Sentinel and trust it? A threshold gives no stated error rate on what it passes, and its errors are not uniform across settings. MRIQC reached 76% accuracy on unseen sites, but recall for scans that should be excluded was only 0.28. Quality Sentinel itself documents demographic quality gaps. Our held-out-site and per-subgroup experiments will show whether thresholding's silent failure rate exceeds its nominal level. If thresholding plus a small uniform check matches our efficiency, weighting has added little, and we will report that.

Why not just review the lowest-scoring cases, as active triage does? Reviewing only flagged cases is a biased sample. It cannot say anything about confident failures, which are precisely the missing-lobe and hallucinated-fragment errors that uncertainty maps miss. The floor probability in our design exists to cover these cases. Comparison against the flag-and-review baseline at equal budget measures what the biased design misses.

Why not just draw a random audit in-house without committing first? If the operator is fully trusted, this works exactly as well. That is why it is an ablation, not a straw man. The difference appears only when someone can re-score, re-threshold, or discard cases after seeing the sample. The manipulation experiments quantify how far such an operator can deflate the reported bound, and they check that the commitment makes this detectable.

Proposed Experiments

We use datasets where the true failures are known. These are NIH Pancreas-CT, the Medical Segmentation Decathlon, and Synapse (the settings of Liu et al.), TotalSegmentator and the DAP Atlas collection (Quality Sentinel), and the aggregated white-matter lesion cohort (González et al.). UK Biobank cardiac data will be added where access permits, following Robinson et al. Segmentations come from off-the-shelf models, and the simulated "expert" is the ground-truth failure label.

To reflect real reader noise, we also flip verdicts so that simulated readers agree at Cohen's kappa of 0.39–0.51, the expert agreement MRIQC reported. We then measure how the bound degrades.

Three measurements settle the proposal. The first is coverage: across many repeated audit draws, the certified bound must contain the true accepted-set failure rate at least 95% of the time. This must hold overall, per sex and age band, and when the quality model is trained on one site and deployed on another. The second is efficiency: the number of reviews needed to certify $\epsilon$ at 1%, 2%, and 5%, compared with uniform review and with the 17% flag budget, plotted against the quality model's correlation with true Dice. The third is manipulation: simulated operators re-score cases, raise thresholds, or drop audited failures after seeing the draw. We report how far each attack deflates the bound without commitment and whether it is detected with commitment.

We will also report the cost of the logging itself. FG-Trac found overhead of under 2% on CIFAR-10, but 2.5–2.9x when per-sample influence scores were computed. We expect to be near the low end, since we log scores and hashes only.

The proposal fails if coverage breaks under realistic reader noise, or if weighted sampling saves under roughly a third of reviews compared with uniform sampling at matched $\epsilon$. In that case the contribution would reduce to "log your spot-checks," which is useful but not new.