A certified medical imaging model cannot be retrained once it ships. When it reaches a new hospital, with a different scanner, endoscope or protocol, it often fails without any outward sign. Avoiding Catastrophic Referral Failures In Medical Images Under Domain Shift shows that under such shifts even Bayesian networks make confident mistakes, so the model's own uncertainty cannot say when to distrust it. Meanwhile, the clinician reading the same image is producing a signal about where the pathology is, and producing it for free: their eye movements. GazeRefine showed that the places and durations a clinician fixates during routine reading can be turned into near-supervised segmentations with no training at all.
We propose to use that signal in a different role. Gaze would not replace the model. It would audit the model. For each case we compare the frozen model's predicted lesion region with the region the reader dwelt on, and use the mismatch to flag the cases where the model has likely failed. The score needs no target-site labels and no change to the model's weights. The same score also tells a hospital which few cases are worth sending to an expert for annotation. In two sentences: a locked model's errors under scanner shift are invisible to its own confidence, but they show up as disagreement with where the clinician actually looked. We measure that disagreement and use it to monitor the model, to trigger review, and to decide what to label.
Core Claim
Our central hypothesis is this. Under cross-center domain shift, a per-case score measuring disagreement between a frozen segmentation model's output and the reader's dwell-weighted gaze will detect model failures better than signals computed from the model alone. Those model-only signals include softmax confidence, entropy and test-time perturbation instability. The advantage should come from the fact that gaze is produced independently of the model's corrupted features. A shuffled-gaze control should fall to chance, which would show the signal is case-specific and not a generic center-of-image prior.
This sits at a specific point among the three directions. Workflow integration is the core mechanism. Clinicians are not asked to inspect heatmaps. Instead, the model's explanation is checked against the clinician's own behavior, which addresses the interpretability-utility gap without adding reading burden. Domain shift is the operating regime, under the regulatory constraint that weights stay locked. The annotation bottleneck is where the payoff appears: flagged cases become a small, well-chosen labeled set. That set can be used to tune post-hoc calibration that would otherwise require a curated out-of-domain validation set.
Research Program
The system has three parts. The first is a locked predictor: a segmentation network trained on source centers and never updated afterwards. The second is a gaze channel that converts fixations and dwell times into a normalized heatmap $g(x)$ over pixels $x$, exactly as GazeRefine does. The third is a disagreement scorer. We study two versions. The raw version measures how much gaze mass falls outside the model's predicted foreground probability $m(x)$, together with how much predicted foreground received no gaze:
$$D_{\text{raw}} = \tfrac{1}{2}\Big(\frac{\sum_x g(x)\,(1-m(x))}{\sum_x g(x)} + \frac{\sum_x m(x)\,\mathbb{1}[g(x)<\tau]}{\sum_x m(x)}\Big),$$
where $\tau$ is a small threshold that marks pixels as unattended. The refined version first runs GazeRefine to turn gaze into a mask $G$, then sets $D_{\text{ref}} = 1 - \text{Dice}(M, G)$, with $M$ the model's thresholded mask.
The refined score inherits GazeRefine's weakness on low-contrast tissue. On prostate MRI, GazeRefine reaches 80.28% Dice against 88.10% on polyps, so the comparison between $D_{\text{raw}}$ and $D_{\text{ref}}$ is itself informative. If the frozen foundation features degrade under shift, $D_{\text{ref}}$ should suffer and $D_{\text{raw}}$ should not.
We then use the score in two ways. The first is per-case referral and site-level drift monitoring, measured with risk-coverage analysis. The second is annotation selection: the top-ranked disagreement cases are sent for expert correction, and those labels are used to fit the per-class scaling and shift parameters of Split Platt Scaling. That method is the post-hoc logit fix from Filos et al., and it needs labeled out-of-domain data that is usually unavailable. Our goal here is to replace their curated validation set with a gaze-selected one of a few dozen cases.
Prior work plays three roles in this program. Filos et al. and the AMOS result that unseen MRI scanners cause a collapse of more than 20% mDice motivate the problem: shift failures are large and self-confidence does not reveal them. GazeRefine supplies the gaze-to-mask method, but it treats gaze as a segmentation prompt and evaluates only in-distribution. We treat gaze as an external check on a different model, under shift. Adapt the Data, Not the Model constrains the design: it argues that certified predictors must stay frozen. Its adapter transforms the inputs and needs labeled target data. Our approach transforms nothing and leaves raw pixels untouched, which avoids the traceability problem that paper raises.
GazeRefine also documents a limitation that bears directly on us: readers search multiple candidates and look at normal anatomy. This can produce false alarms, and we measure it explicitly instead of assuming it away.
Baselines
The naive reference is no failure detection: accept every model output. This gives the risk-coverage curve at full coverage and the raw source-only Dice on each target center.
The first model-only competitor is maximum softmax probability and mean predictive entropy over the predicted foreground. These are what a deployed device would report today. Filos et al. predict they will fail under shift, which is exactly why they must be included.
The strongest existing competitor is test-time perturbation instability. Here the frozen model is run on augmented copies of the input and disagreement across those copies serves as the failure score. This is the uncertainty signal underlying Improved post-hoc probability calibration for out-of-domain MRI segmentation. That work reports out-of-domain ECE of 12.78% against 14.48% for local temperature scaling under simulated artifacts, and 15.37% against 16.78% for bSSFP-to-LGE shift. It is a label-free, weight-free signal that already works under shift, so it is the bar gaze must clear. No reported result was found on Lacuna for failure-detection AUROC with this signal on polyp or prostate data, so we will compute it ourselves. Every baseline uses the same frozen model, the same target images, the same number of forward passes (we match perturbation count to compute budget), and thresholds chosen on source data only.
For the annotation-selection use, the competitors are random selection and entropy-ranked selection of the same number of cases. Each is used to fit Split Platt Scaling.
The key ablations isolate the ingredient that matters, which is case-specific gaze. In the shuffled-gaze control, each image receives another image's gaze. A second control uses a fixed center-bias saliency prior. A third removes dwell time by weighting all fixations uniformly. Beating the baseline means a failure-detection AUROC at least 0.05 above perturbation instability on real cross-center targets, with shuffled gaze near 0.5. For selection, beating the baseline means lower post-calibration referral risk at matched coverage than random or entropy selection with the same labeling budget.
Why not just…?
Why not just use the model's confidence, calibrated with Split Platt Scaling? Filos et al. show that raw confidence collapses into confident errors under shift. Their fix needs a labeled out-of-domain validation set to tune its parameters, and that set is what a new site lacks. We include calibrated confidence as a baseline. Our selection experiment tests whether gaze can supply the labels that make it work, so the two methods are complementary rather than competing.
Why not just use perturbation instability, which needs no eye tracker? This is the real threat. It is cheap, label-free and already shown to help out-of-domain calibration. If it matches gaze disagreement on failure detection, the honest conclusion is that gaze is unnecessary for this purpose. In that case the contribution narrows to annotation selection and to a combined score, which we test by checking whether gaze adds to perturbation instability in a joint logistic model. The decisive comparison is head-to-head on the same frozen model and the same targets.
Why not just let GazeRefine produce the segmentation and drop the locked model? On in-distribution polyps GazeRefine nearly matches nnU-Net (88.10% vs 88.41% Dice). But it is not a certified device, it needs a reader in every case, and its frozen features degrade on low-contrast and shifted images. We report GazeRefine's own Dice on every shifted target. If it consistently beats the locked model under shift, that is a finding worth reporting, but it answers a different deployment question.
Why not just adapt the model at test time with entropy minimization? Regulatory lock forbids weight updates. The evidence also says it hurts: TENT scores 61.20 Dice against 61.95 source-only on CHASE→HRF (On-the-Fly Test-time Adaptation for Medical Image Segmentation), and it drops more than 5% over continual shifts in MGIPT.
Proposed Experiments
Polyp study. The first experiment uses the existing Kvasir-SEG gaze recordings from GazeRefine as the target site, which puts the shift on the side where gaze exists. We train locked polyp models on CVC-ClinicDB and BKAI-IGH, using a classic UNet and TransNetR, and evaluate on Kvasir-SEG. For in-distribution context, TransNetR reports 0.8706 Dice on Kvasir-SEG when trained there. The primary metric is AUROC for detecting cases with Dice below 0.5. Secondary metrics are Spearman correlation between the disagreement score and Dice, and area under the risk-coverage curve.
Prostate study. The second experiment repeats this on NCI-ISBI prostate MRI. We split by acquisition site if the site metadata supports it; if not, we inject TorchIO-style artifacts following Ouyang et al. This is the hard case, where $D_{\text{ref}}$ is expected to weaken and $D_{\text{raw}}$ to hold.
Prospective reader study. The third experiment is a small prospective study on a subset of PolypGen centers, where no gaze exists yet. Endoscopists read with an eye tracker in two arms: with and without the model overlay. This measures two things. The first is reading time, which tests the workflow-cost claim. The second is whether the overlay pulls gaze toward the model's output, which would contaminate the signal in live use. This contamination is the main threat to deployability, and we test it rather than assume it away.
Annotation-selection study. The last experiment is about labeling. We fit Split Platt Scaling on the 20, 50 or 100 top-disagreement cases and compare against random and entropy-selected sets. Each method's labeling budget is counted in expert-correction minutes.
Decisive checks. The proposal succeeds if real-gaze disagreement beats perturbation instability on cross-center targets, if shuffled gaze falls to chance, and if overlay-influenced gaze retains most of the signal. If shuffled gaze performs nearly as well as real gaze, the score is measuring image priors rather than the reader, and the idea fails.
References
- Avoiding Catastrophic Referral Failures In Medical Images Under Domain Shift
- GazeRefine: Expert Gaze as a Test-Time Prompt for Training-Free Medical Image Segmentation
- AMOS: A Large-Scale Abdominal Multi-Organ Benchmark for Versatile Medical Image Segmentation
- Adapt the Data, Not the Model: Input-Space Adaptation for Frozen Time-Series Predictors
- Improved Post-hoc Probability Calibration for Out-of-Domain MRI Segmentation
- On-the-Fly Test-time Adaptation for Medical Image Segmentation
- Multi-Scale Global-Instance Prompt Tuning for Continual Test-time Adaptation in Medical Image Segmentation
- TransNetR: Transformer-based Residual Network for Polyp Segmentation with Multi-Center Out-of-Distribution Testing