Dictation as Free Supervision for Separating Looking from Pointing in Radiologist Gaze

When a radiologist reads an image, an eye tracker records exactly where they looked. That record could replace the hand-drawn boxes and outlines that medical segmentation models need and that experts rarely have time to produce. The obstacle is an old problem from gaze-controlled interfaces called the Midas Touch problem: eyes are always moving, so a system cannot tell a glance made while searching or deliberating from a fixation that means "this is the finding." Current gaze-to-segmentation pipelines mostly ignore the distinction. They sum all fixations into a heatmap, weighted by how long each one lasted, so every hesitation over healthy tissue leaks into the prompt.

This note proposes using the radiologist's own dictation as a free teacher for that distinction. In routine reading, radiologists speak their findings aloud, and gaze on an abnormality peaks about 2.5 seconds before the finding is spoken (REFLACX). The fixations in the window just before each spoken finding are therefore weakly labelled as "intent," and the rest as "exploration," at no annotation cost. We train a small sequence model on these labels. It reads the raw scanpath, meaning the ordered sequence of fixations and the jumps between them, and scores each fixation for intent. It then turns only the intentional fixations into prompts for a promptable segmentation model. At test time it can use speech when speech is present, and it can also run on gaze alone, which is the harder and more useful case.

Core Claim

The claim is that a scanpath intent decoder trained only on dictation timing localizes findings from passive, natural reading gaze better than the aggregation rules used today. Those rules include duration-weighted heatmaps, dwell thresholds, spatial clustering, and a fixed 2.5-second look-ahead window. The claim has two parts.

First, the decoder should work when speech is removed. That would show it has learned something general about what intentional looking is, rather than simply relearning the lag between gaze and speech.

Second, the decoder trained on dictation labels should come close to the same decoder trained on intent labels derived from the expert-drawn ellipses in REFLACX. If it does, the supervision needed to solve the Midas Touch problem costs nothing beyond the reading the radiologist already does.

Relation to Prior Work

Two gaze-to-mask systems motivate the gap because both collapse time.

  • GazeRefine turns fixations into a duration-weighted heatmap. It uses that heatmap to seed foreground and background prototypes in frozen DINOv3 features and refines them by nearest-neighbour propagation. It assumes every sustained fixation is foreground and handles only a single structure. - Temporal Gaze Dynamics as Zero-Shot Prompts blurs fixations into per-slice heatmaps and derives boxes with K-Means for SAM-2 and MedSAM-2. Its authors acknowledge that it discards scanpath order. Its gaze-derived prompts reach Dice 0.750 and 0.759, against 0.834 and 0.844 for manual boxes on WORD, and the authors attribute part of that gap to stray fixations.

We reuse these systems' downstream machinery, prompting a frozen segmentation model. What we replace is the step that decides which fixations count.

Two further approaches avoid the problem by constraining the user rather than decoding the gaze.

We keep natural reading intact and move the burden to the decoder.

The weak-labelling idea comes from the gaze-and-speech corpora. REFLACX supplies the look-ahead finding and the data. FG-CXR already pairs gaze heatmaps with sentences using keyword filters and heuristic time windows, and EGMA uses sentence-level gaze to supervise vision-language alignment. Both use that alignment offline to generate reports or learn representations, and both treat all gaze within a window as relevant. FG-CXR is the closest prior work. It differs from this proposal in three ways:

  • It uses a heuristic window and never learns a per-fixation intent model.
  • At inference it uses predicted gaze, not live gaze.
  • It does not produce localization masks.

Our HCI grounding comes from gaze-contingent decoding of wheelchair navigation intent. That work showed that intentional looking has a separable spatial and temporal signature, with 78.8–86.2% accuracy using SVM and KNN classifiers over three object classes. It needed hand-collected intent labels. We obtain those labels from dictation.

Two results constrain the design:

  • Expert and novice scanpaths differ sharply (Predicting Radiologist Expertise from 3D Gaze). We must therefore evaluate across readers, not within them.
  • Visible gaze cursors disrupt natural search (EyeGrab). The decoder must therefore stay invisible to the reader.

Research Program

Data. We use REFLACX: 3,032 readings of 2,616 chest radiographs by 5 radiologists, recorded with a 1,000 Hz tracker, with word-level timestamped dictation and radiologist-drawn abnormality ellipses.

Weak labels. For each spoken finding $k$ with utterance onset $s_k$, fixation $i$ at time $t_i$ receives a soft intent weight $w_{ik} = p(s_k - t_i)$. Here $p$ is the distribution of lags between gaze and speech. We estimate $p$ from the training readers' data rather than fixing it at 2.5 seconds, and we fit it either per reader or pooled. Fixations with negligible weight for every finding are labelled exploration.

Decoder. The decoder is a small transformer over the fixation sequence. Each fixation carries:

  • its position,
  • its duration,
  • the amplitude and direction of the incoming and outgoing saccade (the rapid eye jump between fixations),
  • how often the gaze revisits the same spot,
  • a frozen patch feature at that location.

The decoder outputs an intent score for each fixation. It also outputs a grouping of fixations into findings, which lets it handle multiple findings per image. It has two modes: speech-conditioned, with finding-onset tokens added to the input, and speech-free.

Prompting. The intent-weighted fixations in each group become a box and a set of positive points for frozen MedSAM. Low-intent fixations that land far from the group become negative points. That gives a hands-free retraction mechanism, which the Gaze-Assisted MedSAM authors note their system lacks.

Analyses. We will check which scanpath features carry intent, how stable the gaze-speech lag is across readers, and whether the decoder's exploration fixations concentrate on anatomy that radiologists typically deliberate over.

Baselines

The naive reference point is the all-fixation duration-weighted heatmap for the full reading, thresholded into a box for MedSAM. This is what GazeRefine and EGMA-style pooling assume. Using it tells us how much Midas Touch noise costs.

The strongest existing heuristic is the fixed look-ahead window. It keeps fixations 0–5 seconds before each finding's onset, centred on the REFLACX 2.5-second peak, in the style of FG-CXR's time-windowing. We also include K-Means boxes from Temporal Gaze Dynamics and a dwell-threshold filter, which keeps only fixations longer than a tuned duration. The heuristic parameters for these baselines (window bounds, cluster count, dwell threshold) are tuned on the same validation readers with the same search budget as the decoder's hyperparameters. All methods use the same frozen MedSAM.

For reference numbers, REFLACX reports a normalized cross-correlation (NCC) of 0.380 between gaze and ellipses, against a non-specific baseline of 0.326. The file reports no result on Lacuna for gaze-prompted segmentation or box IoU on REFLACX. So we will re-run every baseline ourselves rather than quote one. Beating the fixed window means a statistically significant gain in IoU between the prompt box and the ellipse, pooled over held-out readers. Beating the heatmap baseline should also show up as an NCC gain over 0.380.

The upper reference is a manual-box oracle: the bounding box of the radiologist's own ellipse, used as the MedSAM prompt. It sets the ceiling that Temporal Gaze Dynamics found gaze could not reach.

The key ablation is the ellipse-supervised decoder. It uses the same architecture, but intent labels come from whether each fixation falls inside an ellipse. If the dictation-supervised decoder closes most of the gap to it, the labels are genuinely free. A second ablation is a shuffled-lag control, in which weak labels come from random windows. It checks that the decoder is learning from timing and not from fixation duration alone.

Why not just…?

Why not just use the fixed 2.5-second window? This is the real threat. With speech available, a well-tuned fixed window may capture most of the benefit, and if the decoder does not beat it we will report that plainly. The decoder earns its place in two situations. One is speech-free mode, where a window has nothing to anchor on and the comparison is against the heatmap, dwell and K-Means baselines. The other is readers whose lag distribution departs from 2.5 seconds, which the leave-one-reader-out split exposes.

Why not just filter by dwell time? In clinical reading, long fixations often mean deliberation over a differential diagnosis, not selection. The HCI evidence (Talk-and-Gaze, GazeGrasp's 3-second threshold) shows dwell gating either triggers falsely during pauses or becomes too slow. Duration is one of the decoder's input features. The dwell baseline and the shuffled-lag control together measure how much scanpath order adds beyond duration.

Why not just have radiologists draw boxes? Boxes are more accurate: 0.844 against 0.759 Dice with MedSAM-2 (Temporal Gaze Dynamics). They are also faster in active use: 5.7 seconds against 9.7 seconds (Gaze-Assisted MedSAM). Our setting adds zero interaction time, because the labels come from reading that happens anyway. The honest measure is therefore the fraction of oracle-box accuracy recovered at no extra cost. We report it directly, so a reader can judge whether the remaining gap still justifies the extra box-drawing work.

Why not just train a detector on the ellipses? Ellipses are themselves manual annotations, which is exactly the bottleneck. The ellipse-supervised decoder already bounds what such labels buy for the specific task of filtering intent.

Proposed Experiments

Main experiment. We use a leave-one-radiologist-out evaluation on REFLACX and compare all methods on three measures, pooled across findings:

  • IoU between the prompt box and the ellipse,
  • Dice between the MedSAM mask and the filled ellipse,
  • NCC between the intent-weighted gaze map and the ellipse.

We report results separately for images with one finding and images with several, since multiple findings are where heatmap methods should fail.

Decisive checks.

  1. The speech-free decoder must beat the all-fixation heatmap and the K-Means baselines on held-out readers.
  2. The dictation-supervised decoder must come within a small margin of the ellipse-supervised one.
  3. The shuffled-lag control must collapse toward the dwell baseline.

If the first check fails, dictation only supplies timing and the method reduces to the window heuristic. If the second fails, dictation labels are too noisy to substitute for manual labels.

Transfer check. A secondary test applies the frozen speech-free decoder to volumetric CT reading. The candidate data are the LIDC-IDRI sessions in Predicting Radiologist Expertise, scored against LIDC nodule annotations, if the gaze data are obtainable. Success there would indicate that the intent signature carries across modalities without any dictation in the target domain.