Attention over a 3D scan is expensive. In self-attention, every small cube of voxels (a token) is compared with every other one, and because the number of tokens grows with the cube of the scan's side length, the cost grows with roughly its sixth power. A radiologist faces the same volume and is not slowed down by this, because they never look at all of it at once. They scroll and fixate in an ordered sequence, revisit suspicious regions, and stop once they are satisfied. This note proposes a recognition network that reads a CT volume the same way. It makes one cheap low-resolution pass over the whole scan. A small model trained on real radiologist eye-tracking then predicts a sequence of fixations and a stopping point. At each predicted fixation, the network takes a high-resolution "glimpse" whose sampling points can bend to fit local anatomy, a technique called deformable sampling (the kernel's sampling grid is shifted by learned, input-dependent offsets). The network's compute therefore scales with how long a human would look at the scan, not with its voxel count.
The two-sentence version: predicted expert scanpaths decide where, in what order, and for how long a 3D network spends its expensive high-resolution computation, and deformable sampling adapts each glimpse to the anatomy it lands on. We test whether this human-derived, sequential, variable-length placement does better at matched compute than three alternatives: placement learned with no gaze supervision, placement from a static gaze heatmap, and fixed rigid or dilated windows.
Core Claim
Existing work splits along three lines. Deformable methods learn where to sample purely from task gradients. Vision Transformer with Deformable Attention places its keys and values at offsets from a uniform grid of reference points. It also confines this sampling to low-resolution stages because interpolation is costly. The 3D D-LKA Net in Beyond Self-Attention: Deformable Large Kernel Attention for Medical Image Segmentation warps decomposed large 3D kernels everywhere in the volume. Neither method uses any notion of where an expert would look.
Scanpath models predict gaze but are not used to reduce compute. CT-Searcher in CT-ScanGaze: A Dataset and Baselines for 3D Volumetric Scanpath Modeling predicts radiologist fixations, durations, and a stop probability, but runs on a rigid-grid Swin UNETR backbone. In that system the scanpath is the output, not a controller.
Gaze-conditioned models treat gaze as a fixed bias or a static map. Predicting Radiologist Expertise from 3D Gaze Patterns During CT Interpretation adds a log-dwell-time bias to DINOv2 attention on fixed 2D patches, using recorded gaze. Temporal Gaze Dynamics as Zero-Shot Prompts for Volumetric Medical Segmentation collapses gaze into Gaussian heatmaps and names that collapse as its main limitation.
Our claim is that a predicted scanpath can serve as three things at once. It supplies the reference points for 3D deformable sampling. Its stop signal sets a per-scan compute budget. Its order lets the model accumulate evidence in a memory. Together these give a better accuracy-per-FLOP curve on volume-level CT recognition than uniform, learned-unsupervised, or static-density placement. The variable budget directly targets the fixed 512-prototype quota that SeVeR: Selective Visual Exposure and Retrieval for 3D Medical Image Question Answering identifies as its weakness.
The cost is easy to state. Let $C_0$ be the cost of the coarse pass, $T$ the number of predicted fixations before stopping, $k$ the side length of each glimpse in voxels, and $d$ the feature width. The high-resolution work is then about
$$C \approx C_0 + T \cdot k^3 \cdot d^2,$$
which is linear in $T$ and independent of the full-resolution side length $L$. Global context comes from cross-attention between the $T$ glimpse tokens and the coarse tokens. That attention costs $T$ times the coarse token count, not the square of the full token count.
Research Program
The system has four parts.
Coarse encoder. A coarse 3D encoder runs on a volume downsampled by 4× per axis. This is the "glance."
Scanpath controller. The controller follows CT-Searcher's design: 3D positional encodings over axial, coronal, and sagittal coordinates, plus an autoregressive decoder over past fixations. It outputs a 3D mixture-density distribution over the next fixation, as in Pathformer3D: A 3D Scanpath Transformer for 360° Images, along with a duration and a stop probability. We pretrain it on CT-ScanGaze, which has 909 scans and 4,772 minutes of radiologist gaze, using CT-Searcher's synthetic 3D pretraining from chest X-ray gaze to offset data scarcity.
Deformable glimpse module. At each fixation, a query-agnostic offset network borrowed from DAT shifts a $k^3$ grid of trilinear sample points on the full-resolution volume. Predicted duration scales the grid's spatial extent, so long fixations produce wider, denser glimpses. To limit the known failure where unconstrained offsets drift outside meaningful structures, we cap offsets to the glimpse extent and add a weak penalty on their magnitude.
Recurrent memory. A recurrent memory token cross-attends to each new glimpse and to the coarse tokens, then feeds a task head.
Training has two stages. First, the controller is pretrained on gaze alone. Then everything is fine-tuned end to end on the task. Gaze imitation stays in the loss as a regularizer with weight $\lambda$, and gradients reach the fixation positions through the mixture's reparameterized means. The stop decision receives a compute penalty, so the budget varies per scan. Because the controller predicts gaze, no eye tracker is needed at test time. Recorded gaze is used only as an oracle upper bound.
The primary task is volume-level finding recognition on CT-ScanGaze. We train multi-label classifiers over finding categories derived from its 9,332 extracted findings, on the dataset's own splits. This is a task where radiologist gaze is plausibly informative.
The secondary task tests whether a gaze prior learned on one scan population transfers to dense prediction elsewhere. It is NIH Pancreas segmentation, where the decoder fuses coarse features with glimpse features scattered back to their locations. This is a harder test and we expect smaller gains.
We also take a position on the 2D-versus-3D backbone tension raised by the expertise paper, which found DINOv2 beat Med3D and SwinUNETR. We run the coarse encoder both as native 3D and as slice-wise DINOv2 features, and report both.
Baselines
Naive reference: coarse-only. The coarse encoder alone, with no glimpses, at the 4× downsampled resolution. It shows how much the "focus" stage adds at all.
Uniform-glimpse control. The same number of glimpses per scan, placed on a uniform 3D grid of reference points with deformable offsets. This is DAT lifted to 3D.
Strongest efficient full-volume competitors. D-Former from D-Former: A U-shaped Dilated Transformer for 3D Medical Image Segmentation, which alternates local 3D windows with dilated global sampling, and the 3D D-LKA Net. These are the strongest existing answers to volumetric attention cost and to irregular anatomy, respectively.
Token-selection competitor. A greedy-prototype selector in the style of SeVeR, applied to the full-resolution tokens with a fixed budget matched to our mean $T$.
Fairness and the bar to beat. All methods share the same coarse encoder where applicable, the same data splits, the same augmentation, and the same hyperparameter search budget, in GPU-hours, on each model's own knobs. Results are reported as accuracy-versus-measured-FLOPs and peak-memory curves, not single points.
For segmentation, the published bars are 81.22% DSC on NIH Pancreas for 3D D-LKA Net and 88.83% DSC on Synapse for D-Former. We will rerun both under our protocol. For finding recognition on CT-ScanGaze, no reported result found on Lacuna, so every baseline is trained by us.
"Beating" a baseline means one of two things on the finding task. Either we match the best rerun full-volume baseline's macro-AUROC at no more than half its FLOPs, or we exceed it by a margin outside three-seed variance at equal FLOPs. For segmentation, we count it as a positive transfer result to come within one DSC point of D-LKA at substantially lower memory.
Ablations isolating the key ingredient. Gaze-free controller ($\lambda = 0$ and no gaze pretraining). Static-density placement, which samples $T$ points from the controller's marginal 3D fixation density with no order and no memory. Fixed $T$ in place of the learned stop. Rigid instead of deformable glimpses. Recorded-gaze oracle placement on held-out scans that have gaze.
Why not just…?
Why not just learn glimpse locations from the task loss, without any gaze data? This is the most serious threat. DAT and D-LKA show that task-learned offsets already beat rigid grids. Sampling in 3D from a gaze-free policy is also exactly how recurrent attention models are usually built. Our bet is that with only 909 scans, task gradients are too weak to find small, low-contrast findings, and an expert prior helps there. The $\lambda = 0$ ablation settles this directly at matched FLOPs. If it ties gaze supervision, we will report gaze as unnecessary for compute allocation and keep only the variable-budget result.
Why not just use a static gaze heatmap to choose where to look? The Shmykova et al. pipeline shows static gaze heatmaps lose substantial accuracy relative to manual prompts: SAM-2 Dice of 0.750 versus 0.834 with bounding boxes. Kuemmerer et al.'s observation, as summarized in the file, is that density maps miss most of the dependencies between fixations. Still, for a feed-forward sampler, order may genuinely not matter. That is why the static-density ablation keeps the same point budget. Sequence must earn its place through the memory and the stop signal, by showing better accuracy at lower average $T$ on easy scans.
Why not just use D-Former-style dilated windows or SeVeR-style prototype selection? Both already reduce cost to near-linear. However, D-Former's dilated sampling spends as much compute on empty background as on lesions. SeVeR's prototype count is fixed per scan, and its authors themselves report that it cannot adapt to how complex a scan is. If either matches our curve, the gaze machinery is not justified. Our matched-FLOPs comparison is designed to make that outcome visible rather than to argue it away.
Proposed Experiments
Controller fidelity. On held-out CT-ScanGaze scans, the controller should stay within CT-Searcher's reported ScanMatch of 0.1466±0.009 and MultiMatch position score of 0.7859±0.033, both from the CT-ScanGaze paper. If fine-tuning with the task loss degrades these scores, the scanpaths are no longer human-like, and we report how far they drift as $\lambda$ varies.
Primary efficiency experiment. We measure macro-AUROC on CT-ScanGaze finding recognition as a function of FLOPs and peak memory, across all baselines and ablations, with three seeds each.
Decisive checks. Three results matter most. First, the gaze-supervised controller must beat the $\lambda = 0$ controller at equal compute. Second, sequential placement with a learned stop must beat static-density placement at equal or lower average $T$. Third, the learned per-scan $T$ should correlate with finding count. That third check is a direct test of whether the budget adapts to scan complexity.
Diagnostics. The oracle-gaze run bounds how much a better controller could add. Stratifying results by finding size tests the claim that gains concentrate on small lesions, which is where coarse passes and token pruning are expected to fail.
Transfer. Running NIH Pancreas segmentation with the frozen gaze prior tests whether radiologist search behavior learned on one scan population transfers to a different organ and task. We treat a negative result here as informative rather than as a failure of the main claim.
References
- Vision Transformer with Deformable Attention
- Beyond Self-Attention: Deformable Large Kernel Attention for Medical Image Segmentation
- CT-ScanGaze: A Dataset and Baselines for 3D Volumetric Scanpath Modeling
- Predicting Radiologist Expertise from 3D Gaze Patterns During CT Interpretation
- Temporal Gaze Dynamics as Zero-Shot Prompts for Volumetric Medical Segmentation
- SeVeR: Selective Visual Exposure and Retrieval for 3D Medical Image Question Answering
- Pathformer3D: A 3D Scanpath Transformer for 360° Images
- D-former: a U-shaped Dilated Transformer for 3D medical image segmentation