Language-vision models that answer questions about CT scans have a budget problem. A single abdominal CT holds hundreds of slices, far more image tokens than a model can attend to, so current systems must decide in advance what to throw away. They either compress the whole volume into a few dozen summary tokens, drop slices by a fixed rule, or let the language model guess which slice to open next. All three can miss a small lesion that appears on only one or two slices, and missing that lesion is exactly the clinical failure that matters. We propose treating slice reading as active sensing, the robotics idea of choosing each next observation to shrink the agent's uncertainty. The model keeps a cheap, low-resolution view of the entire volume at all times, a "periphery". At each step it opens the full-resolution slice window whose contents are expected to most reduce its uncertainty about the answer to this particular question. It stops once that uncertainty is low.
Put simply for a colleague: instead of compressing a CT scan before asking the question, let the model look at the scan the way a radiologist scrolls, one slice at a time, always jumping to the depth that would most change its mind about the answer. We estimate "would most change its mind" with a small learned predictor that works on the coarse whole-volume view and ignores background, so the loop costs little more than a single forward pass.
Core Claim
The claim is that question-conditioned expected information gain is a better rule for spending a fixed visual-token budget on a 3D volume than either query-agnostic saliency or free-form language-model choice. The advantage should be concentrated on questions whose evidence is spatially small, meaning it spans one or two slices. The advantage should also appear on questions whose answer does not concern the most visually prominent abnormality in the scan.
Formally, let $E_t$ be the evidence gathered after $t$ glimpses: the periphery plus the slice windows already opened. Let $p(a \mid E_t)$ be the answer distribution the frozen multimodal model assigns over candidate answers $a$, and let $w$ index candidate depth windows. The next window is chosen by
$$ w_{t+1} = \arg\max_{w} \; \Big[ H\big(p(a \mid E_t)\big) - \mathbb{E}_{x_w}\, H\big(p(a \mid E_t, x_w)\big) \Big], $$
where $H$ is Shannon entropy and $x_w$ is the unseen high-resolution content of window $w$. The expectation over unseen content is what makes exact computation expensive. It would require simulating each candidate window through the large model. That multi-step lookahead cost is the open problem the active-vision literature identifies, for example in Active Visual Exploration Based on Attention-Map Entropy.
We amortize the expectation. A small network $g_\phi$ reads the periphery tokens, the question embedding, and the current evidence summary, and outputs a predicted post-glimpse entropy $\hat{h}_w$ for every window at once. It is trained by regression against entropies actually measured offline by running the frozen model on sampled glimpse sequences. This is one-step value regression, not reinforcement learning, in the spirit of Active Sensing with Predictive Coding and Uncertainty Minimization, which makes information-gain action selection trainable without a policy network.
Relation to Prior Work
The closest system is 3DMedAgent. Its "think-with-one-slice" loop has a 2D language model pick informative slices sequentially, guided by segmentation-tool memory and text-image heatmaps, and it reaches 66% average accuracy on DeepTumorVQA. We keep its sequential, closed-loop structure. We replace the heuristic slice choice with a scored uncertainty objective and an explicit stopping rule. We also keep visual periphery tokens in context rather than reducing everything seen to text notes.
CARVE motivates the problem and supplies our strongest static competitor. It is a training-free compressor that splits the volume into depth windows and keeps anchor tokens plus tokens that deviate from neighbouring slices, and it preserves 98.1% of full-token performance at a 20% keep ratio. Its analysis that in-plane resolution matters more than slice count directly motivates our design of a coarse periphery plus full-resolution foveal windows. CARVE, however, decides before seeing the question, so it cannot redirect its budget depending on what is being asked.
TumorChain likewise calls for local crops through discrete, organ-driven decisions. It reports difficulty with small tumours at organ boundaries, exactly where a graded uncertainty signal should help.
Two papers supply method components. The first is Active Visual Exploration Based on Attention-Map Entropy, which selects 2D glimpses training-free by decoder attention entropy. We use it as a direct ablation, because it is the obvious training-free alternative to our learned predictor, and it has never been tested on anisotropic volumes. The second is ASSFT, which shows that voxel entropy over CT volumes is swamped by trivial background and restricts entropy to foreground organs. That work applies the idea to choosing which scans to label; we apply it within a scan, masking air, table and body wall from the periphery features that feed $g_\phi$.
The peripheral-foveal robot of Peripheral-Foveal Vision for Real-time Object Recognition and Tracking in Video is the architectural analogy. Its entropy-directed gaze reached 82.9% recall against 43.1% for linear scanning and 62.2% for exhaustive full-frame processing.
Your other Left! constrains our evaluation. It shows that vision-language models answer positional questions from memorized anatomy rather than pixels, collapsing to about 50% when axial slices are flipped. Any gain we report must therefore survive orientation perturbations, or it may reflect priors rather than looking.
Research Program
We build three components around a frozen 2D multimodal backbone. We use Qwen2.5-VL, which TumorChain reports at 44.04% on its tasks, so it leaves room to improve. We also run a second, stronger backbone to check that findings transfer.
The first component is the periphery: the whole volume resampled to low in-plane resolution and pooled per depth window into a few tokens, always present in context.
The second component is the glimpse scorer $g_\phi$, a small transformer over periphery tokens with a foreground mask derived from an off-the-shelf organ segmenter. It is trained on the training splits of DeepTumorVQA and DeepChestVQA. Its regression targets come from roughly a few hundred thousand logged glimpse sequences per dataset, collected by sampling random, uniform and greedy trajectories and recording the backbone's answer entropy after each glimpse.
The third component is the loop. It opens the argmax window at full resolution, appends its tokens, recomputes $p(a \mid E_t)$ from option log-probabilities (or from a verbalized-answer classifier for open questions), and stops when entropy falls below a threshold or the token budget is spent.
The analysis centres on where the gain comes from. We stratify every benchmark question by lesion extent in slices, derived from segmentation masks where available. We also stratify by whether the question targets the most salient abnormality in the scan.
Baselines
The naive reference is uniform strided slice sampling at the same visual-token budget, feeding the same backbone. We also report the full-slice upper bound where it fits in context, so readers see how much of full performance each method retains. Beating uniform sampling is necessary but not interesting.
The strongest static competitor is CARVE, run with its published windowing and anchor/retrieval split, at token budgets matched exactly to ours (20% keep ratio and two tighter settings), on the same backbone. It is training-free, so we give it a tuning sweep over its anchor/retrieval ratio on validation data equal in size to what we use to tune our stopping threshold. Its reported result is 98.1% retention of full-token performance at 20% keep across its VQA benchmarks. Beating it means higher accuracy at equal tokens on M3D-VQA and 3D-RAD, and specifically on the small-lesion stratum.
The strongest sequential competitor is 3DMedAgent's loop, in which the language model chooses slices from tool memory and heatmaps. We reimplement it on our backbone with the same number of opened windows. Its reported DeepTumorVQA figure is 66% average accuracy, from the 3DMedAgent paper. We treat TumorChain's 73.30% zero-shot lesion recognition as a reference from a natively 3D, heavily fine-tuned model, not a matched comparison. For DeepChestVQA, we found no reported absolute result on Lacuna for 3DMedAgent, only a relative "average 20% accuracy gain", so we will rerun it ourselves. Beating it means higher accuracy with the same or fewer windows opened.
Three ablations isolate the key ingredient. The first replaces $g_\phi$ with a query-agnostic abnormality score, trained identically but regressing lesion presence per window instead of answer entropy. This separates "look where it's abnormal" from "look where the answer is uncertain". The second replaces $g_\phi$ with training-free attention-map entropy in the style of AME. The third removes the foreground mask. A final oracle opens windows containing ground-truth lesion voxels and bounds what any selector can achieve.
Why not just…?
Why not just take the slices of the organ the question names? This is the most dangerous alternative. A segmentation tool plus keyword matching is cheap and may capture most of the gain on single-organ questions. We run it as a baseline at matched budget. We expect it to fail on questions spanning organs or boundaries, which is the failure TumorChain reports, and to waste budget on large homogeneous organs where most slices are redundant, the "redundancy trap" that CARVE documents. If organ-guided selection matches us on the boundary and small-lesion strata, the learned scorer is not earning its cost and we will report that.
Why not just let the language model pick the next slice, as 3DMedAgent does? It already works well, at 66% on DeepTumorVQA. What it cannot do is quantify when to stop or compare candidate depths on a common scale. Our matched-window comparison, and a stopping-efficiency curve (accuracy versus windows opened), will show whether an explicit uncertainty score beats language-model judgment. If it does not, the result says heuristic reasoning already approximates information gain, which is itself worth knowing.
Why not just compress everything up front with CARVE? At generous budgets CARVE may be nearly lossless, and we expect our advantage to shrink there. The case for closed-loop selection rests on tight budgets and on question-specific evidence that a question-blind compressor has no reason to keep. The budget sweep and the salient-versus-non-salient question split test exactly this.
Why not just use attention entropy without training anything? AME's training-free rule is attractive. Its evidence, however, comes from 2D natural images, and attention entropy measures reconstruction ambiguity rather than answer uncertainty. It is also exposed to the background dominance ASSFT identified. The AME ablation settles whether training $g_\phi$ is necessary.
Proposed Experiments and Decisive Checks
We evaluate on DeepTumorVQA, DeepChestVQA, M3D-VQA and 3D-RAD for accuracy. We add AMOS-MM report generation as a secondary task, using entropy of a lesion-presence head as the belief when there is no discrete answer. We report accuracy against visual tokens consumed and against windows opened, together with scorer latency.
The first decisive check targets questions where the lesion occupies at most two slices. At a 20% token budget, the proposed method must beat both CARVE and organ-guided selection by a margin exceeding seed-to-seed variation. It must also close a substantial fraction of the gap to the oracle.
The second decisive check uses questions about a non-dominant finding, for example a small renal lesion in a scan with a large hepatic mass. Here question-conditioned information gain must beat the query-agnostic abnormality scorer. If it does not, the "answer-uncertainty" framing adds nothing over a lesion detector.
The third check applies the orientation test from Your other Left! at the volume level. We flip scans left-right and relabel the expected answers accordingly. A method that truly reads pixels should see its laterality answers follow the flip. We report whether active selection flips answers more faithfully than static compression, which would indicate it pushes the model toward visual evidence rather than memorized priors.
References
- Active Visual Exploration Based on Attention-Map Entropy
- Active sensing with predictive coding and uncertainty minimization
- 3DMedAgent: Unified Perception-to-Understanding for 3D Medical Analysis
- CARVE: Cross-Slice Anisotropic Reallocation of Visual Evidence for Efficient 3D Medical Volume Understanding
- TumorChain: Interleaved Multimodal Chain-of-Thought Reasoning for Traceable Clinical Tumor Analysis
- Adapting Medical Vision Foundation Models for Volumetric Medical Image Segmentation via Active Learning and Selective Semi-supervised Fine-tuning
- Peripheral-Foveal Vision for Real-time Object Recognition and Tracking in Video
- Your other Left! Vision-Language Models Fail to Identify Relative Positions in Medical Images