Spending a Phone Screen's Pixels Between Detail and Depth in 3D Scans

An on-call physician reviewing a trauma or stroke CT on a phone faces two problems at once. The first is the small screen. The second is that a CT scan is a 3D volume made of hundreds of stacked image slices, and its meaning lives across those slices. A round bright spot on one slice might be a vessel or a tumor margin, and only its continuation through neighbouring slices tells you which. Medical AI work has found that systems which look at slices in isolation lose this cross-slice context and fail. It has also found that, at a fixed budget of visual tokens, a model gains more from sharper individual slices than from more slices. Separately, mobile visualization work has found that shrinking desktop content uniformly onto a phone makes it unreadable, and that endless scrolling breaks the user's overall picture. Nobody has asked the combined question for a human reader. Given a fixed number of physical pixels, how should a phone split them between in-plane detail and depth context so that a clinician answers 3D questions correctly?

We propose to measure this trade-off directly and to build a display around it. For each AI-proposed finding, the display shows a "depth-folded evidence card": a high-resolution crop at the most informative slice, a thin strip showing how the finding continues across depth, and explicit patient-frame coordinate labels. A layout optimizer sizes these parts for the phone's screen. The core claim is that the best split between detail and depth for a human on a phone is a measurable curve. It should differ from the curve a model shows. The cards built from it should beat both plain full-resolution scrolling and the "key slice plus text" output that current 3D medical agents produce. The study also tests a second question: can a vision-language model reading the same screenshots stand in for human readers when choosing that split? If it can, the method can be tuned cheaply. If it cannot, that is itself a warning about using models as proxies for human readers.

What prior work establishes and leaves open

Three results motivate the problem. Deep convolutional networks for pancreas segmentation in CT imaging showed that independent 2D processing gives jagged, inconsistent 3D organ boundaries. Your other Left! showed that vision-language models answer basic left/right and above/below questions about medical images at roughly chance. They rely on memorized anatomy text rather than the pixels, which matters for any short summary shown on a phone. CARVE compresses 3D scans for multimodal language models by choosing anchor slices and folding the rest into them. It adds a residual branch that scans every unselected slice for local deviations, because tiny lesions can span only one or two slices. It reports that at a fixed token budget, in-plane resolution matters more than slice count. That finding is about a model's input budget. Our proposal asks whether it carries over to a human's pixel budget, where it might not: a person can follow depth continuity from a coarse side-view strip far more cheaply than a model can.

Two systems supply components we reuse. 3DMedAgent uses a 2D multimodal model as a coordinator over 3D tools. Its stages are an organ-metadata memory, 3D heatmaps that narrow down candidate regions, and a loop that inspects one slice at a time and writes findings as text. It reports 66% average accuracy on DeepTumorVQA. We take its candidate regions and selected slices as the input to our cards. It runs entirely on a server and never addresses how a clinician sees its evidence. MobileVisFixer breaks web charts into declarative parts and uses reinforcement learning over 23 layout actions to fix five mobile failures, including out-of-viewport content, unreadable fonts and cluttered text. It uses "reactive geometry" to keep data encodings correct after compaction, and resolved 89% of issues. We reuse its failure taxonomy and its idea of resizing parts while preserving what they encode. It handles only static Cartesian charts, however, and explicitly avoids changing aspect ratios. Our depth strip is a non-uniform rescaling along one axis, so we must test that choice empirically rather than inherit MobileVisFixer's caution. OmniCT shows that adding absolute X/Y/Z positional information and organ-level magnification reduces hallucination in a 7B CT model. We carry that lesson to the display through explicit coordinate labels and organ-relative position text drawn from the segmentation memory rather than from the language model.

Two lines of work constrain the design. The VR and cross-device workspaces of Breaking the Screen and PoVRPoint escape the small screen by adding headsets and tracked pens, which an on-call clinician does not have. We therefore restrict ourselves to a single phone. Both 3DMedAgent and OmniCT depend on an upstream organ segmenter that can fail on pathological anatomy. Our cards therefore always include a residual "elsewhere" indicator computed in the style of CARVE's residual branch over the whole volume, so a segmenter or heatmap miss does not silently remove a finding.

Core Claim

Let $A$ be the pixel area available for one finding on the phone. Let $\alpha \in [0,1]$ be the fraction given to the in-plane crop, with the remaining $1-\alpha$ going to the depth strip and coordinate annotation. Let $\mathrm{Acc}_H(\alpha)$ be human reader accuracy on 3D questions about lesion presence, extent, location and relation to organs, and let $\mathrm{Acc}_M(\alpha)$ be the same accuracy for a vision-language model reading identical screenshots. We claim three things. First, $\mathrm{Acc}_H$ peaks at an interior $\alpha^\*$ that is clearly below 1, so some depth context is worth pixels even on a phone, despite CARVE's model-side result favouring in-plane detail. Second, cards built at $\alpha^\*$ match or beat full-resolution scrolling at lower reading time. Third, whether $\arg\max \mathrm{Acc}_M$ tracks $\alpha^\*$ is an open empirical question that we will settle.

Research Program

The pipeline has three stages. A server stage runs 3DMedAgent's memory and heatmap stages to propose candidate regions, and separately computes CARVE-style residual deviations across all slices. The card stage builds the in-plane crop at the chosen slice. Its depth strip is a coronal or sagittal reformat through the candidate, shown as a narrow band with the current slice marked. Its labels state patient-frame coordinates and organ-relative position computed geometrically from the segmentation rather than generated by the language model. A layout stage on the phone adapts MobileVisFixer's action space to these card parts. Because the card is a raster image, the optimizer's job is to pick $\alpha$, the crop zoom and how many cards appear per screen, subject to minimum legible label size and no clipping.

The analysis stage sweeps $\alpha$ over a grid at two screen sizes to trace $\mathrm{Acc}_H(\alpha)$ and $\mathrm{Acc}_M(\alpha)$. Tapping a card always opens full-resolution scrolling, so the cards are a starting point, not a replacement for the scan.

Baselines

The "do nothing" reference is plain full-resolution slice scrolling on the phone, with standard pinch-zoom and a depth slider. It is what clinicians use today, it loses no information, and it is the real threat: radiologists are trained to integrate depth by scrolling. It will be given the same scans, questions and time limit. Beating it means equal or higher accuracy with significantly lower time-to-answer, or higher accuracy at equal time.

The strongest existing method is the 3DMedAgent output rendered directly: its selected slices as images plus its textual findings. 3DMedAgent reports 66% average accuracy on DeepTumorVQA as a fully automatic system, against 37% for RadFM and 39% for M3D. We found no reported result on Lacuna for any human-reader or mobile-display evaluation of 3DMedAgent or of any 3D medical model. The agent's 66% therefore serves as the automatic reference point, and the human comparison is new. Both conditions receive the same candidate regions, so any difference comes from the display alone. Beating it means higher human accuracy on extent and location questions, and fewer left/right errors.

A uniform-downscale condition fits the whole slice to the screen with no cropping. It represents the naive responsive-design approach that MobileVisFixer argues against.

A thick-slab maximum-intensity projection condition shows each region as a single image. In that image, every pixel shows the brightest value across a slab of neighbouring slices. This is the standard way radiology folds depth into one picture, and it is the simplest competitor to our depth strip.

Two ablations isolate the key ingredients. Setting $\alpha=1$ removes the depth strip and leaves crop plus labels. Replacing the geometric coordinate labels with language-model text tests whether the grounding matters. Every condition uses the same server-side candidates, the same phone, the same layout legibility constraints and an equal design-tuning budget.

Why not just…?

Why not just let clinicians scroll the full-resolution stack? This is the strongest alternative, and we treat it as the primary baseline rather than arguing it away. The mobile literature, summarized via Ryen W White et al., suggests small screens hurt the kind of multi-step browsing needed to build a mental model of complex data. That evidence is not specific to radiology, however. If scrolling matches the cards on both accuracy and time, the proposal's practical claim fails, and we will report that. The $\alpha$ curve would still stand as a finding.

Why not just show the agent's key slice and its text report? CARVE's need for a residual branch shows that single selected slices miss lesions spanning one or two slices. The Wolf et al. result shows that model-written spatial language is unreliable. Both failures are invisible to a reader shown only a slice and text. The "rendered 3DMedAgent" baseline and the language-model-label ablation measure exactly this.

Why not just use slab maximum-intensity projections? They compress depth cheaply and clinicians know them. They also merge overlapping structures, so a lesion beside a vessel can vanish into it. Whether this outweighs the cost of a separate strip is an empirical question that the projection condition answers directly. If projections match the cards, we will adopt them as the depth component.

Why not just send a text report? OmniCT documents confident false findings from general models without spatial grounding, and a text-only display gives the reader no way to check them. We include text-only as a minor condition mainly to quantify unverifiable error.

Proposed Experiments

Questions and scans come from DeepTumorVQA abdominal CT. We use the subset with presence, size, location and organ-relation questions, plus AMOS-MM for multi-organ cases. MedEval-CT-Bench-3D provides the model-reader sweep, where OmniCT's reported 77.63 gives a scale for model accuracy on full inputs.

The model-reader sweep renders screenshots at each $\alpha$ and each screen size and scores a vision-language model on them. It is cheap and covers many scans. The human study recruits radiology residents and attending physicians in a within-subject design that counterbalances conditions across matched cases, all on one phone model. The outcomes are accuracy, time-to-answer, left/right and above/below errors, number of taps into full scrolling, and lesions missed that span two or fewer slices, the case CARVE identifies as hardest.

The decisive checks are as follows. If $\mathrm{Acc}_H(\alpha)$ is highest at $\alpha=1$, depth context is not worth pixels for humans on phones, and the depth strip should be dropped. If the cards fail to reduce time against full-resolution scrolling at non-inferior accuracy, the system has no practical case. If the model-reader optimum falls far from the human optimum, models cannot be used to tune displays for human readers, which is a useful negative result for anyone tempted to use models that way.