Scientists often need to see a structure buried inside a 3D scan: a lesion behind a colon fold, a nucleus under the cortex, a thermal hotspot inside a tangle of flow lines. Doing so means adjusting the view until the outer layers stop blocking the target. This includes camera angle, cut-away planes, and how transparent each tissue is. AI agents are now being asked to drive scientific software. Yet no one can check reproducibly whether an agent succeeded at this basic task. Current agent benchmarks run in sealed, repeatable software environments and grade only files, database state, or text logs. They cannot tell whether the picture an agent left on screen shows the target or hides it.
We propose a visibility contract: a pass/fail check computed from the viewer's saved scene state rather than from a screenshot or a language model's opinion. An independent reference renderer, running on the CPU inside a pinned container, replays the camera, clipping, and transparency settings and measures two things. The first is how much light from the target voxels reaches the camera unblocked. The second is how much surrounding context survives. This turns "occlusion", the field's word for outer structures hiding inner ones, from a subjective visual judgment into a deterministic grading function. That makes the first repeatable benchmark possible for agents working on dense 3D data.
Core Claim
Success at an occlusion task can be graded by three conditions. The target must be visible. The context must be preserved. The underlying data must be left untouched. Once the scene state is exported and re-rendered in a hermetic environment, this verdict is cheap and identical across machines. It also agrees with expert judgment better than a vision-language judge looking at screenshots, and better than matching the agent's final settings against an expert's. The context condition is essential. Without it the trivial solution, making everything except the target transparent, passes any visibility test. With it, the contract rewards what the occlusion literature actually aims for: revealing the interior while keeping enough surrounding anatomy to orient the viewer.
For a rendered state $s$ and target voxel set $T$, define target visibility as
$$V_T(s) = \frac{1}{|T|}\sum_{x \in T} \mathbf{1}\left[\tau_s(x) \ge \theta\right],$$
where $\tau_s(x)$ is the accumulated transmittance from the camera to voxel $x$. Transmittance is the fraction of light not absorbed by material in front of $x$, and it is zero if $x$ is clipped away or off-screen. $\theta$ is a fixed threshold. Context retention $C(s)$ is the fraction of rendered opacity contributed by non-target structure, taken relative to the default view. A task passes when $V_T \ge v^\*$ and $C \ge c^\*$, and when a before/after snapshot shows the source data unchanged.
Where This Sits in Prior Work
The occlusion side supplies the problem and the solution methods, but not a grading criterion. Four papers resolve occlusion by changing the geometry or the view. LMap locally flattens a folded surface and raises visible colon-segment coverage from 72% to 94%. Angle-preserving mappings for multi-branched vessels and Geometry-Aware Planar Embedding of Treelike Structures unroll branched vessels and neurons into flat maps. Previewing Volume Decomposition Through Optimal Viewpoints peels nested layers along a camera path chosen to maximize viewpoint entropy.
Two further methods fade out whatever sits in front of the target instead. Distance Based Enhancement for Focal Region Based Volume Rendering weakens voxels by their distance from a focal cursor. Depth Attenuation Degree Based Visualization weakens them by depth, coupled to physiological data.
All of these methods are evaluated with their own proxies, such as coverage, angle loss, or entropy. Their parameters are tuned by hand. We reuse coverage-style reasoning as the basis of $V_T$. We use the focal-attenuation method as a scripted oracle baseline, since it is exactly the "fade the foreground" strategy an agent might discover. The Rational Agent Benchmark for Data Visualization cautions that a visually rich image can carry no decision value, and this constrains our claim. Visibility is necessary but not sufficient, so we validate the contract against a downstream question about the target.
The execution side supplies the method for sealed, repeatable grading, but it is blind to images. Agent-Diff grades agents by the net change in containerized application state rather than by their action traces. Our contract is a state-based check of the same kind, applied to rendered scene state instead of API records. The SciAgent audit harness pins scientific tools in containers and uses a separate verifier model on logs and numeric outputs. It has no visual checks, so an OpenFOAM run can converge while its visualization hides the hotspot. MacArena and Computer Agent Arena run computer-use agents in fresh virtual machines, but only on 2D office and desktop apps.
On the imaging side, SABER and Fractal make volumetric processing reproducible. Fractal reports over 99.99% replicate agreement. Both leave viewing to manual inspection in external viewers such as napari. SP-Mind runs a code-writing agent over containerized imaging tools, but on 2D tissue slides. None of this work grades whether a 3D view reveals anything.
Research Program
We will build container images holding napari and ParaView with offscreen rendering. Both applications expose a scripting interface and a GUI. The images will be launched copy-on-use in the MacArena style: each task starts from an immutable image and discards it afterwards. The key engineering step is a scene-state exporter. It serializes the camera, clipping planes, per-structure transfer functions (the mapping from voxel value to color and opacity), and any local deformations to a JSON record. A separate verifier image then renders that record with a CPU software rasterizer and a fixed ray-marching step to compute $V_T$ and $C$. Using the CPU keeps GPU-driver differences out of the verdict. The agent never touches this path.
We then build roughly 200 tasks. Each task gives a volume and a natural-language request: "show the aneurysm neck without removing the parent vessel," or "make the rack hotspot visible through the streamlines." Target masks come from datasets that include segmentations, or are planted synthetically so the ground truth is exact. The sources are all drawn from the research file:
- the sheep heart and nuclear-physics volumes from the viewpoint-decomposition paper;
- HCP cortical meshes and colonoscopy surfaces as used by LMap;
- Kasthuri EM and XRM vessel volumes processed by SABER;
- OME-Zarr organoid stacks from Fractal;
- thoracic CT from CT-RATE and RAD-ChestCT, with planted low-contrast nodules. These are the subtle findings that Big, Bright, or Invisible showed frozen encoders fail to detect.
- an OpenFOAM rack-cooling case following SciAgent.
Thresholds $v^\*$ and $c^\*$ are fixed on a held-out calibration split before any agent is scored. Agents are evaluated in two modes: code-writing through each viewer's Python interface, and GUI control through screenshots and clicks.
Baselines
The naive reference point is doing nothing: leave the default camera and transfer function and score the scene. This measures how hard each task is. It anchors the claim that tasks are nontrivial, by analogy to LMap's 72% default-view coverage on colon segments.
The strongest non-agent competitor is a scripted oracle. It is given the target location and applies distance-based focal attenuation around it, then picks the maximum-entropy viewpoint from the decomposition paper. This is what the occlusion literature would do with perfect knowledge of the target. An agent matching it under the contract would count as strong performance. The oracle runs in the same container with the same exporter.
For agents, the closest reported numbers are on neighbouring tasks, not on 3D occlusion:
- OpenAI Computer Use Preview reaches 31.83% overall success on MacArena (MacArena).
- SP-Mind reaches 68.9% execution accuracy on SP-Bench (SP-Mind).
No reported result was found on Lacuna for agents on volumetric occlusion tasks. We will therefore run the same frontier models ourselves, with the same step cap of 50 as Computer Agent Arena, the same prompts and the same token budget in both modes.
For the verifier question, the competitors are three alternative graders applied to identical final states. The first is a vision-language judge reading the final screenshot plus the task text, in the style of SciAgent's fresh-context verifier. The second is a parameter-matching grader comparing the agent's final settings to an expert reference. The third is the viewer's own GPU render of a target-colored ID buffer, meaning an image where each pixel records which object it shows. The contract beats them if two things hold. It must have higher agreement with blinded expert verdicts, measured by Cohen's kappa with overlapping bootstrap intervals excluded. It must also have lower pass rates on adversarial gaming attempts.
The key-ingredient ablation drops the context term $C$ and keeps visibility only. It shows whether context retention is what stops the "delete everything else" exploit. A second ablation swaps the CPU reference renderer for in-viewer GPU rendering across hosts, isolating the value of hermetic re-rendering.
Why not just…?
Why not just ask a vision-language model whether the target is visible in the screenshot? This is cheap and handles arbitrary tasks. It is a real competitor for loose tasks. But SciAgent's authors note that a model verifier's "VERIFIED" only means the evidence looks consistent under heuristic scrutiny, and small low-contrast targets are exactly what vision encoders miss according to Big, Bright, or Invisible. The agreement study and the adversarial study settle this directly. If the judge matches the contract's kappa with experts and resists gaming equally well, we will report that the contract adds reproducibility but not accuracy.
Why not just compare the agent's final settings to an expert's? This is the 3D analogue of call-trace matching. Agent-Diff argues it fails whenever several valid solutions exist, and occlusion tasks have many: a different angle plus a clipping plane can reveal the same nucleus as a transparency change. We measure this by collecting two independent expert solutions per task. If both expert solutions pass the contract while each fails parameter matching against the other, parameter matching is shown to reject correct work.
Why not just render an ID buffer in the viewer itself? This is the most serious threat, because it is close to our method. The difference lies in who renders and where. An in-viewer ID buffer trusts the agent-controlled process and the host GPU. It also counts a target pixel as visible even when semi-transparent foreground heavily attenuates it, since an ID buffer records which object sits in each pixel, not how much light survives. If our cross-host experiment shows GPU verdicts identical to CPU verdicts, and binary ID visibility matches transmittance-weighted visibility on expert agreement, then the simpler design wins and we will adopt it. That would be a useful finding in its own right.
Proposed Experiments and Decisive Checks
The first decisive check is validity. Three blinded experts per domain view final states from agents, oracles and deliberately bad settings, and mark whether the target is clearly visible with adequate context. We report kappa for each grader. As a rational-agent-style secondary check, experts also answer a factual question about the target from the image alone, such as the side of a stenosis or the presence of a nodule. We test whether contract passes predict correct answers better than the alternatives. This addresses the concern that visibility without decision value is meaningless.
The second check is reproducibility. The same 500 exported states are verified on three hosts with different CPUs and GPUs. We report the verdict flip rate for the CPU reference path versus the in-viewer GPU path, targeting zero flips for the former.
The third check is robustness. A red-team agent is explicitly instructed to pass each grader without genuinely revealing the target, for example by blanking context, zooming onto an unrelated bright structure, or editing the data. We report the gaming success rate for each grader, with and without the context term.
Only after these three checks do we report agent results: success by mode, domain, and target contrast. We also test whether exposing the focal-attenuation operation as a callable tool closes the gap to the oracle.
The proposal fails if experts reject contract-passing views at high rates, or if the screenshot judge proves equally accurate and equally hard to game. Either outcome would be clear, and publishable as a negative result.
References
- LMap: Shape-Preserving Local Mappings for Biomedical Visualization
- Angle-preserving mappings for the visualization of multi-branched vessels
- Geometry-Aware Planar Embedding of Treelike Structures
- Previewing Volume Decomposition Through Optimal Viewpoints
- Distance Based Enhancement for Focal Region Based Volume Rendering
- Depth Attenuation Degree Based Visualization for Cardiac Ischemic Electrophysiological Feature Exploration
- The Rational Agent Benchmark for Data Visualization
- Agent-Diff: Benchmarking LLM Agents on Enterprise API Tasks via Code Execution with State-Diff-Based Evaluation
- Audit-Grade Harness for Agent-driven Scientific Computation Workflows
- MacArena: Benchmarking Computer Use Agents on an Online macOS Environment
- Computer Agent Arena: Toward Human-Centric Evaluation and Analysis of Computer-Use Agents
- Toward a scalable framework for reproducible processing of volumetric, nanoscale neuroimaging datasets
- Fractal: Towards FAIR bioimage analysis at scale with OME-Zarr-native workflows
- SP-Mind: An Autonomous Reasoning Agent for Spatial Proteomics Analysis
- Big, Bright, or Invisible: A Frozen-Feature Benchmark of 3D CT Foundation Models