ScienceVision: Can AI Observe and Reason Like a Scientist?

Tianyi Ma, Jing Gu, Haozhe Zhao, Liang Chen, Yunfan Zhang, Haijun Su, Jialong Wu, Le Xue, Arnav Das, Dustin Tran, Yinfei Yang, Andrew Dai, Kuan LiarXiv 2026 · arXiv

TL;DR

ScienceVision is a new cross-disciplinary benchmark designed to test if AI can reason about 3D scientific data like a human expert. By evaluating 12 state-of-the-art models across domains like biology and medical imaging, the researchers found that giving AI direct access to 3D files and code-execution tools significantly boosts performance over 2D images, though even the best models still only earn about half the available credit.

What Problem Does This Paper Address?

Scientific discovery isn't just about looking at pictures; it requires understanding complex 3D spatial relationships—think of the way a protein folds or how mechanical parts fit together in a CAD assembly. While we have plenty of AI benchmarks for text and 2D images, scientific 3D evaluations have historically been siloed within specific fields (e.g., just medical imaging or just chemistry), making it impossible to compare "scientific reasoning" capabilities across disciplines.

The authors address this gap by introducing ScienceVision, a unified benchmark that forces AI to move beyond simple pattern recognition. It asks: Can an AI agent observe a 3D object, decide which "slice" or "view" is important, and apply domain-specific knowledge to answer complex questions?

Methods

The researchers developed a pipeline that combines synthetic question generation with rigorous domain-expert review. They evaluated 12 models (including GPT-6 Astra and Claude Fable 5.1) using an "agent harness"—a setup where the AI isn't just a chatbot, but an agent that can interact with files and execute code.

The study compared three distinct evaluation settings:

  1. Harness + 3D: The model has access to the native 3D file (like a .pdb for proteins or .step for CAD) and can write code to analyze it.
  2. Harness + 3D + MCP: The same as above, but augmented with the Model Context Protocol (MCP). This exposes expert-inspired tools for visual inspection, such as changing camera views, isolating specific targets, or taking geometric measurements.
  3. Harness + 2D: The model only sees a set of prescribed, static 2D renders of the object.

Crucially, all answers are scored deterministically using task-specific rules (e.g., numerical tolerances for measurements), removing the bias of using an "LLM judge."

Datasets

The core of the paper is the ScienceVision benchmark, which consists of 250 expert-level questions. These are evenly split (50 each) across five domains:

  • Biology: Protein contacts and spatial relations.
  • Chemistry: Molecular conformation and geometry.
  • Medical Imaging: Localization and slice comparison in CT volumes.
  • Mechanical CAD: Part counts and assembly motion.
  • Architecture: Spatial continuity and dimensional judgments.

The dataset covers 22 subdomains, providing a granular look at where models succeed or fail.

The benchmark covers a wide array of subdomains across five scientific fields.

Key Insights and Ideas

  • The "3D Gap" is Real: Models perform significantly better when they can "touch" the 3D data via code and file access rather than just looking at 2D pictures. On average, the native-artifact workflow outperformed the 2D-image workflow by 17.2 percentage points.
  • Tooling is a Double-Edged Sword: Surprisingly, adding specialized MCP inspection tools (like measurement and slicing tools) didn't provide a universal boost. While it helped in Biology, it actually hindered performance in Chemistry for some models, suggesting that current agents struggle to decide when and how to use complex visual tools effectively.
  • Spatial Reasoning vs. Domain Knowledge: The benchmark reveals that a "correct" answer often requires both. For example, in chemistry, a model might correctly identify two atoms but fail to calculate the dihedral angle because it doesn't follow the specific scientific convention required for the answer.

Key Results

  • Top Performance: GPT-6 Astra (using the Codex harness) achieved the highest mean score of 0.5291 out of 1.0.
  • Leaderboard Trends: As shown in Figure 1, there is a clear hierarchy among models, with the GPT-6 and Claude Fable families leading the pack.
  • Domain Strengths: Models generally performed best in Biology (up to 0.65) and Chemistry, but struggled significantly with Medical Imaging—specifically "lesion pointing" tasks, where scores hovered around a dismal 0.10.
Performance of 12 model configurations on the ScienceVision benchmark. GPT-6 Astra leads, but no model exceeds a 53% success rate.

Limitations and Future Work

The authors are transparent about several hurdles:

  • Reliability: Even with native 3D access, models are far from "reliable." A 53% score means the AI is still missing nearly half the expert-level reasoning required.
  • Ambiguity: An independent audit found that some CAD questions had underspecified targets, highlighting the difficulty of creating "perfect" 3D benchmarks.
  • The "Black Box" of Reasoning: It remains unclear whether failures happen because the model can't "see" the target, can't "reason" about the science, or simply can't format the output correctly.

Future work will likely focus on improving how agents interact with visual inspection tools and expanding the benchmark to include even more complex, multi-step scientific discovery workflows.

Summary updated

Abstract

Scientific discovery builds on accurate observations interpreted through domain knowledge and reasoning. For scientific 3D data, this requires understanding structures and spatial relationships, from protein contacts to anatomical organization. Existing multimodal benchmarks largely evaluate text and 2D images, while scientific 3D evaluations vary in representations and protocols across disciplines, limiting cross-disciplinary comparison. To assess these foundational capabilities, we introduce ScienceVision, a cross-domain benchmark for scientific 3D understanding and reasoning through expert-level visual question answering (VQA). We construct 250 questions, with 50 each from Biology, Chemistry, Medical Imaging, Mechanical CAD, and Architecture, through a pipeline combining synthetic question generation with domain-expert review and annotation. Each task pairs a native 3D artifact with a scientific question, a specified response format, and a task-specific reference answer, requiring domain knowledge to be applied to geometric or topological evidence. We evaluate 12 models on the same questions in three settings: visual question answering with native 3D artifacts accessed through an agent harness's file and code-execution tools; the same workflow augmented with expert-inspired viewer inspection and measurement tools exposed through the Model Context Protocol (MCP); and visual question answering from prescribed rendered 2D images using an agent harness. All settings use the same task-specific deterministic final-answer scoring. The inspection environment supports view changes, target isolation, and slice inspection. In this main comparison, the best mean answer score is 0.53 out of 1. Mean scores in the native-artifact workflow without additional MCP tools exceed those in the rendered-image workflow for every model, by 17.2 percentage points on average.
BibTeX
@misc{ma2026sciencevision,
  author = {Tianyi Ma and Jing Gu and Haozhe Zhao and Liang Chen and Yunfan Zhang and Haijun Su and Jialong Wu and Le Xue and Arnav Das and Dustin Tran and Yinfei Yang and Andrew Dai and Kuan Li},
  title = {{ScienceVision: Can AI Observe and Reason Like a Scientist?}},
  year = {2026},
  eprint = {2611.04087},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.04087},
}
Figure