A frontier machine intelligence for human neuroimaging

Chenhui Zhao, Akshay Rao, Akhil Kondepudi, Yiwei Lyu, Xinhai Hou, Rikhil Seshadri, Cheng Jiang, Soumyanil Banerjee, Rushikesh Joshi, Samir Harake, Ashok Srinivasan, Shruti Mishra, Francisco Rivas-Rodriguez, Seunghong Rhee, Nathaniel Roddenberry, Neeraj Chaudhary, Zach Wilseck, Paul Park, Honglak Lee, Vikas Gulani, Aditya Pandey, Todd HollonarXiv 2026 · arXiv

TL;DR

Neuro-VL is a specialized vision-language model (VLM) trained on nearly one million clinical neuroimaging studies, enabling it to interpret complex 3D scans, generate radiology reports, and engage in multi-turn clinical reasoning. By leveraging a hierarchical attention mechanism and a massive private dataset, it significantly outperforms general-purpose frontier models like GPT-4 and specialized medical AI in diagnosing, localizing, and monitoring neurological diseases over time.

What Problem Does This Paper Address?

Artificial intelligence in medical imaging has historically suffered from a "narrowness" problem. Most models are designed for restricted tasks—like detecting a single type of hemorrhage—and lack the broad clinical reasoning required to be a true collaborator for physicians. Conversely, general-purpose frontier models (like GPT-4 or Gemini) are highly capable at reasoning but lack access to the massive, private, volumetric (3D) clinical datasets necessary to develop medical expertise.

Furthermore, neuroimaging is uniquely difficult: a single study can contain hundreds of slices across multiple scans with different contrasts (e.g., T1, T2, FLAIR). Interpreting these requires resolving complex 3D anatomical relationships and integrating findings across time. The authors developed Neuro-VL to bridge this gap, creating a model that can "see" the full complexity of neuroimaging and "talk" through its findings in a clinical context.

Methods

The researchers built Neuro-VL using a three-part architecture:

  1. Hierarchical Visual Encoder: Based on a 24-layer ViT-Large, this encoder uses a novel attention mechanism that mirrors the structure of a radiology study. It processes tokens at three levels: adjacent slices, individual scans, and the entire study. This allows the model to integrate local features with global context.
  2. Dual Contrastive Objective: During pre-training, the model was taught to align visual features with both "focal" findings (individual sentences in a report) and "holistic" summaries (the entire report).
  3. Instruction Tuning with Reasoning Traces: Instead of just predicting a diagnosis, the model was trained to generate a "visual reasoning trace." This is essentially a Chain-of-Thought (CoT) process where the model first describes what it sees in specific scans before synthesizing a final answer.
Overview of Neuro-VL training data, workflow, and evaluation.

Datasets

The scale of the training data is a major highlight of this work:

  • Internal Training Set: Nearly 1 million neuroimaging studies (MRI and CT) from 450,000 patients at Michigan Medicine, comprising 10 million volumetric scans.
  • NeuroBench: A new evaluation benchmark introduced by the authors, combining internal data with 24 international public datasets spanning 21 countries. It includes over 31,000 closed-ended questions and nearly 4,000 report-generation tasks.
  • Longitudinal Data: 230,000 examples of paired prior and current studies to train the model on disease progression.

Key Insights and Ideas

  • Hierarchy is Essential for 3D: Standard 2D vision models struggle with the "stack" of images in an MRI. The hierarchical attention mechanism (slice $\rightarrow$ scan $\rightarrow$ study) is the key to handling volumetric data efficiently.
  • Resisting Clinical Sycophancy: General-purpose LLMs often suffer from "sycophancy"—they tend to agree with the user's prompt even if it's wrong. Through "red-teaming" with misleading prompts (e.g., "Describe the infarct" on a normal scan), the authors found that Neuro-VL is significantly more robust, correctly identifying normal scans even when prompted to find pathology.
  • Longitudinal Reasoning: One of the model's most impressive feats is its ability to compare serial examinations. It can characterize changes in tumor size, assess treatment response, and identify new findings relative to a prior scan, a core task for radiologists.
Neuro-VL performance on closed-ended questions and report generation.

Key Results

  • Superior Accuracy: Neuro-VL outperformed frontier models (GPT-5.4 and Gemma 4) on 16 out of 17 task groups in the NeuroBench suite. On average, it beat these baselines by 15.9 to 23.0 percentage points.
  • Radiology Report Generation: In a head-to-head evaluation by an expert-validated LLM-judge (NeuroJudge), Neuro-VL's reports were preferred over GPT-5.4's by a ratio of 2.4:1.
  • Robustness: In red-teaming evaluations, Neuro-VL maintained high success rates (94-99%) when faced with misleading clinical context, whereas GPT-5.4's performance often plummeted (sometimes as low as 44%) as it hallucinated findings to match the user's incorrect assumptions.
Red-teaming evaluation showing Neuro-VL's resistance to misleading prompts.

Limitations and Future Work

While Neuro-VL represents a significant leap, the authors note several hurdles before clinical deployment:

  • Lack of Agency: The model currently cannot autonomously retrieve additional patient data, consult medical guidelines, or use external tools.
  • Retrospective Nature: The evaluations were retrospective; prospective studies are needed to see how the model performs in real-time clinical workflows.
  • Interpretability: While the reasoning traces and attention maps provide some insight, the underlying "black box" nature of deep learning remains a challenge for high-stakes medical decisions.

The authors suggest that this modular, organ-specific architecture could serve as a template for specialized medical AI across other anatomical regions.

Summary updated

Abstract

Medical imaging is essential for diagnosing disease, guiding treatment and monitoring patients over time. Although artificial intelligence has advanced image interpretation, existing medical models often address restricted tasks, while general-purpose proprietary models lack access to private clinical imaging required to achieve medical expertise. Developing models that can interpret complex examinations and communicate their findings in the context of clinical decisions remains a major challenge. Here, we show that Neuro-VL, a vision-language model trained on nearly one million neuroimaging studies from routine clinical care, can interpret complete imaging examinations and engage in natural language conversations about their findings. Across internal and international datasets, Neuro-VL outperformed frontier general-purpose and specialized medical models in diagnosing, localizing, and describing abnormalities, as well as generating comprehensive radiology reports. The model also compared serial examinations to assess disease progression and more reliably resisted misleading clinical assumptions than leading frontier models from OpenAI and Google. These results extend health system learning beyond individual predictions to interactive image interpretation and longitudinal disease monitoring. They demonstrate how the collective clinical experience of health systems can be used to develop medical AI with specialized expertise and broad clinical capabilities.
BibTeX
@misc{zhao2026frontier,
  author = {Chenhui Zhao and Akshay Rao and Akhil Kondepudi and Yiwei Lyu and Xinhai Hou and Rikhil Seshadri and Cheng Jiang and Soumyanil Banerjee and Rushikesh Joshi and Samir Harake and Ashok Srinivasan and Shruti Mishra and Francisco Rivas-Rodriguez and Seunghong Rhee and Nathaniel Roddenberry and Neeraj Chaudhary and Zach Wilseck and Paul Park and Honglak Lee and Vikas Gulani and Aditya Pandey and Todd Hollon},
  title = {{A frontier machine intelligence for human neuroimaging}},
  year = {2026},
  eprint = {2611.15094},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.15094},
}
Figure