NeuroABench: A Multimodal Evaluation Benchmark for Neurosurgical Anatomy Identification

Ziyang Song, Zelin Zang, Xiaofan Ye, Boqiang Xu, Long Bai, Jinlin Wu, Hongliang Ren, Hongbin Liu, Jiebo Luo, Zhen Lei

Introduction

While Multimodal Large Language Models (MLLMs) have demonstrated impressive capabilities in general computer vision, their application in specialized fields like neurosurgery remains a significant challenge. Most existing surgical AI research focuses on high-level workflows or tool detection. However, the core of neurosurgery is the precise identification of fine-grained anatomical structures. Misidentifying a tiny vessel or nerve can lead to catastrophic outcomes.

Existing medical datasets often rely on static, "clean" images like MRIs or CT scans. They fail to capture the dynamic and "messy" reality of live operations where tissues are deformed, blood obscures views, and structures deviate from textbook appearances. NeuroABench addresses this gap by providing a rigorous benchmark to evaluate MLLMs on their ability to identify anatomy during actual procedures.

Examples of fine-grained anatomical identification tasks in NeuroABench, mapping surgical frames to specific brain regions.

The NeuroABench Dataset

NeuroABench is a comprehensive multimodal benchmark consisting of:

  • 9 hours of annotated surgical video.
  • 89 distinct procedures covering 32 different surgical approaches.
  • 68 core anatomical structures, such as the Internal Cerebral Veins, Tentorium, and Arachnoid Membranes.
  • 1,079 VQA pairs in a multiple-choice format, including a "None of the above" option to minimize random guessing.

Compared to existing benchmarks like OmniMedVQA, NeuroABench offers a significantly higher density of fine-grained intraoperative categories (68 vs. 18–48), making it a more demanding test of anatomical knowledge.

Methodology

The researchers employed a multi-stage pipeline to ensure high-quality ground truth data:

  1. Curation: Selection of 89 high-quality videos and 32 teaching manuals from the Neurosurgical Atlas.
  2. AI-Assisted Labeling: OpenAI-o1 analyzed manuals to create structured workflows, while Gemini-1.5-Pro processed videos to align frames with anatomical landmarks using visual and audio cues.
  3. Expert Validation: Senior neurosurgeons conducted multiple review rounds to ensure clinical accuracy.
  4. VQA Generation: Final generation of 1,079 questions designed to test identification under varying conditions.
The NeuroABench pipeline, illustrating data collection, AI-assisted video annotation, and expert-validated VQA generation.

Experimental Results and Insights

The evaluation of over 10 state-of-the-art models revealed a substantial performance gap between AI and human experts:

  • Top Performers: Gemini-2.0-Flash led with 40.87% accuracy, followed by Claude-3.5-Sonnet at 40.41%.
  • Open-Source Performance: Qwen2.5-VL-72B was the strongest open-source model, achieving 37.44%.
  • Human Comparison: While the best AI models outperformed the lowest-scoring human trainee (28%), they lagged behind the human average (46.5%) and the top trainee (56%).

A key finding is the "Textbook Trap": models perform well on structures that resemble textbook diagrams but struggle significantly when anatomy is deformed by surgical tools like retractors. This suggests a lack of "spatial common sense" in current MLLMs. Furthermore, specialized "Medical MLLMs" hit a performance ceiling (~36%), indicating that general medical training data lacks sufficient intraoperative context.

Conclusion and Future Work

NeuroABench highlights that we are still far from reliable AI surgical assistants. Future research must move beyond simple identification toward deeper spatial reasoning and the utilization of temporal context—tracking structures as they are manipulated over time. Expanding the human baseline to include senior surgeons will also help establish a more definitive performance ceiling for the field.

Summary updated

Abstract

Multimodal Large Language Models (MLLMs) have shown significant potential in surgical video understanding. With improved zero-shot performance and more effective human-machine interaction, they provide a strong foundation for advancing surgical education and assistance. However, existing research and datasets primarily focus on understanding surgical procedures and workflows, while paying limited attention to the critical role of anatomical comprehension. In clinical practice, surgeons rely heavily on precise anatomical understanding to interpret, review, and learn from surgical videos. To fill this gap, we introduce the Neurosurgical Anatomy Benchmark (NeuroABench), the first multimodal benchmark explicitly created to evaluate anatomical understanding in the neurosurgical domain. NeuroABench consists of 9 hours of annotated neurosurgical videos covering 89 distinct procedures and is developed using a novel multimodal annotation pipeline with multiple review cycles. The benchmark evaluates the identification of 68 clinical anatomical structures, providing a rigorous and standardized framework for assessing model performance. Experiments on over 10 state-of-the-art MLLMs reveal significant limitations, with the best-performing model achieving only 40.87% accuracy in anatomical identification tasks. To further evaluate the benchmark, we extract a subset of the dataset and conduct an informative test with four neurosurgical trainees. The results show that the best-performing student achieves 56% accuracy, with the lowest scores of 28% and an average score of 46.5%. While the best MLLM performs comparably to the lowest-scoring student, it still lags significantly behind the group's average performance. This comparison underscores both the progress of MLLMs in anatomical understanding and the substantial gap that remains in achieving human-level performance.
BibTeX
@misc{song2025neuroabench,
  author = {Ziyang Song and Zelin Zang and Xiaofan Ye and Boqiang Xu and Long Bai and Jinlin Wu and Hongliang Ren and Hongbin Liu and Jiebo Luo and Zhen Lei},
  title = {{NeuroABench: A Multimodal Evaluation Benchmark for Neurosurgical Anatomy Identification}},
  year = {2025},
  eprint = {2512.06921},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2512.06921},
}
Figure