Introduction
While Multimodal Large Language Models (MLLMs) have demonstrated impressive capabilities in general computer vision, their application in specialized fields like neurosurgery remains a significant challenge. Most existing surgical AI research focuses on high-level workflows or tool detection. However, the core of neurosurgery is the precise identification of fine-grained anatomical structures. Misidentifying a tiny vessel or nerve can lead to catastrophic outcomes.
Existing medical datasets often rely on static, "clean" images like MRIs or CT scans. They fail to capture the dynamic and "messy" reality of live operations where tissues are deformed, blood obscures views, and structures deviate from textbook appearances. NeuroABench addresses this gap by providing a rigorous benchmark to evaluate MLLMs on their ability to identify anatomy during actual procedures.
The NeuroABench Dataset
NeuroABench is a comprehensive multimodal benchmark consisting of:
- 9 hours of annotated surgical video.
- 89 distinct procedures covering 32 different surgical approaches.
- 68 core anatomical structures, such as the Internal Cerebral Veins, Tentorium, and Arachnoid Membranes.
- 1,079 VQA pairs in a multiple-choice format, including a "None of the above" option to minimize random guessing.
Compared to existing benchmarks like OmniMedVQA, NeuroABench offers a significantly higher density of fine-grained intraoperative categories (68 vs. 18–48), making it a more demanding test of anatomical knowledge.
Methodology
The researchers employed a multi-stage pipeline to ensure high-quality ground truth data:
- Curation: Selection of 89 high-quality videos and 32 teaching manuals from the Neurosurgical Atlas.
- AI-Assisted Labeling: OpenAI-o1 analyzed manuals to create structured workflows, while Gemini-1.5-Pro processed videos to align frames with anatomical landmarks using visual and audio cues.
- Expert Validation: Senior neurosurgeons conducted multiple review rounds to ensure clinical accuracy.
- VQA Generation: Final generation of 1,079 questions designed to test identification under varying conditions.
Experimental Results and Insights
The evaluation of over 10 state-of-the-art models revealed a substantial performance gap between AI and human experts:
- Top Performers: Gemini-2.0-Flash led with 40.87% accuracy, followed by Claude-3.5-Sonnet at 40.41%.
- Open-Source Performance: Qwen2.5-VL-72B was the strongest open-source model, achieving 37.44%.
- Human Comparison: While the best AI models outperformed the lowest-scoring human trainee (28%), they lagged behind the human average (46.5%) and the top trainee (56%).
A key finding is the "Textbook Trap": models perform well on structures that resemble textbook diagrams but struggle significantly when anatomy is deformed by surgical tools like retractors. This suggests a lack of "spatial common sense" in current MLLMs. Furthermore, specialized "Medical MLLMs" hit a performance ceiling (~36%), indicating that general medical training data lacks sufficient intraoperative context.
Conclusion and Future Work
NeuroABench highlights that we are still far from reliable AI surgical assistants. Future research must move beyond simple identification toward deeper spatial reasoning and the utilization of temporal context—tracking structures as they are manipulated over time. Expanding the human baseline to include senior surgeons will also help establish a more definitive performance ceiling for the field.
Summary updated