Introduction
Medical Visual Question Answering (Med-VQA) is a specialized domain where AI systems must answer clinical questions based on radiology images such as CT, MRI, and X-rays. While general-domain VQA has seen significant progress, Med-VQA has faced a primary bottleneck: the lack of high-quality, annotated data. Existing benchmarks like VQA-RAD are limited in scale and lack semantic grounding—the explicit mapping of organ boundaries and locations. Furthermore, many clinical inquiries require external knowledge (e.g., organ function) that cannot be derived from pixels alone.
To address these gaps, the authors introduce SLAKE, a large-scale bilingual dataset designed to provide a comprehensive, knowledge-enhanced training ground for medical AI.
The SLAKE Dataset
SLAKE was constructed using a multi-pronged approach to ensure clinical relevance and technical depth:
- Image Annotation: Experienced physicians manually labeled 642 images across five body regions (Head, Neck, Chest, Abdomen, and Pelvic cavity). These annotations include both masks for semantic segmentation and bounding boxes for object detection.
- Knowledge Graph (KG) Construction: To support reasoning beyond visual patterns, the authors extracted over 52,000 medical triplets (e.g.,
<Heart, Function, Promote blood flow>) from a large-scale knowledge base. - Question Generation: A template-based system, refined by medical professionals, generated 14,028 QA pairs. These are categorized into "vision-only" questions (e.g., "Is there a lung?") and "knowledge-based" questions (e.g., "Which organ system does this belong to?").
Comparative Analysis
SLAKE represents a significant advancement over the previous gold standard, VQA-RAD. As shown in the table below, SLAKE is roughly double the size in terms of images and four times larger in terms of QA pairs. Crucially, it is the first major bilingual (English/Chinese) Med-VQA dataset to include semantic labels and a structured Knowledge Graph.
Key Insights and Results
The authors validated the dataset using a Stacked Attention Network (SAN) and a specialized version, VGGseg, pre-trained on the dataset's segmentation masks. Their findings highlight several critical points:
- Semantic Grounding is Essential: Pre-training the visual backbone on segmentation tasks improved VQA accuracy from 72.73% to 75.36% on vision-only questions. This suggests that explicitly teaching a model to "see" anatomy improves its ability to reason about it.
- The Utility of Knowledge Graphs: Integrating the KG improved performance on knowledge-based questions by 2.0%, demonstrating that models can move beyond simple pattern matching toward actual medical reasoning.
- Bilingual Accessibility: By providing data in both English and Chinese, SLAKE enables the development of models that can assist clinicians and patients in diverse linguistic environments.
- Bias Mitigation: The dataset was explicitly balanced for "Yes/No" answers to prevent models from achieving high scores through simple frequency-based guessing.
Limitations and Future Work
While SLAKE is a major step forward, challenges remain. The dataset size (642 images) is still small compared to general-domain AI benchmarks. Future work will focus on scaling to thousands of patients, increasing the complexity of multi-step clinical reasoning, and integrating more specialized professional medical ontologies to deepen the Knowledge Graph's utility. SLAKE serves as a foundational benchmark for the next generation of clinically intelligent AI systems.
Summary updated