SkinMind: Equipping Agentic RAG with Human-Inspired Graph Probing for Reliable Dermatology AI

Jing Hao, Weinan Zhou, Zhihong Lu, Yaling Shen, Yiming Xu, Yefeng Liu, Siyuan Yan, Shuaibo Li, Ming Dai, Junjun He, Zuyi Zhu, Jinjie Gu, Lijun Wu, Jian WangarXiv 2026 · arXiv

TL;DR

SkinMind is a new multimodal agentic framework that brings human-like "progressive discovery" to dermatology AI. By iteratively probing a massive, custom-built knowledge graph of over 1 million medical nodes and images, it significantly outperforms standard AI models and traditional retrieval methods in diagnostic accuracy while using context much more efficiently.

What Problem Does This Paper Address?

Dermatology is a visual and evidence-heavy field. To be reliable, an AI needs to synthesize information from textbooks, clinical guidelines, and thousands of reference images.

Current AI often uses Retrieval-Augmented Generation (RAG) to look up facts. However, standard RAG is "one-shot" and "flat": it takes a user's question, grabs a few related text chunks, and hopes for the best. This approach fails in complex medical cases because it can’t follow "multi-hop" relationships (e.g., connecting a symptom to a drug, then checking that drug's contraindications) or adapt the level of detail it retrieves based on what it learns mid-reasoning. Furthermore, most RAG systems struggle to seamlessly integrate visual evidence with structured medical facts.

Methods

The authors introduce two core components: SkinMind-KG and the SkinMind Agentic Loop.

1. SkinMind-KG (The Knowledge Base)

The team built a massive dermatology Knowledge Graph (KG) by processing 62 million tokens of text from 769 clinical guidelines and textbooks, alongside 423,000 clinical images.

  • Structured Schema: It uses 23 entity types (like Symptom, Drug, Morphology) and 25 relation types (like TREATED_BY or HAS_RISK_FACTOR).
  • Conflict Resolution: When different textbooks contradicted each other, the authors used an automated "detect-then-resolve" pipeline that used web searches and LLM reasoning to reconcile the data.

2. The Agentic Loop & Probing Tools

Instead of a single search, SkinMind uses an "orchestrator" model that acts like a doctor. It uses Slow-Fast Thinking: simple questions are answered immediately, while complex cases enter a "Slow Path" loop. In this loop, the agent uses eight specialized graph-probing tools (see Figure 2):

  • find_node: To locate a starting point in the graph.
  • probe_relations: To see what kinds of information are available for a disease.
  • search_image: To find visually similar cases.
  • read_chunks: To dive into the raw source text when a summary isn't enough.

This follows the principle of progressive disclosure: the agent only looks at high-level summaries first and only "pays" the computational cost to read deep details when it decides they are necessary.

Figure 2: Overview of SkinMind. The orchestrator adaptively explores SkinMind-KG through hierar- chical graph-probing tools to construct query-specific evidence subgraphs for response generation.

Datasets

The system was evaluated on six diverse dermatology Visual Question Answering (VQA) benchmarks:

  • SkinCon, 25Cls, Derm-Edu, DDI, and SNU134-4: Multiple-choice benchmarks covering diagnosis and clinical reasoning.
  • Morph-R: A benchmark requiring free-form, open-ended responses.
  • SkinMind-KG: The newly introduced corpus mentioned above, used as the retrieval source.

Key Insights and Ideas

  • Agency Beats Volume: Surprisingly, SkinMind uses 5.7x fewer evidence tokens than competing methods like LightRAG. Because the agent is "picky" about what it retrieves, it avoids cluttering its "brain" (context window) with irrelevant information.
  • Multimodal Anchoring: By linking images directly to disease nodes in the graph, the model can "see" a symptom in a query image, find a similar reference image, and then immediately jump to the structured medical facts associated with that disease.
  • Human-Inspired Search: The "progressive disclosure" design mimics how a human researcher works—starting with an index, checking a table of contents, and then reading specific paragraphs.

Key Results

SkinMind demonstrated a clear lead over both "naked" AI models and other state-of-the-art RAG systems:

  • Accuracy Boost: It improved average scores over direct answering by 8.70 points (using GPT-5.4-mini) and 8.89 points (using Gemini-2.5-Flash).
  • Consistency: It outperformed specialized models like DermoGPT-RL and generalist RAG methods like GraphRAG across all six benchmarks.
  • Qualitative Success: In case studies, SkinMind correctly identified rare conditions like Lichen amyloidosis by systematically ruling out look-alikes through targeted probing of morphology and symptoms.
A case study showing SkinMind's iterative reasoning process to reach a correct diagnosis.

Limitations and Future Work

While powerful, the agent isn't perfect. The authors noted "failure cases" where the model became over-confident in an early hypothesis and stopped exploring too soon, or failed to verify specific growth-rate details.

Future Work includes:

  • Specialized Tool Training: Fine-tuning models specifically to be better at "graph probing" to encourage more rigorous comparative exploration.
  • Expanded Modalities: Potentially integrating more types of clinical data beyond just text and standard photos.

Summary updated

Abstract

Reliable dermatology AI requires access to verifiable evidence distributed across heterogeneous textual and visual sources. However, conventional retrieval-augmented generation (RAG) typically relies on one-shot retrieval from flat corpora, limiting its ability to adapt retrieval granularity, explore multi-hop relations, and integrate multimodal clinical evidence. We introduce SkinMind, a multimodal agentic RAG framework that equips policy models with human-inspired graph probing for autonomous evidence seeking. SkinMind organizes retrieval as an iterative interaction between an orchestrator and a graph environment. Through eight hierarchical graph-probing tools, the orchestrator progressively identifies relevant entities, inspects node-level information, traverses relations and paths, and retrieves source-text and visual evidence. This progressive-disclosure design allows the model to adaptively determine the retrieval modality, exploration direction, and evidence granularity while constructing a query-specific evidence subgraph. To support autonomous graph probing, we construct SkinMind-KG from a 62M-token corpus of dermatology guidelines and textbooks and 423K images, guided by a dermatologist-informed schema and a conflict-resolution mechanism. The resulting graph contains over 1.0M nodes and 2.0M edges, unifying textual, relational, and visual evidence for retrieval. Across six dermatology VQA benchmarks and two policy-model settings, SkinMind improves the average score over direct answering by 8.70 and 8.89 points with GPT-5.4-mini and Gemini-2.5-Flash, respectively, while consistently outperforming competing RAG methods. Together, SkinMind and SkinMind-KG provide an effective foundation for evidence-grounded multimodal reasoning in dermatology.
BibTeX
@misc{hao2026skinmind,
  author = {Jing Hao and Weinan Zhou and Zhihong Lu and Yaling Shen and Yiming Xu and Yefeng Liu and Siyuan Yan and Shuaibo Li and Ming Dai and Junjun He and Zuyi Zhu and Jinjie Gu and Lijun Wu and Jian Wang},
  title = {{SkinMind: Equipping Agentic RAG with Human-Inspired Graph Probing for Reliable Dermatology AI}},
  year = {2026},
  eprint = {2611.08935},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.08935},
}
Figure