TL;DR
MEDDxAgent is a modular, multi-agent framework designed for interactive differential diagnosis (DDx) that mimics real-world clinical workflows by iteratively gathering patient history, retrieving external medical knowledge, and refining diagnostic strategies. By moving away from the unrealistic assumption of complete upfront patient profiles, the framework achieves over a 10% accuracy improvement across various large language models (LLMs) while providing fully transparent and explainable reasoning logs.
What Problem Does This Paper Address?
Differential Diagnosis (DDx) is the iterative process by which physicians narrow down a list of potential diseases based on evolving patient data. While LLMs show immense promise in medicine, existing automated DDx systems suffer from several critical flaws:
- The "Complete Profile" Assumption: Most systems assume a complete patient profile (all symptoms and medical history) is available at the outset. In reality, doctors start with minimal information (e.g., age, gender, chief complaint) and must ask follow-up questions.
- Single-Attempt Diagnosis: Many models operate in a single-turn paradigm, failing to update hypotheses as new information emerges.
- Isolated Optimization: Existing research often focuses solely on the diagnosis strategy, ignoring the tightly coupled processes of history-taking and external knowledge retrieval.
MEDDxAgent addresses these gaps by framing DDx as an interactive, multi-turn investigative process.
Methods
The MEDDxAgent framework utilizes the ReAct (Thought, Action, Observation) paradigm to orchestrate sequential reasoning and actions. It unifies three modular components under a central coordinator:
- DDxDriver (Orchestrator): The central hub that manages the patient profile, schedules diagnostic actions, logs intermediate reasoning for traceability, and enforces stopping criteria.
- History Taking Simulator: An interactive environment driven by two LLMs—one simulating the patient (with access to the full profile) and one simulating the doctor (asking targeted questions based on conversational goals provided by the DDxDriver).
- Knowledge Retrieval Agent: Formulates targeted search queries, extracts medical concepts, and searches external databases (PubMed and Wikipedia) to synthesize up-to-date, evidence-based summaries.
- Diagnosis Strategy Agent: Generates and ranks potential diseases. It supports zero-shot Chain-of-Thought (CoT) reasoning as well as a dynamic few-shot approach, which uses embedding-based retrieval (via BioClinicalBERT or BGE) to find and present clinically similar reference cases to guide the model.
The authors experimented with two learning mechanisms: Fixed Iteration (cycling through history-taking, retrieval, and strategy sequentially) and Dynamic Iteration (allowing the DDxDriver to dynamically choose the next best component).
Datasets
To robustly evaluate the framework across diverse medical domains, the authors introduced a comprehensive DDx benchmark by standardizing three distinct datasets into a unified, interactive format:
- DDxPlus: A large-scale synthetic dataset containing 1.3 million respiratory cases across 49 pathologies.
- iCraft-MD: A dataset of 394 skin diseases, adapted from static dermatological clinical vignettes into an interactive question-asking setup.
- RareBench: A dataset expanding the diagnostic scope to 421 rare diseases, utilizing regional subsets from Europe (RAMEDIS), Canada (MME), and China (PUMCH).
Key Insights and Ideas
- Fixed Structure Outperforms Dynamic Flexibility: Surprisingly, the Fixed Iteration pipeline consistently beat Dynamic Iteration. When given total freedom, smaller or general LLMs often fell into loops of redundant history-taking rather than efficiently transitioning to knowledge retrieval or final strategy refinement.
- The Interactive Penalty and Recovery: Evaluating models on partial profiles ($n=0$ questions asked) causes a massive drop in diagnostic accuracy compared to full-profile baselines. However, interactive history-taking quickly recovers this performance, proving that diagnostic accuracy is heavily bottlenecked by information-seeking capability.
- Specialized Medical LLMs Underperform Generalists: In an interesting twist, domain-specific medical LLMs (fine-tuned on medical QA/chat) performed worse than general-purpose models like GPT-4o or Llama-3.1-70B. The authors hypothesize that medical fine-tuning often degrades the model's general instruction-following capabilities, which are vital for complex agentic orchestration.
Key Results
The authors evaluated MEDDxAgent using Ground Truth Pathology Accuracy (GTPA@k) and a novel metric, $\Delta$ Progress, which measures how effectively the system pushes the correct diagnosis up the leaderboard across successive iterations.
- Accuracy Boost: MEDDxAgent achieved over a 10% absolute improvement in GTPA@1 across both large (70B) and small (8B) LLMs compared to non-interactive baselines.
- Exceeding Full-Profile Baselines: On the DDxPlus dataset, GPT-4o's zero-shot accuracy with a full patient profile was 0.69. When deployed within the iterative MEDDxAgent framework (starting with only a partial profile), its accuracy rose to 0.86 by the third iteration.
- Scale Matters: While larger models (GPT-4o, Llama-3.1-70B) continuously improved across 3 full iterations, smaller models (Llama-3.1-8B) tended to plateau after the second iteration, particularly on complex skin and rare disease datasets.
Limitations and Future Work
- Modality Constraints: The benchmark is entirely text-based. Real-world DDx relies heavily on multimodal data like medical imaging (X-rays, dermoscopy), lab results, and genomic data.
- Language and Geography: The evaluation was restricted to English-language datasets, limiting generalizability to non-English clinical environments where medical terminology and case presentations differ.
- Latency and Cost: The multi-agent, multi-turn nature of the framework introduces significant computational overhead and latency, which must be addressed via parallelism or caching before deployment in live clinical settings.
Summary updated