MEDDxAgent: A Unified Modular Agent Framework for Explainable Automatic Differential Diagnosis

Daniel Rose, Chia-Chien Hung, Marco Lepri, Israa Alqassem, Kiril Gashteovski, Carolin Lawrence

TL;DR

MEDDxAgent is a modular, multi-agent framework designed for interactive differential diagnosis (DDx) that mimics real-world clinical workflows by iteratively gathering patient history, retrieving external medical knowledge, and refining diagnostic strategies. By moving away from the unrealistic assumption of complete upfront patient profiles, the framework achieves over a 10% accuracy improvement across various large language models (LLMs) while providing fully transparent and explainable reasoning logs.

What Problem Does This Paper Address?

Differential Diagnosis (DDx) is the iterative process by which physicians narrow down a list of potential diseases based on evolving patient data. While LLMs show immense promise in medicine, existing automated DDx systems suffer from several critical flaws:

  1. The "Complete Profile" Assumption: Most systems assume a complete patient profile (all symptoms and medical history) is available at the outset. In reality, doctors start with minimal information (e.g., age, gender, chief complaint) and must ask follow-up questions.
  2. Single-Attempt Diagnosis: Many models operate in a single-turn paradigm, failing to update hypotheses as new information emerges.
  3. Isolated Optimization: Existing research often focuses solely on the diagnosis strategy, ignoring the tightly coupled processes of history-taking and external knowledge retrieval.

MEDDxAgent addresses these gaps by framing DDx as an interactive, multi-turn investigative process.

Methods

The MEDDxAgent framework utilizes the ReAct (Thought, Action, Observation) paradigm to orchestrate sequential reasoning and actions. It unifies three modular components under a central coordinator:

Figure 1: MEDDxAgent facilitates differential diagno- sis by iteratively narrowing down a patient’s possible disease. DDxDriver acts as the central orchestrator. It receives an interactive environment via a simulator (His- tory Taking) and can access two agents (Knowledge Retrieval, Diagnosis Strategy).
  • DDxDriver (Orchestrator): The central hub that manages the patient profile, schedules diagnostic actions, logs intermediate reasoning for traceability, and enforces stopping criteria.
  • History Taking Simulator: An interactive environment driven by two LLMs—one simulating the patient (with access to the full profile) and one simulating the doctor (asking targeted questions based on conversational goals provided by the DDxDriver).
  • Knowledge Retrieval Agent: Formulates targeted search queries, extracts medical concepts, and searches external databases (PubMed and Wikipedia) to synthesize up-to-date, evidence-based summaries.
  • Diagnosis Strategy Agent: Generates and ranks potential diseases. It supports zero-shot Chain-of-Thought (CoT) reasoning as well as a dynamic few-shot approach, which uses embedding-based retrieval (via BioClinicalBERT or BGE) to find and present clinically similar reference cases to guide the model.

The authors experimented with two learning mechanisms: Fixed Iteration (cycling through history-taking, retrieval, and strategy sequentially) and Dynamic Iteration (allowing the DDxDriver to dynamically choose the next best component).

Figure 2: The architecture of the MEDDxAgent framework. MEDDxAgent unifies a central orchestrator (DDx- Driver), a simulator (History Taking) and two agents (Knowledge Retrieval, Diagnosis Strategy). The framework follows the ReAct (Yao et al., 2023) paradigm (thought, action, observation), enabling sequential reasoning and action steps with transparent logging of all interactions through the iterative learning process.

Datasets

To robustly evaluate the framework across diverse medical domains, the authors introduced a comprehensive DDx benchmark by standardizing three distinct datasets into a unified, interactive format:

  • DDxPlus: A large-scale synthetic dataset containing 1.3 million respiratory cases across 49 pathologies.
  • iCraft-MD: A dataset of 394 skin diseases, adapted from static dermatological clinical vignettes into an interactive question-asking setup.
  • RareBench: A dataset expanding the diagnostic scope to 421 rare diseases, utilizing regional subsets from Europe (RAMEDIS), Canada (MME), and China (PUMCH).

Key Insights and Ideas

  • Fixed Structure Outperforms Dynamic Flexibility: Surprisingly, the Fixed Iteration pipeline consistently beat Dynamic Iteration. When given total freedom, smaller or general LLMs often fell into loops of redundant history-taking rather than efficiently transitioning to knowledge retrieval or final strategy refinement.
  • The Interactive Penalty and Recovery: Evaluating models on partial profiles ($n=0$ questions asked) causes a massive drop in diagnostic accuracy compared to full-profile baselines. However, interactive history-taking quickly recovers this performance, proving that diagnostic accuracy is heavily bottlenecked by information-seeking capability.
  • Specialized Medical LLMs Underperform Generalists: In an interesting twist, domain-specific medical LLMs (fine-tuned on medical QA/chat) performed worse than general-purpose models like GPT-4o or Llama-3.1-70B. The authors hypothesize that medical fine-tuning often degrades the model's general instruction-following capabilities, which are vital for complex agentic orchestration.

Key Results

The authors evaluated MEDDxAgent using Ground Truth Pathology Accuracy (GTPA@k) and a novel metric, $\Delta$ Progress, which measures how effectively the system pushes the correct diagnosis up the leaderboard across successive iterations.

Performance across datasets and models
  • Accuracy Boost: MEDDxAgent achieved over a 10% absolute improvement in GTPA@1 across both large (70B) and small (8B) LLMs compared to non-interactive baselines.
  • Exceeding Full-Profile Baselines: On the DDxPlus dataset, GPT-4o's zero-shot accuracy with a full patient profile was 0.69. When deployed within the iterative MEDDxAgent framework (starting with only a partial profile), its accuracy rose to 0.86 by the third iteration.
  • Scale Matters: While larger models (GPT-4o, Llama-3.1-70B) continuously improved across 3 full iterations, smaller models (Llama-3.1-8B) tended to plateau after the second iteration, particularly on complex skin and rare disease datasets.

Limitations and Future Work

  • Modality Constraints: The benchmark is entirely text-based. Real-world DDx relies heavily on multimodal data like medical imaging (X-rays, dermoscopy), lab results, and genomic data.
  • Language and Geography: The evaluation was restricted to English-language datasets, limiting generalizability to non-English clinical environments where medical terminology and case presentations differ.
  • Latency and Cost: The multi-agent, multi-turn nature of the framework introduces significant computational overhead and latency, which must be addressed via parallelism or caching before deployment in live clinical settings.

Summary updated

Abstract

Differential Diagnosis (DDx) is a fundamental yet complex aspect of clinical decision-making, in which physicians iteratively refine a ranked list of possible diseases based on symptoms, antecedents, and medical knowledge. While recent advances in large language models (LLMs) have shown promise in supporting DDx, existing approaches face key limitations, including single-dataset evaluations, isolated optimization of components, unrealistic assumptions about complete patient profiles, and single-attempt diagnosis. We introduce a Modular Explainable DDx Agent (MEDDxAgent) framework designed for interactive DDx, where diagnostic reasoning evolves through iterative learning, rather than assuming a complete patient profile is accessible. MEDDxAgent integrates three modular components: (1) an orchestrator (DDxDriver), (2) a history taking simulator, and (3) two specialized agents for knowledge retrieval and diagnosis strategy. To ensure robust evaluation, we introduce a comprehensive DDx benchmark covering respiratory, skin, and rare diseases. We analyze single-turn diagnostic approaches and demonstrate the importance of iterative refinement when patient profiles are not available at the outset. Our broad evaluation demonstrates that MEDDxAgent achieves over 10% accuracy improvements in interactive DDx across both large and small LLMs, while offering critical explainability into its diagnostic reasoning process.
BibTeX
@inproceedings{rose2025meddxagent,
  author = {Daniel Rose and Chia-Chien Hung and Marco Lepri and Israa Alqassem and Kiril Gashteovski and Carolin Lawrence},
  title = {{MEDDxAgent: A Unified Modular Agent Framework for Explainable Automatic Differential Diagnosis}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2025},
}
Figure