TL;DR
3MDBench is a new open-source multi-agent benchmark designed to evaluate Large Vision-Language Models (LVLMs) in realistic, multimodal telemedicine consultations. By simulating diverse patient temperaments and evaluating both diagnostic accuracy and communication quality, the framework reveals that combining interactive dialogue with domain-specific visual cues boosts diagnostic performance by up to 20%.
What Problem Does This Paper Address?
Telemedicine relies heavily on effective doctor-patient communication, where a patient's emotional state and communication style shape how symptoms are articulated. However, current medical AI benchmarks are largely static—relying on multiple-choice questions, text-only inputs, or rigid, non-reactive dialogue scripts. They fail to capture the dynamic, multimodal nature of real-world consultations where doctors must interpret both verbal descriptions and visual evidence (like skin rashes) while managing varying patient personalities.
Methods
The authors introduce a multi-agent simulation framework consisting of three main components:
- Patient Agent: Powered by Llama-3-8B, this agent simulates patient variability by adopting one of four classical temperaments (sanguine, choleric, melancholic, phlegmatic), which alters its responsiveness and expressiveness during the interview.
- Doctor Agent: The LVLM being evaluated (e.g., GPT-4o-mini, Qwen2-VL, MedGemma), tasked with asking information-seeking questions, interpreting symptom images, and providing a final diagnosis.
- Assessor Agent: Powered by Llava-OneVision, this agent extracts the final diagnosis and evaluates the Doctor Agent's clinical competence across 8 core criteria adapted from the standard Mini-Clinical Evaluation Exercise (Mini-CEX).
Additionally, they test a novel strategy of injecting top-3 predictions from a fine-tuned convolutional neural network (EfficientNetV2-XL) directly into the Doctor Agent's text context to assist its visual reasoning.
Datasets
The paper introduces 3MDBench, a multimodal dataset containing 2,996 test cases across 34 diagnoses spanning five medical domains (Dermatology, Dentistry, Nail disorders, Ophthalmology, and Throat/mucosae). Images were curated from 6 open-source datasets (including Kaggle, ISIC Archive, and Google SCIN) and enriched with GPT-4o-mini-generated text descriptions (basic and additional complaints) validated by medical experts to mimic real-world telemedicine distributions.
Key Insights and Ideas
- Visuals drive efficiency: Giving the Doctor Agent access to symptom images significantly reduced the number of dialogue turns required to reach a diagnosis, proving that visual data streamlines clinical workflows.
- The "Inward Reasoning" trade-off: Forcing models to generate internal rationales (Chain-of-Thought) improved diagnostic accuracy but slightly degraded their communication quality scores, suggesting a tension between thinking deeply and maintaining conversational empathy.
- Uncooperative patients degrade AI performance: Phlegmatic patients (who give brief, passive answers) led to shorter dialogues and lower clinical competence scores because the Doctor Agents failed to adapt and probe deeper.
- Neuro-symbolic synergy: General-purpose LVLMs struggle with fine-grained medical image classification. However, pairing them with a lightweight, domain-specific vision model (ConvNet) yields massive performance gains without expensive full-model fine-tuning.
Key Results
- Dialogue and reasoning help: Engaging in multimodal dialogue with internal reasoning improved GPT-4o-mini's diagnostic F1 score by 6.5% compared to non-dialogue settings.
- Massive boost from external cues: Injecting top-3 predictions from the fine-tuned EfficientNetV2-XL into the LVLM's context boosted the F1 score by up to 20%, achieving a top score of 70.3% and outperforming human-clinician baselines given full text symptom lists.
- Fine-tuning bias in medical models: Domain-specific models like MedGemma underperformed on everyday outpatient photos (e.g., skin rashes) compared to general-purpose models, likely due to a pre-training bias toward specialist clinical imaging like X-rays and CT scans.
Limitations and Future Work
- Synthetic symptoms: The additional patient symptoms were synthetically generated using GPT-4o-mini, which may introduce factual inaccuracies or distributional biases.
- Constrained diagnostic choices: The benchmark currently restricts the Doctor Agent to choosing a single diagnosis from a predefined set of 34 conditions, reducing clinical realism.
- Future directions: Future iterations aim to expand the disease categories, support free-text diagnostic outputs mapped to standard medical ontologies (ICD-10/UMLS), and explore more nuanced, data-driven patient simulation models beyond the four basic temperaments.
Summary updated