3MDBench: Medical Multimodal Multi-agent Dialogue Benchmark

Ivan Sviridov, Amina Miftakhova, Artemiy Tereshchenko, Galina Zubkova, Pavel Blinov, Andrey V. Savchenko

Abstract

Though Large Vision-Language Models (LVLMs) are being actively explored in medicine, their ability to conduct telemedicine consultations combining accurate diagnosis with professional dialogue remains underexplored. In this paper, we present 3MDBench (Medical Multimodal Multi-agent Dialogue Benchmark), an open-source framework for simulating and evaluating LVLM-driven telemedical consultations. 3MDBench simulates patient variability through four temperament-based Patient Agents and an Assessor Agent that jointly evaluate diagnostic accuracy and dialogue quality. It includes 3013 cases across 34 diagnoses drawn from real-world telemedicine interactions, combining textual and image-based data. The experimental study compares diagnostic strategies for popular LVLMs, including GPT-4o-mini, LLaVA-3.2-11B-Vision-Instruct, and Qwen2-VL-7B-Instruct. We demonstrate that multimodal dialogue with internal reasoning improves F1 score by 6.5% over non-dialogue settings, highlighting the importance of context-aware, information-seeking questioning. Moreover, injecting predictions from a diagnostic convolutional network into the LVLM's context boosts F1 by up to 20%.

Introduction

Telemedicine has transformed healthcare delivery, yet evaluating the AI systems that power these digital consultations remains a significant challenge. Most existing benchmarks rely on static, multiple-choice questions or scripted interactions that fail to capture the dynamic nature of a real-world medical dialogue. In practice, a diagnosis is the culmination of an interactive process where the clinician must navigate patient temperaments, interpret visual evidence (such as a photo of a skin lesion), and probe for missing information.

3MDBench addresses these gaps by providing a dynamic, multi-agent environment designed to simulate full-fledged medical consultations. By integrating visual data with multi-turn dialogue, the benchmark tests whether an AI "doctor" can lead an effective, information-seeking conversation to reach a correct clinical conclusion.

The 3MDBench Framework

The core of 3MDBench is a multi-agent simulation pipeline that models the interaction between a patient and a clinician, followed by a rigorous assessment phase.

The 3MDBench multi-agent simulation and assessment pipeline, showing the interaction between Doctor and Patient agents and the subsequent evaluation by an Assessor agent.

As illustrated in the framework above, the system comprises three primary roles:

  • Patient Agent: Built on Llama-3-8B, this agent simulates a patient based on specific case data, including a basic complaint ($BC_k$), additional symptoms ($AC_k$), and a visual complaint image ($P_k$).
  • Doctor Agent: This is the LVLM under evaluation. It must analyze the patient's text and images, ask clarifying questions, and eventually provide a diagnosis ($̂{D}_k$) and treatment plan.
  • Assessor Agent: A high-performing LVLM (Llava-OneVision) acts as a medical supervisor. It evaluates the Doctor Agent’s clinical competence using criteria adapted from the Mini-CEX (Mini-Clinical Evaluation Exercise) and extracts the final diagnosis for scoring against the ground truth ($D_k$).

Simulating Patient Variability

A key innovation of 3MDBench is the modeling of patient temperaments ($T_k$). To force the Doctor Agent to adapt its communication style, the Patient Agent is assigned one of four classical temperaments:

Examples of patient responses across different temperaments, illustrating how personality affects the dialogue flow.
  • Sanguine: Talkative, humorous, and expressive.
  • Choleric: Assertive, impatient, and direct.
  • Melancholic: Detailed, worried, and cautious.
  • Phlegmatic: Brief, passive, and often providing minimal information.

Experimental results show that these temperaments significantly impact the diagnostic process. For instance, Phlegmatic patients often lead to shorter conversations and lower diagnostic accuracy if the Doctor Agent fails to probe deeply enough for necessary details.

Dataset and Methodology

The 3MDBench dataset comprises 3,013 cases covering 34 diagnoses across five medical domains: Dermatology, Dentistry, Nail disorders, Ophthalmology, and Throat/mucosae. The data was compiled from six open-source medical image repositories and enriched with realistic textual symptoms validated by human physicians.

Beyond standard dialogue, the researchers explored the impact of "External Cues." This involves feeding the top-3 predictions from a specialized medical image classifier (EfficientNetV2-XL) directly into the LVLM’s text prompt. This technique tests the synergy between domain-specific "expert eyes" and the general-purpose reasoning of the LVLM.

Key Results and Insights

  • Dialogue as a Diagnostic Tool: Engaging in multi-turn conversation with internal reasoning improved F1 scores by 6.5 percentage points compared to "one-shot" image-based diagnosis. This confirms that interactive questioning is vital for resolving clinical ambiguity.
  • The Power of Specialized Cues: The most significant performance boost came from External Cues. Injecting domain-specific vision model predictions into the context increased the F1 score by nearly 20%, suggesting that general-purpose models benefit greatly from specialized visual pre-processing.
  • Top Performers: GPT-4o-mini achieved the highest overall performance with a 70.3% F1 score when utilizing the full suite of dialogue, images, and external cues. While open-source models like MedGemma-27B showed strong humanistic care, they still lag behind closed-source models in fine-grained diagnostic accuracy.

Conclusion

3MDBench provides a robust, open-source framework for evaluating the next generation of telemedical AI. While current LVLMs show promise in maintaining professional and empathetic dialogue, the benchmark highlights the need for improved fine-grained visual recognition and more proactive information-seeking strategies. Future work will expand the diagnostic breadth and incorporate more complex cognitive biases into the Patient Agents to further bridge the gap between simulation and real-world clinical practice.

Summary updated

Abstract

Though Large Vision-Language Models (LVLMs) are being actively explored in medicine, their ability to conduct telemedicine consultations combining accurate diagnosis with professional dialogue remains underexplored. In this paper, we present 3MDBench (Medical Multimodal Multi-agent Dialogue Benchmark), an open-source framework for simulating and evaluating LVLM-driven telemedical consultations. 3MDBench simulates patient variability through four temperament-based Patient Agents and an Assessor Agent that jointly evaluate diagnostic accuracy and dialogue quality. It includes 3013 cases across 34 diagnoses drawn from real-world telemedicine interactions, combining textual and image-based data. The experimental study compares diagnostic strategies for popular LVLMs, including GPT-4o-mini, LLaVA-3.2-11B-Vision-Instruct, and Qwen2-VL-7B-Instruct. We demonstrate that multimodal dialogue with internal reasoning improves F1 score by 6.5% over non-dialogue settings, highlighting the importance of context-aware, information-seeking questioning. Moreover, injecting predictions from a diagnostic convolutional network into the LVLM's context boosts F1 by up to 20%. Source code is available at https://anonymous.4open.science/r/3mdbench_acl-0511.
BibTeX
@inproceedings{sviridov2025mdbench,
  author = {Ivan Sviridov and Amina Miftakhova and Artemiy Tereshchenko and Galina Zubkova and Pavel Blinov and Andrey V. Savchenko},
  title = {{3MDBench: Medical Multimodal Multi-agent Dialogue Benchmark}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2025},
}
Figure