3MDBench: Medical Multimodal Multi-agent Dialogue Benchmark

Ivan Sviridov, Amina Miftakhova, Artemiy Tereshchenko, Galina Zubkova, Pavel Blinov, Andrey Savchenko

TL;DR

3MDBench is a new open-source multi-agent benchmark designed to evaluate Large Vision-Language Models (LVLMs) in realistic, multimodal telemedicine consultations. By simulating diverse patient temperaments and evaluating both diagnostic accuracy and communication quality, the framework reveals that combining interactive dialogue with domain-specific visual cues boosts diagnostic performance by up to 20%.

What Problem Does This Paper Address?

Telemedicine relies heavily on effective doctor-patient communication, where a patient's emotional state and communication style shape how symptoms are articulated. However, current medical AI benchmarks are largely static—relying on multiple-choice questions, text-only inputs, or rigid, non-reactive dialogue scripts. They fail to capture the dynamic, multimodal nature of real-world consultations where doctors must interpret both verbal descriptions and visual evidence (like skin rashes) while managing varying patient personalities.

Methods

The authors introduce a multi-agent simulation framework consisting of three main components:

  1. Patient Agent: Powered by Llama-3-8B, this agent simulates patient variability by adopting one of four classical temperaments (sanguine, choleric, melancholic, phlegmatic), which alters its responsiveness and expressiveness during the interview.
  2. Doctor Agent: The LVLM being evaluated (e.g., GPT-4o-mini, Qwen2-VL, MedGemma), tasked with asking information-seeking questions, interpreting symptom images, and providing a final diagnosis.
  3. Assessor Agent: Powered by Llava-OneVision, this agent extracts the final diagnosis and evaluates the Doctor Agent's clinical competence across 8 core criteria adapted from the standard Mini-Clinical Evaluation Exercise (Mini-CEX).
Figure 2: Agents’ interaction pipeline in 3MDBench (left) and an example instance (right). The dialogue DIk begins with the Patient Agent of sanguine temperament Tk, whose first reply contains the complaint image Pk and the basic complaint BCk highlighted in blue; an additional complaint from the list ACk appears later in orange, and the final diagnosis ˆDk—identified and validated by the Assessor Agent—is shown in green. The Assessor Agent, based on DIk and ˆDk, further provides a structured evaluation of diagnostic performance as well as consultation and communication quality.

Additionally, they test a novel strategy of injecting top-3 predictions from a fine-tuned convolutional neural network (EfficientNetV2-XL) directly into the Doctor Agent's text context to assist its visual reasoning.

Datasets

The paper introduces 3MDBench, a multimodal dataset containing 2,996 test cases across 34 diagnoses spanning five medical domains (Dermatology, Dentistry, Nail disorders, Ophthalmology, and Throat/mucosae). Images were curated from 6 open-source datasets (including Kaggle, ISIC Archive, and Google SCIN) and enriched with GPT-4o-mini-generated text descriptions (basic and additional complaints) validated by medical experts to mimic real-world telemedicine distributions.

Key Insights and Ideas

  • Visuals drive efficiency: Giving the Doctor Agent access to symptom images significantly reduced the number of dialogue turns required to reach a diagnosis, proving that visual data streamlines clinical workflows.
  • The "Inward Reasoning" trade-off: Forcing models to generate internal rationales (Chain-of-Thought) improved diagnostic accuracy but slightly degraded their communication quality scores, suggesting a tension between thinking deeply and maintaining conversational empathy.
  • Uncooperative patients degrade AI performance: Phlegmatic patients (who give brief, passive answers) led to shorter dialogues and lower clinical competence scores because the Doctor Agents failed to adapt and probe deeper.
Patient responses vary significantly based on assigned temperament
  • Neuro-symbolic synergy: General-purpose LVLMs struggle with fine-grained medical image classification. However, pairing them with a lightweight, domain-specific vision model (ConvNet) yields massive performance gains without expensive full-model fine-tuning.

Key Results

  • Dialogue and reasoning help: Engaging in multimodal dialogue with internal reasoning improved GPT-4o-mini's diagnostic F1 score by 6.5% compared to non-dialogue settings.
  • Massive boost from external cues: Injecting top-3 predictions from the fine-tuned EfficientNetV2-XL into the LVLM's context boosted the F1 score by up to 20%, achieving a top score of 70.3% and outperforming human-clinician baselines given full text symptom lists.
  • Fine-tuning bias in medical models: Domain-specific models like MedGemma underperformed on everyday outpatient photos (e.g., skin rashes) compared to general-purpose models, likely due to a pre-training bias toward specialist clinical imaging like X-rays and CT scans.

Limitations and Future Work

  • Synthetic symptoms: The additional patient symptoms were synthetically generated using GPT-4o-mini, which may introduce factual inaccuracies or distributional biases.
  • Constrained diagnostic choices: The benchmark currently restricts the Doctor Agent to choosing a single diagnosis from a predefined set of 34 conditions, reducing clinical realism.
  • Future directions: Future iterations aim to expand the disease categories, support free-text diagnostic outputs mapped to standard medical ontologies (ICD-10/UMLS), and explore more nuanced, data-driven patient simulation models beyond the four basic temperaments.

Summary updated

Abstract

Though Large Vision-Language Models (LVLMs) are being actively explored in medicine, their ability to conduct complex real-world telemedicine consultations combining accurate diagnosis with professional dialogue remains underexplored. This paper presents 3MDBench (Medical Multimodal Multi-agent Dialogue Benchmark), an open-source framework for simulating and evaluating LVLM-driven telemedical consultations. 3MDBench simulates patient variability through temperament-based Patient Agent and evaluates diagnostic accuracy and dialogue quality via Assessor Agent. It includes 2996 cases across 34 diagnoses from real-world telemedicine interactions, combining textual and image-based data. The experimental study compares diagnostic strategies for widely used open and closed-source LVLMs. We demonstrate that multimodal dialogue with internal reasoning improves F1 score by 6.5% over non-dialogue settings, highlighting the importance of context-aware, information-seeking questioning. Moreover, injecting predictions from a diagnostic convolutional neural network into the LVLM's context boosts F1 by up to 20%. Source code is available at https://github.com/univanxx/3mdbench.
BibTeX
@inproceedings{sviridov2025mdbench,
  author = {Ivan Sviridov and Amina Miftakhova and Artemiy Tereshchenko and Galina Zubkova and Pavel Blinov and Andrey Savchenko},
  title = {{3MDBench: Medical Multimodal Multi-agent Dialogue Benchmark}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2025},
}
Figure