Dr. Copilot: A Multi-Agent Prompt Optimized Assistant for Improving Patient-Doctor Communication in Romanian

Andrei Niculae, Adrian Cosma, Cosmin Dumitrache, Emilian RadoiEMNLP 2025

TL;DR

This paper introduces Dr.Copilot, a novel multi-agent LLM system designed to improve the quality of patient-doctor communication in Romanian telemedicine. By providing targeted feedback on response presentation rather than medical content, Dr.Copilot has demonstrated significant improvements in patient satisfaction and doctor response quality in a real-world deployment.

What Problem Does This Paper Address?

Telemedicine has seen a surge in adoption, but a critical challenge remains: ensuring effective communication between doctors and patients. While doctors prioritize medical accuracy, patients often judge the quality of advice based on how it's communicated. This can lead to negative reviews and decreased patient satisfaction, even when the medical advice is sound. Existing guidelines for improving communication are often too general and easily overlooked. Furthermore, the Romanian language is under-represented in medical NLP research, making it difficult to leverage existing LLM tools. This paper addresses the need for a system that can provide specific, actionable feedback to Romanian-speaking doctors to enhance their communication skills within telemedicine platforms, without interfering with the medical content itself.

Methods

The core of Dr.Copilot is a multi-agent system built using large language models (LLMs). It comprises three key agents:

  • Scoring Agent: This agent evaluates doctor responses based on 17 interpretable quality measures, such as empathy, clarity, and completeness of problem addressing. It uses automatically optimized prompts via DSPy to ensure structured and consistent scoring.
  • Recommender Agent: This agent takes the scores from the Scoring Agent and generates tailored, constructive suggestions for improving the doctor's response. The recommendations are framed as suggestions, not directives, to maintain doctor autonomy.
  • Reconciliation Agent: Used for evaluation, this agent incorporates the recommender's suggestions into the original response and then re-scores it. This allows for measuring the effectiveness of the recommendations.

A key methodological innovation is the use of DSPy, a framework for automatic prompt optimization. This approach allows for efficient training of specialized agents with limited labeled data, crucial for low-resource languages like Romanian. The system leverages open-weight models to ensure privacy and on-premise deployment.

Datasets

The study utilized anonymized real-world data from a Romanian telemedicine platform. This dataset consisted of approximately 100,000 text consultations. For training and evaluation, a subset of 100 question-response pairs, annotated with both positive and negative user reviews, was selected. These pairs were annotated by two company employees with domain knowledge.

Key Insights and Ideas

  • Focus on Presentation, Not Content: The paper highlights that improving the presentation of medical advice can significantly boost patient satisfaction, even more so than minor medical inaccuracies.
  • The Power of Specific Feedback: Generic communication guidelines are often ignored. Dr.Copilot's success stems from providing specific, case-by-case feedback tailored to each interaction.
  • Multi-Agent Systems for Nuance: Breaking down the task into scoring, recommending, and reconciling allows for a more nuanced and effective approach to improving communication.
  • DSPy for Low-Resource Languages: Automatic prompt optimization with DSPy proves effective for developing LLM applications in languages with limited available data and resources.
  • Real-World Impact: The study demonstrates that LLM-based tools can be successfully deployed in sensitive domains like healthcare, leading to measurable improvements in user experience.

Key Results

  • Measurable Improvement in Response Quality: In a live deployment with 41 doctors, Dr.Copilot's suggestions led to a 10% overall relative improvement across metrics when incorporated by doctors.
  • Significant Increase in Patient Satisfaction: Responses that incorporated Dr.Copilot's suggestions received positive reviews from patients in 47.5% of cases, compared to 25.83% for responses that did not. This represents an 83.89% increase in the like-to-response ratio.
  • Effective Prompt Optimization: The MedGemma-27B model with the SIMBA prompt optimizer achieved the best overall performance across metrics in the scoring agent evaluation.
  • Low Latency: The system achieved an average response evaluation time of five seconds, which is crucial for user adoption.

Limitations and Future Work

The authors acknowledge several limitations:

  • Limited Doctor Evaluation: The study involved 41 doctors, and they expect results to improve with larger-scale evaluations.
  • Romanian Language Benchmarking: While Gemma models show multilingual capabilities, their specific performance on Romanian medical conversations requires further benchmarking.
  • Potential for Misinterpretation: While Dr.Copilot doesn't offer medical advice, there's a risk of patients losing trust if they suspect AI involvement, or if doctors rely too heavily on suggestions.
  • Jailbreaking Risks: As with any LLM system processing user input, Dr.Copilot could be a target for jailbreaking attacks, which might be mitigated by guardrails and red-teaming.

Future work could involve expanding the evaluation to more doctors and exploring more comprehensive benchmarks for Romanian medical LLMs. The authors also suggest investigating methods to further mitigate potential risks associated with AI in healthcare communication.

Summary updated

BibTeX
@inproceedings{niculae2025dr,
  author = {Andrei Niculae and Adrian Cosma and Cosmin Dumitrache and Emilian Radoi},
  title = {{Dr. Copilot: A Multi-Agent Prompt Optimized Assistant for Improving Patient-Doctor Communication in Romanian}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2025},
}
Figure