Benchmarking Large Language Models on Communicative Medical Coaching: A Dataset and a Novel System

Hengguan Huang, Songtao Wang, Hongfu Liu, Hao Wang, Ye Wang

Introduction

Effective clinical communication is a cornerstone of patient-centered healthcare, directly influencing patient satisfaction and diagnostic accuracy. However, training medical learners—students and junior doctors—in these skills often requires resource-intensive, human-led roleplay. While Natural Language Processing (NLP) has advanced diagnostic assistants and patient-facing chatbots, the professional development of clinicians through simulated practice has remained largely underserved.

Existing AI dialogue systems are typically optimized for engagement rather than pedagogical coaching. To bridge this gap, the ChatCoach framework introduces a human-AI cooperative system designed to provide real-time, structured feedback to medical learners within a safe, simulated environment. By integrating a novel Generalized Chain-of-Thought (GCoT) prompting strategy and a specialized dataset, ChatCoach enables LLMs to function as both realistic patient simulators and expert mentors.

The ChatCoach Framework

The ChatCoach system architecture is built around two primary AI agents that interact with a human learner:

  1. The Patient Agent: This agent simulates a patient based on detailed medical profiles and specific disease contexts, providing a realistic conversational partner for the doctor-in-training.
  2. The Coach Agent: This agent monitors the live dialogue between the human doctor and the Patient Agent. It identifies errors in medical terminology or clinical reasoning and provides immediate, constructive feedback.

To address the scarcity of high-quality medical coaching data, the researchers employed a Multi-agent Data Generation approach. They utilized LLM-based agents (Patient, Doctor, and Coach) to interact with medical knowledge bases, synthesizing realistic training dialogues that intentionally include common medical errors to train the Coach Agent effectively.

The ChatCoach framework (a) and the multi-agent data generation process using external resources (b).

Generalized Chain-of-Thought (GCoT)

Standard Chain-of-Thought (CoT) prompting often fails in open-ended medical scenarios because it relies on manually crafted reasoning steps that do not scale. To solve this, the authors developed Generalized Chain-of-Thought (GCoT), which automates reasoning through two steps:

  • Variable Inference: The LLM analyzes input-output samples to identify generalizable variables (e.g., [incorrect symptom], [correct medication name]).
  • Prompt Generation: These variables are used to create a structured prompt template. This ensures the model follows a consistent logic—identifying terms, cross-referencing a knowledge base, and generating feedback—without requiring human intervention for every unique case.

By abstracting specific data points into general variables, the LLM remains focused on the underlying logic of coaching rather than getting distracted by the nuances of a specific medical case.

The ChatCoach Dataset

The paper introduces the ChatCoach Dataset, the first benchmark specifically designed for communicative medical coaching. It comprises 3,500 conversations containing 13,666 utterances, built upon the MedDialog dataset and a comprehensive disease database covering 99 conditions. The dataset is structured into training, validation, and a high-quality testing set. The testing set was meticulously annotated by medical professionals to evaluate two critical tasks: the Detection of terminology misuse and the subsequent Correction of those errors.

Experimental Results

The researchers benchmarked GCoT against standard instruction prompting, Vanilla CoT, and Zero-shot CoT. The results demonstrate that GCoT significantly outperforms existing methods in both detection and correction tasks.

Quantitative performance comparison of GCoT against various training-based and prompting-based methods.

In the detection task, GCoT achieved a 34.2 BLEU-2 score, markedly higher than Zero-shot CoT (27.6). For the correction task, GCoT led with a 65.4 BERTScore, indicating high semantic alignment with expert medical advice. Furthermore, human evaluations rated GCoT higher in Clarity (3.10/4.0) and Constructiveness (2.68/4.0) compared to baseline CoT. Notably, GCoT reduced "Overly Divergent Advice" from 7.14% to 0.79%, ensuring the feedback remains practical and concise.

Discussion and Future Work

While ChatCoach represents a significant step forward, the authors noted a tendency toward "Excessive Coaching," where the AI may critique medically acceptable responses for lacking specific professional jargon. This suggests a need for more nuanced distinguishing between "technically incomplete" and "clinically wrong" statements.

Future research will focus on expanding the framework to handle complex, multi-turn diagnostic reasoning and improving the "non-linguistic" aspects of coaching, such as providing emotional encouragement and managing the overall flow of clinical dialogue.

Summary updated

Abstract

Traditional applications of natural language processing (NLP) in healthcare have predominantly focused on patient-centered services, enhancing patient interactions and care delivery, such as through medical dialogue systems. However, the potential of NLP to benefit inexperienced doctors, particularly in areas such as communicative medical coaching, remains largely unexplored. We introduce “ChatCoach”, a human-AI cooperative framework designed to assist medical learners in practicing their communication skills during patient consultations. ChatCoach differentiates itself from conventional dialogue systems by offering a simulated environment where medical learners can practice dialogues with a patient agent, while a coach agent provides immediate, structured feedback. This is facilitated by our proposed Generalized Chain-of-Thought (GCoT) approach, which fosters the generation of structured feedback and enhances the utilization of external knowledge sources. Additionally, we have developed a dataset specifically for evaluating Large Language Models (LLMs) within the ChatCoach framework on communicative medical coaching tasks. Our empirical results validate the effectiveness of ChatCoach.
BibTeX
@inproceedings{huang2024benchmarking,
  author = {Hengguan Huang and Songtao Wang and Hongfu Liu and Hao Wang and Ye Wang},
  title = {{Benchmarking Large Language Models on Communicative Medical Coaching: A Dataset and a Novel System}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2024},
  doi = {10.18653/v1/2024.findings-acl.94},
}
Figure