TL;DR
This paper introduces a novel framework for generating realistic, high-quality doctor-patient dialogues to improve the accuracy of Electronic Medical Record (EMR) generation. By combining intent graph planning, dual-agent simulation, and a rule-reward quality control system, they significantly enhance dialogue realism and downstream EMR quality, reducing physician editing time.
What Problem Does This Paper Address?
Generating accurate Electronic Medical Records (EMRs) from doctor-patient dialogues is crucial for healthcare, but it's hampered by a lack of large-scale, realistic medical dialogue data. Existing synthetic dialogue methods often produce rigid and medically inconsistent conversations, limiting their usefulness for training EMR generation models. Real-world dialogues are complex, nuanced, and contain natural disfluencies that are hard to replicate. This paper aims to bridge this gap by creating a scalable and privacy-compliant method for synthesizing high-quality medical dialogues.
Methods
The proposed framework has three core components:
- Intent Graph Planning: This involves creating a structured graph of medical intents and their valid transitions, mined from real doctor-patient conversations. This graph guides the generation of diverse and authentic dialogue flows.
- Dual-Agent Simulation: Two large language model agents, a Doctor Agent and a Patient Agent (both based on DeepSeek-V3), iteratively generate utterances. They are conditioned on the planned intents, patient profiles, and medical knowledge, ensuring context-aware and role-specific dialogue.
- Rule-Reward Quality Control: This system ensures the generated dialogues are medically accurate and linguistically natural. It uses:
- Rule-Based Filtering: Handcrafted rules check for speaker role consistency, intent alignment, linguistic constraints (length, repetition), and medical validity (no prohibited terms, unsafe advice, or contradictions).
- Reward Model: A learned reward model scores candidate utterances based on fluency, contextual relevance, and medical accuracy, trained using a self-supervised preference-learning strategy on real dialogues. The best-scoring, rule-passing utterance is selected for each turn.
Datasets
- Training/Evaluation Data: A dataset of 617 real doctor-patient consultations from seven departments of a tertiary hospital in China. These dialogues are long (average 66 turns) and annotated with intents.
- Public Datasets (for comparison): DBMHG, IMCS-V2-MRG, and MTS-Dialog are mentioned as examples of publicly available datasets, characterized by shorter, task-focused dialogues.
- Cross-Institution Data: A held-out dataset from a second hospital was used to evaluate generalization.
Key Insights and Ideas
- The "High-Quality" Dialogue Gap: The paper clearly illustrates the significant disparity between the short, idealized dialogues in public datasets and the long, complex, and nuanced nature of real clinical conversations.
- Intent Graphs as Dialogue Blueprints: Using an intent graph provides a structured way to guide dialogue generation, ensuring logical flow and covering a wider range of clinical interactions than simple templates.
- Dual Agents for Realism: Simulating doctor and patient agents with distinct roles and prompting styles is crucial for capturing the natural back-and-forth and specific language use in clinical settings.
- Hybrid Quality Control is Key: Combining explicit medical rules with a learned reward model offers a robust way to ensure both safety and naturalness, a critical balance for medical applications. The self-supervised reward model training is particularly noteworthy for its scalability.
- Synthetic Data's Power: The results strongly suggest that high-quality synthetic dialogues can significantly augment real data, approaching the performance of real-only baselines and offering a privacy-compliant solution.
Key Results
- Dialogue Quality: The proposed framework achieved a 97.6% rule pass rate and only 0.2% illegal transitions, significantly outperforming baselines. It also produced longer dialogues (54 turns) with richer utterances (32.5 tokens/turn).
- EMR Generation Improvement: When augmenting a fixed set of 500 real dialogues with synthetic data, the framework consistently improved EMR quality. With 500 synthetic dialogues, BLEU scores increased from 42.5 (Real-only) to 45.6, KeyErr decreased from 4.2% to 3.3%, Edit Count dropped from 3.5 to 2.3, and LLM Judge scores reached 4.5.
- Privacy-Compliant Solution: The "Ours-only" setting (500 synthetic dialogues, no real data) achieved a BLEU score of 41.8, approaching the Real-only baseline, demonstrating its potential for privacy-constrained scenarios.
- Cross-Institution Generalization: The model showed robust performance on data from a different hospital, with only a slight drop in BLEU and significantly fewer edits compared to baselines.
Limitations and Future Work
- Limited Scope of Validation: While experiments covered multiple departments and one cross-institution evaluation, broader multi-center and cross-regional validation is needed to ensure robustness across diverse clinical practices.
- Reward Model Training: The reward model was trained without direct physician evaluations, which might limit its ability to capture nuanced clinical judgments. Future work plans to integrate real-time clinician feedback.
- Extending Intent Graphs: The authors suggest extending the intent graph to cover more complex scenarios like cross-department consultations and longitudinal multi-visit interactions.
- Broader Applications: While focused on EMR generation, the framework's components could be adapted for other clinical NLP tasks like coding, information extraction, and clinical QA.
Summary updated