Aligning Medical Conversational AI through Online Reinforcement Learning with Information-Theoretic Rewards

Tanvi Verma, Yang Zhou, Rick Siow Mong Goh, Yong LiuarXiv 2026 · arXiv · OpenReview

TL;DR

This paper introduces Information Gain Fine-Tuning (IGFT), a novel method for training medical AI to conduct patient interviews without needing pre-recorded human conversations. By using online reinforcement learning with information-theoretic rewards, IGFT enables AI models to learn effective questioning strategies through simulated patient interactions, leading to more efficient and targeted information gathering.

What Problem Does This Paper Address?

Training AI for medical tasks, especially for something as nuanced as patient history-taking, is challenging. Current methods often rely on expensive, expert-annotated conversations, which are scarce and may not capture the full spectrum of patient presentations or optimal questioning techniques. Furthermore, existing approaches are often limited to learning from static datasets, preventing them from discovering more efficient or adaptive questioning strategies through exploration. This paper tackles the problem of aligning medical conversational AI to perform effective patient interviews by developing a system that can learn these skills autonomously, without human supervision, and adaptively.

Methods

The core of the proposed solution is Information Gain Fine-Tuning (IGFT), an approach that combines online reinforcement learning (RL) with information-theoretic rewards.

  • Online RL: Instead of learning from a fixed dataset, IGFT trains models by having them interact with simulated patients. The AI asks questions, receives responses, and then updates its strategy based on the outcome. This "self-play" mechanism allows the model to generate its own training data.
  • Information-Theoretic Rewards: The reward signal is designed to quantify how much a question helps the AI "learn" about the patient's condition. This is framed as an information acquisition problem, where the goal is to reduce uncertainty about the patient's medical state. The reward is calculated based on the information gain achieved by a question, measured by how many key clinical entities (symptoms, history, etc.) are revealed.
  • Hybrid Reward Function: To ensure clinical appropriateness, the information gain reward is augmented with quality assessments from GPT-4o-mini. These assessments evaluate aspects like clinical relevance, patient engagement, and specificity of the question.
  • Group Relative Policy Optimization (GRPO): This is the specific RL algorithm used for fine-tuning language models. GRPO is a ranking-based method that compares multiple generated questions and optimizes the policy towards those with higher rewards, leading to more stable learning, especially in language generation tasks.
  • LoRA (Low-Rank Adaptation): This technique is used for efficient fine-tuning of large language models, reducing computational requirements.

Datasets

  • Training: The models were fine-tuned on 350 vignettes from the Avey AI Benchmark vignette suite. These vignettes contain patient information including demographics, chief complaints, HPI, and other medical details. The Avey dataset is described as containing "concise HPIs."
  • Evaluation: The models were evaluated on held-out HPIs from both the Avey dataset (for in-domain generalization) and the MIMIC-IV dataset. MIMIC-IV data is noted for containing "longer, more elaborate HPIs," serving as a challenging test for generalization.

Key Insights and Ideas

  • Learning without Human Supervision: The most significant insight is that AI can learn effective medical history-taking skills through simulated interactions and information-theoretic rewards, drastically reducing the reliance on expensive, pre-collected human data.
  • Information Gain as a Guiding Principle: Formalizing medical history-taking as an information acquisition problem provides a principled way to guide AI learning. Questions that reduce uncertainty about the patient's condition are rewarded, leading to more targeted and efficient information gathering.
  • Balancing Information with Quality: Simply maximizing information gain isn't enough. The integration of LLM-based quality assessments (relevance, engagement, specificity) is crucial to ensure that the AI asks not just informative, but also safe and clinically appropriate questions.
  • Reasoning Capabilities Matter: The study suggests that models with stronger reasoning capabilities (like DeepSeek-R1-Distill-Qwen-7B) benefit more from information-theoretic rewards, showing larger relative improvements, indicating that reasoning helps in leveraging these rewards effectively.
  • Generalization Beyond Data Format: The models trained on concise Avey data demonstrated good generalization to the more complex and elaborate MIMIC-IV HPIs, suggesting that the learned questioning strategies are fundamental and transferable.

Key Results

  • Improved Performance: IGFT significantly improved the performance of both Llama-3.1-8B-Instruct and DeepSeek-R1-Distill-Qwen-7B models on HPI entity extraction tasks.
  • DeepSeek-R1-Distill-Qwen-7B (IGFT): Achieved an F1 score of 0.408 on Avey (a 10.9% improvement over its base model) and 0.289 on MIMIC (a 12.9% improvement).
  • Llama-3.1-8B-Instruct (IGFT): Achieved an F1 score of 0.384 on Avey and 0.336 on MIMIC.
  • Outperforming Baselines: Both IGFT-trained models outperformed OpenAI's model on MIMIC and surpassed medical domain-specific baselines like HuatuoGPT and UltraMedical, which were optimized for single-turn QA rather than multi-turn conversations.
  • Precision-Oriented Behavior: IGFT consistently improved precision while maintaining or slightly improving recall, indicating that the models learned to ask targeted questions rather than encouraging broad disclosure.
  • Generalization: The models showed strong generalization from concise Avey HPIs to more elaborate MIMIC-IV HPIs.

Limitations and Future Work

  • Computational Cost: The current method's reliance on LLM-based reward computation is computationally expensive. The authors suggest reward model distillation as a potential solution.
  • Pre-extracted Entities: The current framework requires pre-extracted entities from ground truth HPIs for reward calculation. An end-to-end approach that jointly learns entity extraction and questioning would be more practical.
  • Turn-by-Turn Optimization: The current approach optimizes each turn independently rather than planning the entire dialogue strategically. Future work could explore lookahead mechanisms for better multi-turn planning.
  • Clinical Validation: While promising, the models require extensive validation before clinical deployment.
  • Ethical Considerations: Deploying AI in healthcare necessitates careful ethical considerations, including patient informed consent, transparency, and addressing potential healthcare disparities reflected in training data.
  • Broader Applicability: The framework's principles could extend to other domains requiring strategic information gathering, such as technical support or legal interviews.

Summary updated

Abstract

We present Information Gain Fine-Tuning (IGFT), a novel approach for training medical conversational AI to conduct effective patient interviews and generate comprehensive History of Present Illness (HPI) without requiring pre-collected human conversations. IGFT combines online Group Relative Policy Optimization (GRPO) with information-theoretic rewards, enabling models to learn from self-generated conversations with simulated patients. Unlike existing approaches that rely on expensive expert-annotated conversations or static datasets, our online RL framework allows models to discover effective questioning strategies through exploration. Our key innovation is an information gain reward function that tracks which clinical entities such as symptoms, temporal patterns, and medical history, are revealed during conversation. Each question's reward is computed based on its expected information gain combined with GPT-4o-mini quality assessments across dimensions including clinical relevance, patient engagement, and specificity. This hybrid approach ensures models learn to ask targeted, clinically appropriate questions that efficiently gather diagnostic information. We fine-tune two models using LoRA: Llama-3.1-8B-Instruct and DeepSeek-R1-Distill-Qwen-7B (a reasoning-optimized model). Training exclusively on Avey data containing concise HPIs, we evaluate generalization to MIMIC data with longer, more elaborate HPIs. DeepSeek-R1-Distill-Qwen-7B (IGFT) achieves F1 scores of 0.408 on Avey (10.9% improvement over base) and 0.289 on MIMIC (12.9% improvement), while Llama-3.1-8B-Instruct (IGFT) reaches 0.384 and 0.336 respectively. Both models outperform OpenAI's model on MIMIC and surpass medical domain-specific baselines like HuatuoGPT and UltraMedical, which were optimized for single-turn medical QA rather than multi-turn conversations.
BibTeX
@misc{verma2026aligning,
  author = {Tanvi Verma and Yang Zhou and Rick Siow Mong Goh and Yong Liu},
  title = {{Aligning Medical Conversational AI through Online Reinforcement Learning with Information-Theoretic Rewards}},
  year = {2026},
  eprint = {2601.17828},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2601.17828},
}
Figure