Aligning Large Language Models with Healthcare Stakeholders: A Pathway to Trustworthy AI Integration

Kexin Ding, Mu Zhou, Akshay Chaudhari, Shaoting Zhang, Dimitris N. MetaxasarXiv 2025 · DOI · arXiv

Introduction

The rapid advancement of Large Language Models (LLMs) like GPT-4 has demonstrated immense potential across various sectors. However, in healthcare, the "alignment" problem remains a critical barrier. General-purpose LLMs are primarily trained to predict the next word in a sequence—a goal that often conflicts with the high-stakes requirements of clinical practice where accuracy, privacy, and medical ethics are paramount. This paper reviews the technical strategies necessary to bridge the gap between raw model capabilities and the specific needs of healthcare stakeholders.

The Stakeholder Landscape

Effective alignment is not a one-size-fits-all solution; it must be tailored to the diverse needs of the healthcare ecosystem.

Healthcare stakeholders including clinicians and nurses require specialized AI alignment.
  1. Clinicians: Require decision support that prioritizes reasoning and seamless workflow integration.
  2. Patients: Need accessible, empathetic, and jargon-free health education.
  3. Educators: Seek tools for curriculum development and objective student assessment.
  4. Payers (Insurers): Depend on structured data extraction from complex Electronic Health Records (EHRs) for efficient claims processing.

The Technical Pathway to Alignment

Transforming a general LLM into a clinical partner involves a multi-stage lifecycle:

  • Pre-training: Models are exposed to massive biomedical corpora, such as PubMed and PMC, to build a foundational understanding of medical terminology.
  • Intermediate Training: Techniques like Parameter-Efficient Fine-Tuning (PEFT), specifically LoRA (Low-Rank Adaptation), allow models to learn medical tasks without the prohibitive cost of full retraining.
  • Post-training: Reinforcement Learning from Human Feedback (RLHF) is used to reward models for safe and helpful outputs, ensuring they adhere to clinical guidelines.
  • Inference Stage: Advanced prompting, such as Chain-of-Thought (CoT), encourages models to generate step-by-step reasoning, mirroring a clinician’s diagnostic process and increasing transparency.
The iterative process of aligning AI models with human feedback and clinical workflows.

Key Milestones and Results

The synthesis of these alignment strategies has led to significant quantitative improvements:

  • Med-PaLM 2 reached a benchmark of 86.5% on MedQA, demonstrating the power of specialized alignment.
  • GatorTron, trained on 90 billion words of clinical text, significantly improved predictions for hospital readmissions.
  • CliniDigest showcased the utility of summarization, condensing over 10,000 words of clinical trial data into 200-word summaries with high fidelity.

Future Challenges and Directions

Despite progress, several hurdles remain. Linguistic bias persists, as most alignment efforts focus on English. Furthermore, the "black box" nature of RLHF necessitates more objective reward models. Future integration must also become multimodal, incorporating medical imaging and "omics" data. Finally, as these models evolve, regulatory frameworks must adapt to treat LLMs as "Software as a Medical Device" (SaMD) to ensure long-term safety and reliability.

Summary updated

Abstract

The wide exploration of large language models (LLMs) raises the awareness of alignment between healthcare stakeholder preferences and model outputs. This alignment becomes a crucial foundation to empower the healthcare workflow effectively, safely, and responsibly. Yet the varying behaviors of LLMs may not always match with healthcare stakeholders' knowledge, demands, and values. To enable a human-AI alignment, healthcare stakeholders will need to perform essential roles in guiding and enhancing the performance of LLMs. Human professionals must participate in the entire life cycle of adopting LLM in healthcare, including training data curation, model training, and inference. In this review, we discuss the approaches, tools, and applications of alignments between healthcare stakeholders and LLMs. We demonstrate that LLMs can better follow human values by properly enhancing healthcare knowledge integration, task understanding, and human guidance. We provide outlooks on enhancing the alignment between humans and LLMs to build trustworthy real-world healthcare applications.
BibTeX
@misc{ding2025aligning,
  author = {Kexin Ding and Mu Zhou and Akshay Chaudhari and Shaoting Zhang and Dimitris N. Metaxas},
  title = {{Aligning Large Language Models with Healthcare Stakeholders: A Pathway to Trustworthy AI Integration}},
  year = {2025},
  eprint = {2505.02848},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2505.02848},
}
Figure