Interdisciplinary Development and Fine-Tuning of CARDIO, a Large Language Model for Cardiovascular Health Education in HIV Care: Tutorial

Ryan Rullo, Ali Maatouk, Tinglin Huang, Jialin Chen, Weibo Qiu, Giselle O’Connor, Julie A. Womack, Tatiana Sadak, Christine Rodriguez, Tania de Jesús Espinosa, Pedro Carneiro, Ami Marshall, Rex Ying, S. Raquel RamosJournal of Medical Internet Research 2025 · DOI

TL;DR

This paper details the creation of CARDIO, a specialized Large Language Model (LLM) designed to provide cardiovascular health education to individuals with HIV. By combining an interdisciplinary approach with rigorous data curation and a novel fine-tuning pipeline, they significantly improved the model's accuracy, readability, and cultural relevance, laying the groundwork for AI-driven health promotion in this vulnerable population.

What Problem Does This Paper Address?

Millions of people in the US live with chronic illnesses, with a significant portion managing multiple conditions. Cardiovascular disease (CVD) is a leading cause of death, and individuals with HIV are at an even higher risk due to factors like chronic inflammation and medication side effects. Effective patient education is crucial for managing these risks, but current methods often face barriers like limited patient understanding, complex medical jargon, and time constraints. While AI shows promise in healthcare, there's a notable gap in its application for personalized, culturally sensitive cardiovascular health education specifically for people with HIV. This paper aims to bridge that gap by providing a detailed tutorial on developing such an AI tool.

Methods

The authors employed a multi-stage, interdisciplinary approach:

  1. Team Development: An interdisciplinary team of nursing, public health, and computer science experts was assembled. They established shared goals, developed a common language, and defined key metrics.
  2. Data Curation: Publicly available, evidence-based sources on CVD prevention, HIV care, and general health knowledge were scraped. This included authoritative websites, research papers, and governmental resources. Crucially, they also incorporated content from HIV forums to capture patient-expressed needs and concerns, ensuring cultural relevance.
  3. Benchmarking: Several existing LLMs were benchmarked to assess their baseline performance.
  4. Fine-Tuning: A dual-stage fine-tuning process was used:
  5. Low-Rank Adaptation (LoRA): This technique was applied to improve the model's alignment and overall performance.
  6. Reinforcement Learning (RL): Group Relative Policy Optimization was used with reward signals to incentivize responses at a fifth-grade reading level and minimize technical jargon.
  7. Evaluation: A custom rubric was developed to evaluate model answers based on accuracy, readability, and professionalism. This was supplemented by Flesch-Kincaid readability scores and a jargon score. Expert review was integral throughout the process.

Datasets

  • Training Data: A comprehensive dataset was curated by scraping publicly available, authoritative, evidence-based sources on CVD prevention, HIV care, and general health knowledge. This was supplemented by content from publicly available HIV forums.
  • Fine-tuning Dataset: Generated using GPT-4 through multiturn question-and-answer conversations, designed for an accessible reading level and avoidance of complex medical terminology.
  • Benchmarking: Several LLMs were benchmarked using this curated dataset.

Key Insights and Ideas

  • Interdisciplinary Collaboration is Key: The success of developing a complex AI tool like CARDIO hinges on seamless collaboration between domain experts (nursing, public health) and technical experts (computer science). Bridging the "language" gap between these fields was crucial.
  • Patient Voices Matter: Incorporating data from patient forums was vital for ensuring the LLM's cultural relevance and addressing the specific needs and concerns of the target population.
  • Readability and Jargon are Critical: Standard LLM evaluation metrics (like BLEU) were insufficient. The authors developed custom metrics focusing on accuracy, readability (aiming for a 5th-grade level), and professionalism (minimizing jargon) to ensure the LLM's output was truly accessible to patients.
  • Iterative Refinement is Essential: The fine-tuning process was not a one-off event but an iterative cycle of generation, expert evaluation, and refinement, leading to significant improvements.

Key Results

  • Improved Performance: After fine-tuning, the CARDIO LLM showed marked improvements:
  • Accuracy: Increased from 4.16 to 5.0 (on a 1-5 scale).
  • Readability: Increased from 4.63 to 4.98 (on a 1-5 scale).
  • Professionalism: Increased from 4.58 to 4.98 (on a 1-5 scale).
  • Kincaid Score: Decreased from 8.54 to 7.17 (indicating an easier reading level, closer to the goal of 6).
  • Jargon Score: Decreased from 4.44 to 2.92 (indicating less medical jargon).
  • Baseline LLMs Underperformed: Pre-existing LLMs showed poor n-gram agreement and had issues with readability and professionalism.
  • Custom Metrics Proved Effective: The developed custom metrics were more effective in evaluating the LLM's suitability for patient education than traditional NLP metrics.

Limitations and Future Work

  • Algorithmic Bias: Despite mitigation strategies, the risk of algorithmic bias remains a concern. Ongoing evaluation and refinement are necessary.
  • Energy Consumption: AI model development can be energy-intensive. While they used energy-efficient servers, broader considerations for large-scale deployment remain.
  • Generalizability: Ensuring the model's performance across diverse populations and settings requires further validation.
  • Custom Metrics Subjectivity: While experts achieved consistent judgments, the custom metrics (Accuracy, Readability, Professionalism) are bespoke and require broader revalidation.
  • Future Work: The authors plan to conduct patient pilot testing to evaluate the model's usability, clarity, and relevance in real-world contexts. Further refinement based on this feedback is anticipated. They also aim to explore policy implications for integrating such LLMs into clinical practice and electronic health records.

Summary updated

Abstract

We present a data-based, step-by-step tutorial for interdisciplinary development of CARDIO, a specialized LLM, for cardiovascular health education in HIV care. Through comprehensive data curation and scraping, systematic benchmarking, and a dual-stage fine-tuning pipeline, CARDIO's performance improved markedly (accuracy 5.0, readability 4.98, professionalism 4.98, Kincaid 7.17, and Jargon 2.92). Although patient pilot testing remains forthcoming, our results demonstrate that targeted data curation, rigorous benchmarking, and iterative fine-tuning have provided a robust evaluation of the model's potential. By building an LLM tailored to cardiovascular health promotion and patient education, this work lays the foundation for innovative artificial intelligence-driven strategies to manage comorbid conditions in people living with HIV.
BibTeX
@article{rullo2025interdisciplinary,
  author = {Ryan Rullo and Ali Maatouk and Tinglin Huang and Jialin Chen and Weibo Qiu and Giselle O’Connor and Julie A. Womack and Tatiana Sadak and Christine Rodriguez and Tania de Jesús Espinosa and Pedro Carneiro and Ami Marshall and Rex Ying and S. Raquel Ramos},
  title = {{Interdisciplinary Development and Fine-Tuning of CARDIO, a Large Language Model for Cardiovascular Health Education in HIV Care: Tutorial}},
  journal = {Journal of Medical Internet Research},
  year = {2025},
  doi = {10.2196/77053},
}
Figure