Health-LLM: Personalized Retrieval-Augmented Disease Prediction System

Mingyu Jin, Qinkai Yu, Dong Shu, Chong Zhang, Lizhou Fan, Wenyue Hua, Suiyuan Zhu, Yanda Meng, Zhenting Wang, Mengnan Du, Yongfeng Zhang

TL;DR

This paper introduces Health-LLM, a novel system that leverages Large Language Models (LLMs) and Retrieval Augmented Generation (RAG) to predict diseases and provide personalized health advice from patient health reports. By integrating medical knowledge and employing a unique feature scoring mechanism, Health-LLM significantly outperforms existing methods in disease prediction accuracy.

What Problem Does This Paper Address?

The healthcare field is awash in data, particularly patient health reports. While AI, especially LLMs, shows immense promise for improving healthcare outcomes, effectively utilizing this complex and often inconsistently characterized data for personalized disease prediction remains a significant challenge. Traditional methods struggle to keep pace with the nuances of individual health needs. This paper aims to bridge this gap by developing a system that can intelligently extract, score, and utilize information from health reports to provide accurate disease predictions and tailored health recommendations.

Methods

Health-LLM employs a multi-stage approach:

  1. Question Generation & Feature Extraction: The system first generates questions based on the patient's health report and a medical knowledge base. This process is enhanced by Retrieval Augmented Generation (RAG), which retrieves relevant medical information to enrich the LLM's understanding. This allows the LLM to ask targeted questions to extract symptoms and other relevant features.
  2. Feature Scoring with Llama Index: The Llama Index framework is used to analyze the extracted information. The LLM assigns a confidence score (0-1) to various health aspects, indicating their perceived severity or relevance. This scoring mechanism is crucial for quantifying the importance of different features.
  3. Automated Feature Engineering (CAAFE): The system incorporates Context-Aware Automated Feature Engineering (CAAFE). This technique uses LLMs to iteratively refine and optimize features, merging or deleting them to improve prediction accuracy. This moves towards a fully automated feature engineering process.
  4. Disease Prediction with XGBoost: Finally, the processed features and their scores are fed into an XGBoost model for disease prediction. This model is trained to identify potential diseases and provide personalized health advice.

Datasets

The primary dataset used for evaluation is IMCS-21, a dataset for telemedicine consultations in China. It contains 4,116 annotated samples across 10 pediatric diseases. The authors also mention using a large number of health reports for experimentation, though specific details beyond IMCS-21 are not elaborated upon for training or evaluation.

Key Insights and Ideas

  • RAG is Crucial for Medical Context: Integrating RAG with LLMs is vital for grounding their predictions in actual medical knowledge, preventing hallucinations, and improving accuracy in a specialized domain like healthcare.
  • Scoring Features Matters: Assigning confidence scores to extracted features, rather than just using binary presence/absence, provides a richer representation for downstream prediction models.
  • Automated Feature Engineering is Key: The move towards automated feature engineering (CAAFE) streamlines the process and allows LLMs to discover semantically relevant features that might be missed by manual methods.
  • LLMs as Intelligent Intermediaries: Health-LLM demonstrates how LLMs can act as intelligent intermediaries, not just for generating text, but for querying, scoring, and synthesizing complex information from diverse sources.

Key Results

Health-LLM achieved a significant improvement in disease prediction:

  • Accuracy: 0.833
  • F1 Score: 0.762

This performance surpasses existing methods, including direct inference from large models like GPT-4 (with RAG achieving 0.68 accuracy and 0.71 F1 score) and fine-tuned LLaMA-2 models. An ablation study also highlighted the importance of both retrieval-augmented knowledge and CAAFE for achieving these results.

Limitations and Future Work

  • Dataset Specificity: While IMCS-21 is a valuable dataset, it focuses on pediatric diseases in China. The generalizability of Health-LLM to other demographics, disease types, and geographical regions may require further validation.
  • LLM Integration Complexity: Integrating LLMs from different sources and ensuring consistent performance can be complex.
  • Future Work: The authors suggest that Health-LLM serves as a proof-of-concept, highlighting the vast potential for further AI applications in healthcare, particularly in enhancing precision, efficiency, and personalization. Future work could involve expanding the knowledge base, exploring more sophisticated feature engineering techniques, and conducting larger-scale clinical trials.

Summary updated

Abstract

Recent advancements in artificial intelligence (AI), especially large language models (LLMs), have significantly advanced healthcare applications and demonstrated potentials in intelligent medical treatment. However, there are conspicuous challenges such as vast data volumes and inconsistent symptom characterization standards, preventing full integration of healthcare AI systems with individual patients' needs. To promote professional and personalized healthcare, we propose an innovative framework, Heath-LLM, which combines large-scale feature extraction and medical knowledge trade-off scoring. Compared to traditional health management applications, our system has three main advantages: (1) It integrates health reports and medical knowledge into a large model to ask relevant questions to large language model for disease prediction; (2) It leverages a retrieval augmented generation (RAG) mechanism to enhance feature extraction; (3) It incorporates a semi-automated feature updating framework that can merge and delete features to improve accuracy of disease prediction. We experiment on a large number of health reports to assess the effectiveness of Health-LLM system. The results indicate that the proposed system surpasses the existing ones and has the potential to significantly advance disease prediction and personalized health management.
BibTeX
@misc{jin2024health,
  author = {Mingyu Jin and Qinkai Yu and Dong Shu and Chong Zhang and Lizhou Fan and Wenyue Hua and Suiyuan Zhu and Yanda Meng and Zhenting Wang and Mengnan Du and Yongfeng Zhang and Yanda Meng},
  title = {{Health-LLM: Personalized Retrieval-Augmented Disease Prediction System}},
  year = {2024},
  eprint = {2402.00746},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2402.00746},
}
Figure