Integrating ChatGPT into Secure Hospital Networks: A Case Study on Improving Radiology Report Analysis

Kyungsu Kim, Junhyun Park, Saúl Langarica, Adham Mahmoud Alkhadrawi, Synho DoarXiv 2024 · OpenReview

TL;DR

This paper tackles the challenge of securely integrating powerful AI models like ChatGPT into hospital networks for radiology report analysis. They achieve this by developing a novel "sentence-level knowledge distillation" technique that allows a secure, on-premises model to mimic ChatGPT's performance without compromising patient data privacy, reaching over 95% accuracy in anomaly detection.

What Problem Does This Paper Address?

The healthcare industry is increasingly exploring the use of AI, particularly large language models (LLMs) like ChatGPT, to improve efficiency and accuracy in tasks like analyzing radiology reports. However, a major hurdle is the sensitive nature of patient data and strict privacy regulations (like HIPAA and GDPR). Directly using cloud-based LLMs raises significant security concerns. While some studies have explored on-premises solutions, they often require extensive human annotation for training. This paper aims to bridge this gap by creating a secure, on-premises AI system that can leverage the power of cloud-based LLMs without compromising data privacy and without relying on manual labeling.

Methods

The core innovation lies in their Sentence-level Knowledge Distillation (S-KD) approach. Instead of distilling knowledge from entire documents (Document-level KD or D-KD), they break down radiology reports into individual sentences. A cloud-based "teacher" model (like ChatGPT) is used to label these sentences as "normal," "abnormal," or "uncertain." This sentence-level labeling is then used to train a smaller, secure "student" model that runs on-premises.

A key advancement is the introduction of an "uncertain" label, which allows the model to express its confidence in predictions, making it more interpretable for clinicians. They also incorporate contrastive learning into the knowledge distillation process. This helps the model better distinguish between classes, particularly the underrepresented "normal" class, leading to improved accuracy and specificity.

Datasets

The study primarily utilizes the MIMIC-CXR dataset, a publicly available database of chest X-ray reports. They split this dataset into training and testing sets, applying their high-confidence label extraction method to filter and prepare the data. The final training set contained 11,158 documents and 172,105 sentences, while the test set comprised 2,832 documents and 40,779 sentences.

Key Insights and Ideas

  • Sentence-level is key: Breaking down reports into sentences for distillation is more effective than document-level, especially for identifying subtle anomalies.
  • Uncertainty is valuable: Explicitly modeling uncertainty in predictions significantly improves interpretability and reliability for clinicians.
  • Privacy-first AI: It's possible to achieve high performance with LLMs in healthcare without sacrificing patient data privacy by using secure, on-premises models trained via knowledge distillation.
  • Reducing annotation burden: Knowledge distillation from a powerful teacher model (like ChatGPT) can bypass the need for extensive manual labeling, a major bottleneck in medical AI development.

Key Results

The S-KD approach achieved over 95% accuracy in detecting anomalies in radiology reports when using the RadBERT-Roberta model. This significantly outperformed the traditional document-level KD approach, showing improvements across various metrics like accuracy, specificity, and sensitivity. For instance, the S-KD method on RadBERT achieved 95.06% accuracy, a substantial improvement over D-KD's 85.52%. The introduction of contrastive learning further boosted performance, particularly in correctly identifying the "normal" class.

Limitations and Future Work

The authors acknowledge several limitations:

  • Dataset validation: The study was validated on a single, albeit large, public dataset (MIMIC-CXR). Future work should include validation on diverse public datasets.
  • Ground truth: The study relies on GPT-3.5 for labeling, not radiologist-verified ground truth. Future research could integrate radiologist feedback to further refine the model.
  • Data privacy during training: While the final model is secure, the initial training data still needs to be sent to the cloud-based GPT-3.5. This necessitates robust de-identification protocols for practical hospital implementation.

Despite these limitations, the research demonstrates a promising path towards secure, efficient, and minimally supervised in-hospital AI systems for radiology report analysis, with potential applications in other medical NLP tasks.

Summary updated

Abstract

This study demonstrates the first in-hospital adaptation of a cloud-based AI, similar to ChatGPT, into a secure model for analyzing radiology reports, prioritizing patient data privacy. By employing a unique sentence-level knowledge distillation method through contrastive learning, we achieve over 95% accuracy in detecting anomalies. The model also accurately flags uncertainties in its predictions, enhancing its reliability and interpretability for physicians with certainty indicators. These advancements represent significant progress in developing secure and efficient AI tools for healthcare, suggesting a promising future for in-hospital AI applications with minimal supervision.
BibTeX
@misc{kim2024integrating,
  author = {Kyungsu Kim and Junhyun Park and Saúl Langarica and Adham Mahmoud Alkhadrawi and Synho Do},
  title = {{Integrating ChatGPT into Secure Hospital Networks: A Case Study on Improving Radiology Report Analysis}},
  year = {2024},
  eprint = {2402.09358},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2402.09358},
}
Figure