Medical Specialty Recommendations by an Artificial Intelligence Chatbot on a Smartphone: Development and Deployment

Hyeonhoon Lee, Jae-Hyun Kang, Jonghyeon YeoJournal of Medical Internet Research 2021 · DOI

Abstract

With the increasing need for telemedicine during the current COVID-19 pandemic, an AI chatbot with a deep learning-based NLP model that can recommend a medical specialty to patients through their smartphones would be exceedingly useful. This chatbot allows patients to identify the proper medical specialist in a rapid and contactless manner, based on their symptoms, thus potentially supporting both patients and primary care providers.

Introduction

Navigating the healthcare system is often the first hurdle for a patient. During the COVID-19 pandemic, traditional primary care—which usually acts as the "traffic controller" for specialist referrals—became harder to access. A significant technical challenge in automating this process is the "language gap." Most existing medical Natural Language Processing (NLP) models are trained on Electronic Medical Records (EMRs), which use professional terminology (e.g., "erythema"). In contrast, patients describe symptoms in "patient-friendly" natural language (e.g., "red itchy spots"). This research details the development and deployment of a smartphone-based AI chatbot designed to bridge this gap by understanding patient vernacular to provide accurate specialty recommendations.

Methods

The researchers constructed an NLP pipeline and evaluated several deep learning architectures to find the optimal balance between accuracy and deployability.

  • LSTM Variants: Four versions of Long Short-Term Memory (LSTM) networks were tested, varying the use of Attention mechanisms and comparing FastText pretrained embeddings against embeddings trained specifically on the study's dataset.
  • BERT: A Bidirectional Encoder Representations from Transformers (BERT) model was fine-tuned as a state-of-the-art benchmark.
  • Deployment: The final system was implemented using the "Alpha" open-source framework. It features a frontend chat interface and a backend cloud-based server on Google Cloud Platform (GCP), utilizing Kubernetes Engine (GKE) and Docker containers for scalability.

Data Collection and Processing

To ensure the model understood layperson language, data was sourced from HiDoc, a popular Korean web-based healthcare platform where patients pose questions to doctors.

Data processing flowchart from initial collection to final dataset.

As shown in the flowchart, the team initially collected 118,008 sentences. Following rigorous data cleansing to remove duplicates, ambiguous entries, and insufficient descriptions, a final dataset of 51,134 sentences was established, categorized into 26 medical specialties. Dermatology, Psychology, and Neurology were the most represented classes.

Evaluation and Results

The researchers identified the F1-score as the most critical metric for evaluation due to the significant class imbalance in the dataset (e.g., a high volume of dermatology queries versus fewer emergency medicine queries).

Definitions of Precision, Recall, and F1-score.

Performance Comparison

  • BERT: Achieved the highest performance with an AUC of 0.964 and an F1-score of 0.768.
  • LSTM (Custom Embeddings): Followed closely with an AUC of 0.965 and an F1-score of 0.739.

Despite BERT's superior accuracy, the LSTM model with custom embedding vectors was selected for deployment. BERT proved too computationally intensive for real-time requests on the cloud server's limited resources. The LSTM model provided a "lighter" alternative that maintained high classification performance while ensuring the chatbot remained responsive for mobile users. Statistical significance was confirmed using the Wilcoxon signed-rank test (P < .01).

Discussion and Limitations

While the chatbot demonstrates high accuracy, several limitations remain. The HiDoc dataset primarily represents users under 65, potentially introducing demographic bias. Older adults may use different language or find text-based interfaces challenging, suggesting a need for future voice-based integrations. Additionally, the model is currently limited to 26 specialties and relies on a single data source. Future work will focus on clinical validation through real-world trials to evaluate the chatbot's impact on clinical decision-making.

Summary updated

Abstract

With the increasing need for telemedicine during the current COVID-19 pandemic, an AI chatbot with a deep learning-based NLP model that can recommend a medical specialty to patients through their smartphones would be exceedingly useful. This chatbot allows patients to identify the proper medical specialist in a rapid and contactless manner, based on their symptoms, thus potentially supporting both patients and primary care providers.
BibTeX
@article{lee2021medical,
  author = {Hyeonhoon Lee and Jae-Hyun Kang and Jonghyeon Yeo},
  title = {{Medical Specialty Recommendations by an Artificial Intelligence Chatbot on a Smartphone: Development and Deployment}},
  journal = {Journal of Medical Internet Research},
  year = {2021},
  doi = {10.2196/27460},
}
Figure