A Continual Pre-training Approach to Tele-Triaging Pregnant Women in Kenya

Wenbo Zhang, Hangzhi Guo, Prerna Ranganathan, Jay Patel, Sathyanath Rajasekharan, Nidhi Danayak, Manan Gupta, Amulya YadavAAAI 2023 · DOI

TL;DR

This paper introduces TRIM-AI, a novel NLP framework that automatically triages pregnant women's health concerns via SMS in Kenya. By leveraging continual pre-training on code-mixed English-Swahili messages, TRIM-AI significantly improves the accuracy and efficiency of identifying high-risk cases, reducing the burden on human helpdesk agents and potentially saving lives.

What Problem Does This Paper Address?

Maternal and neonatal mortality rates remain alarmingly high in Kenya, with a significant portion attributed to delays in seeking care. Jacaranda Health's PROMPTS system, an SMS-based tele-triage service, aims to bridge this gap by allowing pregnant women to seek medical advice. However, the sheer volume of SMS messages (~1.1 million per month) overwhelms human helpdesk agents, making it difficult to accurately assess the urgency of each case and prioritize those most in need. The challenges are compounded by the use of code-mixed language (English and Swahili), abbreviations, and informal language in the SMS messages, making automated analysis difficult.

Methods

The core of TRIM-AI is a deep learning framework built upon a state-of-the-art multilingual pre-trained language model, XLM-ROBERTa-large. The authors introduce two key innovations:

  • Continual Pre-training: To adapt XLM-ROBERTa-large to the specific domain of code-mixed SMS messages from PROMPTS, they employ a continual pre-training step using unlabeled data from the SalesForce repository. This process creates a domain-specific model, XLM-ROBERTa-JH, which is better equipped to handle the nuances of the target language and message style.
  • Multi-class Classification with Intent-to-Risk Mapping: TRIM-AI models the problem as a multi-class classification task, where each SMS message is assigned to one of 58 different "intents" (medical symptoms/conditions). These intents are then mapped to scalar risk scores by domain experts. The model outputs a probability distribution over intents, which is then used to calculate a weighted risk score, representing the emergency level of the user's condition.

Preprocessing involves Unigram segmentation for subword tokenization, which is robust to noise and code-mixing.

Datasets

  • FreshDesk Repository: This is the primary supervised dataset, containing 107,714 human-annotated code-mixed SMS messages with ground-truth intents. This dataset was used for training and evaluating TRIM-AI and baseline models. It was split into an 80% training set and a 20% hold-out test set.
  • SalesForce Repository: This repository contains 939,819 unlabeled SMS messages received since July 2021. This data was crucial for the continual pre-training step to adapt the language model to the specific domain of PROMPTS messages.

Key Insights and Ideas

  • Code-Mixing is a Major Hurdle: Standard NLP models struggle with code-mixed languages, and even powerful multilingual models like XLM-ROBERTa-large require domain adaptation to perform optimally.
  • Continual Pre-training is Crucial for Domain Adaptation: The authors demonstrate that adapting a general-purpose multilingual model to the specific domain of PROMPTS SMS messages through continual pre-training significantly boosts performance.
  • Automated Triage is Feasible and Beneficial: TRIM-AI shows that it's possible to automate the critical task of emergency level assessment for SMS messages, leading to tangible improvements in operational efficiency and user satisfaction.
  • Focus on High-Risk Cases Matters: The model's superior performance on critical (high-risk) intents is particularly important, as it directly addresses the need to prioritize care for the most vulnerable patients.

Key Results

  • Predictive Performance: TRIM-AI achieved a weighted F₁ score of 0.774, significantly outperforming baseline models, including hierarchical neural networks and other pre-trained language models. It achieved ~5% higher weighted metrics than its closest competitor, XLM-ROBERTa-base.
  • Comparison with Vertex AI: TRIM-AI outperformed Jacaranda Health's existing Vertex AI model by 15.4% in weighted F₁ score. Crucially, TRIM-AI showed significantly higher precision on critical intents like "bleeding" and "baby fever_discharge" (20-30% higher).
  • Inference Speed: The inference time for generating predictions for a single SMS message on a single Nvidia Tesla K80 GPU is approximately 28 milliseconds, meeting the low-latency requirements for real-time triage.
  • Pilot Deployment (A/B Test): In a real-world pilot, TRIM-AI led to a ~17% increase in accuracy in identifying high-risk medical conditions compared to the Vertex AI model. This resulted in a ~12% reduction in the helpdesk's workload.
  • Cost-Effectiveness: The deployment of TRIM-AI is estimated to save Jacaranda Health approximately $200 per month by reducing the need for Google Translate API usage.

Limitations and Future Work

  • Limitations: While TRIM-AI shows strong performance, the paper acknowledges that its representation learning ability for code-mixed sentences is still not perfect, even after adaptation. The authors also note that the Vertex AI model sometimes outperforms TRIM-AI on less critical intents.
  • Future Work: The authors suggest that further research could explore smaller-sized PLMs with continual pre-training to achieve similar performance to larger models, potentially reducing computational costs. The success of TRIM-AI also opens avenues for expanding its application to other areas of healthcare or similar domains dealing with noisy, code-mixed text data.

Summary updated

BibTeX
@inproceedings{zhang2023continual,
  author = {Wenbo Zhang and Hangzhi Guo and Prerna Ranganathan and Jay Patel and Sathyanath Rajasekharan and Nidhi Danayak and Manan Gupta and Amulya Yadav},
  title = {{A Continual Pre-training Approach to Tele-Triaging Pregnant Women in Kenya}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2023},
  doi = {10.1609/aaai.v37i12.26709},
}
Figure