Transforming Tuberculosis Care: Optimizing Large Language Models for Enhanced Clinician-Patient Communication

Daniil Filienko, Mahek Nizar, Javier Roberti, Denise Galdamez, Haroon Jakher, Sarah Iribarren, Weichao Yuwen, Martine De Cock

TL;DR

This paper introduces an AI-powered system that uses Large Language Models (LLMs) to enhance communication between healthcare providers and Tuberculosis (TB) patients, particularly in low-resource settings. By integrating LLMs into a digital adherence technology, the system aims to improve patient engagement and treatment outcomes, while also addressing critical privacy concerns.

What Problem Does This Paper Address?

Tuberculosis (TB) remains a major global health threat, disproportionately affecting low- and middle-income countries where healthcare systems are often strained. Limited access to care and high patient-to-provider ratios hinder effective patient support, communication, and ultimately, treatment completion. While digital adherence technologies (DATs) show promise, they still require significant human involvement. This paper tackles the challenge of augmenting these DATs with LLMs to provide more efficient, personalized, and empathetic communication, thereby bridging the gap in care and improving patient outcomes. A key challenge addressed is ensuring patient privacy, especially given the stigma associated with TB, and the potential for LLMs to inadvertently leak sensitive information.

Methods

The researchers developed a multi-agent LLM system designed to act as a human-supervised treatment supporter for Spanish-speaking TB patients. Their approach involved several key techniques:

  • Prompt Engineering: They explored various prompt engineering techniques, including zero-shot, few-shot, and Retrieval Augmented Generation (RAG), to optimize LLM performance for TB-specific queries.
  • Retrieval Augmented Generation (RAG): To ensure medical accuracy, a RAG pipeline was implemented, integrating Spanish-language TB resources from reputable sources like the CDC, WHO, and Mayo Clinic.
  • Multi-Agent Classification: A two-step pipeline was developed where an initial LLM agent classifies patient queries as either empathy-seeking or information-heavy. This classification then routes the query to an empathy-optimized agent or a fact-focused RAG agent, respectively.
  • Privacy-Preserving Techniques: To mitigate privacy risks, they employed differentially private text sanitization using the UMLDP algorithm. This process replaces words in text with similar words based on Euclidean distance in the embedding space, aiming to protect Personally Identifiable Information (PII) before it's processed by the LLM.
  • Culturally and Linguistically Appropriate Responses: The models were trained using few-shot examples that incorporated local dialects and cultural nuances, specifically Argentinian Spanish, to ensure empathetic and relevant communication.

Datasets

The paper mentions using several types of data:

  • Publicly Available TB Guidelines: These were used to enhance the medical accuracy of the LLM.
  • Previous TB Trial Messages: Manually crafted dialogue samples from prior TB trials were used.
  • Manually Crafted Dialogue Samples: These were created to mimic real conversations between patients and treatment supporters.
  • Anonymized Messages from a TB Trial in Argentina: These were used for few-shot examples to ensure linguistic appropriateness and capture local dialect.
  • Patient Messages: These were used in the RAG process for text sanitization.

No new benchmark datasets were explicitly introduced, but the work leverages existing resources and real-world patient interaction data.

Key Insights and Ideas

  • LLMs as Empathetic and Informative Tools: The research highlights the potential of LLMs to simultaneously provide empathetic support and accurate medical information, a dual role crucial for vulnerable patient populations.
  • The Importance of Cultural Nuance: Simply using Spanish isn't enough; adapting to specific regional dialects and cultural contexts (like Argentinian Spanish with its "voseo" pronoun) is vital for building trust and ensuring effective communication.
  • Privacy is Paramount, but a Trade-off Exists: While privacy-preserving techniques are essential, especially for sensitive health data, there's a delicate balance to strike. Higher privacy guarantees (lower epsilon values) can sometimes lead to a decrease in model utility and accuracy.
  • Multi-Agent Systems Offer Specialization: Breaking down the task into specialized agents (e.g., for empathy vs. information) can lead to more nuanced and appropriate responses, though integrating them effectively remains a challenge.
  • RAG Can Be a Double-Edged Sword: While RAG improves medical accuracy by accessing external knowledge, it can also introduce errors or conflicting information if not carefully managed, potentially leading to user anxiety.

Key Results

The evaluation of six primary models revealed varying performance across empathy, medical accuracy, and linguistic accuracy.

  • Empathy: Models 2 and 5 performed well in generating empathetic responses, with Model 5 being a top performer. However, a common pitfall was misclassifying emotional messages and providing generic reassurance instead of addressing specific concerns. Models also struggled with exploring deeper feelings beyond close-ended questions.
  • Medical Accuracy: The inclusion of RAG generally improved medical accuracy, but some models showed a decrease in specificity, leading to potential user anxiety. In some instances, RAG provided conflicting information, resulting in incorrect or overly cautious advice.
  • Linguistic Accuracy: Models generally demonstrated good grammar and vocabulary in Argentinian Spanish. A notable limitation was inconsistency in pronoun usage (tú vs. vos vs. usted) and verb conjugation, though Model 0 showed promising use of the "voseo" form.
  • Privacy Evaluation: The privacy ablation study indicated that as privacy levels increased (lower epsilon), linguistic accuracy scores tended to be higher, suggesting that less noise leads to better utility. However, some results were unexpected, like a significant drop in medical accuracy at a specific epsilon value.

Limitations and Future Work

The authors acknowledge several limitations:

  • Specificity of Use-Case: The findings are specific to TB care in Argentinian Spanish and may not generalize directly to other medical contexts or cultures without adaptation.
  • Privacy Evaluation Metric: The privacy evaluation relied on epsilon values rather than a membership inference attack (MIA), which could provide a more robust understanding of sanitization performance.
  • LLM Guardrails: OpenAI's safety guardrails sometimes prevented certain responses, limiting the full evaluation of some models.
  • RAG Challenges: Overlapping information in the RAG database and the potential for incorrect retrieval remain challenges.
  • Gender Bias: The models exhibited inconsistency in gender-inclusive language, defaulting to masculine forms.

Future work includes:

  • Improving Medical Knowledge: Integrating knowledge graphs to capture more precise relations and mitigate imprecise medical advice.
  • Bias Mitigation: Exploring culture-specific alignment and novel prompting techniques to address biases.
  • Enhanced Privacy Evaluation: Conducting membership inference attacks to better assess privacy guarantees.
  • Developing More Robust Systems: Creating multi-agent frameworks that can independently improve each agent for specific tasks.
  • Building More Precise Tools: Developing systems capable of detecting and correcting inaccurate medical advice and privacy leakage before responses reach the user.

Summary updated

Abstract

Tuberculosis (TB) is the leading cause of death from an infectious disease globally, with the highest burden in low- and middle-income countries. In these regions, limited healthcare access and high patient-to-provider ratios impede effective patient support, communication, and treatment completion. To bridge this gap, we propose integrating a specialized Large Language Model into an efficacious digital adherence technology to augment interactive communication with treatment supporters. This AI-powered approach, operating within a human-in-the-loop framework, aims to enhance patient engagement and improve TB treatment outcomes.
BibTeX
@inproceedings{filienko2024transforming,
  author = {Daniil Filienko and Mahek Nizar and Javier Roberti and Denise Galdamez and Haroon Jakher and Sarah Iribarren and Weichao Yuwen and Martine De Cock},
  title = {{Transforming Tuberculosis Care: Optimizing Large Language Models for Enhanced Clinician-Patient Communication}},
  booktitle = {GenAI4Health @ AAAI},
  year = {2024},
  url = {https://openreview.net/forum?id=lt1ljFWInb},
}
Figure