TL;DR
This paper introduces "Marie Bot Wellbeing," a novel conversational agent designed to improve healthcare engagement by addressing the low adoption rates of digital health tools. It achieves this by grounding its responses in a human avatar and offering multiple interaction modes, aiming to bridge the gap left by the lack of human interaction in traditional healthcare settings.
What Problem Does This Paper Address?
The increasing availability of digital health technologies hasn't translated into widespread adoption, despite patient interest. A key hypothesis for this low adoption is the absence of human interaction, a crucial element in doctor-patient relationships. This lack of personal connection can lead to reduced adherence to health advice and monitoring. The paper aims to tackle this by creating a conversational agent that emulates aspects of in-person medical consultations, specifically by using a visual human avatar and offering diverse interaction styles to make the experience more relatable and engaging.
Methods
Marie Bot Wellbeing employs a multi-modal approach to conversational AI, integrating three core interaction modes:
- MedicalQABot: This mode focuses on answering medical questions. It utilizes a retrieval-based approach, drawing answers from a curated knowledge vault of reputable sources. A key feature is "grounding," where answers are visually linked to a human avatar, allowing users to see affected body parts. The system also incorporates calibrated confidence scores to indicate uncertainty and a "graceful failure" mechanism where one mode can take over if another fails.
- SocialBot: Designed to combat social isolation and loneliness, this mode engages users in conversations about general topics and current events. It aims to be a conversational companion, providing information and showing interest in the user's input.
- Chatbot: This serves as a fallback mode, offering conversational capabilities but with less substantive information, primarily for chitchat.
The system's dialog management involves a binary classifier to predict whether a user's utterance is medical or social in nature. If a medical question cannot be answered by MedicalQABot, the system attempts to transition to SocialBot. If SocialBot lacks information, it falls back to the Chatbot.
Datasets
The paper describes the datasets used for training and evaluation:
- Medline Data: A collection of 1031 high-level medical topics, used to extract questions and answers for the MedicalQA knowledge vault and to train the medical topic classifier.
- MedDialog Dataset: A large dataset of 0.26 million dialogues between patients and doctors, used for fine-tuning language models.
- MedQuAD Dataset: Contains over 47,000 medical question-answer pairs from NIH websites, used as positive examples for the mode prediction classifier.
- News Category Dataset (Kaggle): Used as negative examples for the mode prediction classifier, excluding "Healthy Living" and "Wellness" topics.
- Reddit Data: Questions and comments from 30 subreddits, used to populate the SocialBot's knowledge vault.
Key Insights and Ideas
- Avatar Grounding for Comprehension: Visualizing medical information on a human avatar significantly enhances understanding and can help users articulate their conditions more precisely. This addresses the abstract nature of medical knowledge.
- Multi-Modal Interaction for Engagement: Offering distinct modes (MedicalQA, Social, Chat) caters to different user needs and preferences, moving beyond single-purpose health apps and increasing the likelihood of sustained engagement.
- Calibrated Confidence and Graceful Failure: The emphasis on knowing what the system doesn't know (calibrated confidence) and having fallback mechanisms (graceful failure) is crucial for building trust and reliability in a healthcare context where errors can have severe consequences.
- Bridging the Human Interaction Gap: The core idea is that emulating aspects of human interaction, even through an avatar and varied conversational styles, can mitigate the adoption barrier faced by purely functional digital health tools.
Key Results
The paper presents preliminary results and evaluation metrics for its components:
- Interaction Mode Prediction: Achieved high accuracy, with validation accuracy of 0.9970 and test accuracy of 0.9972, indicating strong performance in distinguishing between medical and social utterances.
- Medical Topic Classifier: Demonstrated good performance with train accuracy of 0.8812 and test accuracy of 0.8358.
- MedicalQA Retriever: Achieved a precision of 0.7585, recall of 0.7621, and an F-1 score of 0.7603, suggesting reasonable effectiveness in retrieving relevant medical answers.
- Chatbot Evaluation: A fine-tuned language model ChatBot achieved a significantly lower Negative Log Likelihood (NLL) of 3.2750 and Perplexity (PPL) of 26.4423 compared to a pre-trained model, indicating improved conversational quality.
Limitations and Future Work
The authors acknowledge several limitations and outline future directions:
- Data Quantity and Quality: A significant challenge is acquiring a larger and more diverse set of questions from reputable sources to improve coverage and the agent's ability to answer a wider range of queries.
- Avatar Comprehensiveness: The current avatar has a limited number of body parts. Future work aims to create a more comprehensive avatar by leveraging medical ontologies like SNOMED CT.
- User Privacy: The system is designed to respect user privacy by not asking for identifiers and storing data anonymously.
- False Information Mitigation: The risk of providing false information is addressed by relying on human-curated answers from reputable sources.
- System Capabilities Transparency: The goal is to be clear about the agent's capabilities to manage user expectations.
Future work will focus on expanding the knowledge base, enhancing the avatar's detail, and further refining the conversational capabilities of all interaction modes.
Summary updated