TL;DR
This paper introduces MILD Bot, a novel multidisciplinary question-answering system designed to support childhood cancer survivors in South Korea. By combining an intent classifier with a semantic textual similarity model, MILD Bot aims to provide accurate, empathetic, and tailored information, addressing a critical gap in specialized support for this vulnerable population.
What Problem Does This Paper Address?
Childhood cancer survival rates have significantly improved, leading to a growing population of survivors who face complex psychosocial and social needs. In South Korea, there's a notable shortage of experts specifically trained to address these unique challenges. This lack of specialized support leaves survivors with limited access to reliable information and can exacerbate difficulties in adjusting to life after cancer, especially given the stigma associated with disclosing medical history. The COVID-19 pandemic has highlighted the potential of digital health technologies, including conversational agents, but their application for childhood cancer survivors remains limited. This paper tackles this unmet need by developing a dedicated AI-powered support system.
Methods
The MILD Bot is built upon a dual-component architecture:
- Intent Classifier: This component analyzes a survivor's query to determine the underlying intent and identify the most appropriate expert (pediatric oncologist, social worker, or psychological/mental health professional) to address it. This is achieved through a few-shot learning approach using GPT-4 Turbo on a manually categorized dataset.
- Semantic Textual Similarity (STS) Model: Once the intent is identified, the STS model matches the user's query to similar questions within a predefined database. This database contains questions and answers from both expert sources and peer survivors. The goal is to retrieve the most relevant and helpful response.
The system also incorporates domain-adaptive training and vocabulary expansion to improve its understanding of the specific nuances of childhood cancer survivorship.
Datasets
The MILD Bot utilizes a combination of existing and newly created datasets:
- Existing Korean STS Datasets:
- Korean STS (Kakao Brain): 8,628 sentence pairs.
- KLUE STS: 12,187 sentence pairs from practical contexts.
- Question pair v2: 6,888 sentence pairs with binary similarity labels.
- ParaKQC: 100 sets of 10 sentence pairs focusing on parallel topics and intentions.
- New Datasets:
- Childhood Cancer Survivor Domain Corpus: A 2.2GB corpus comprising 860 academic articles, 2 publications, and 18,565 news articles, augmented with keywords recommended by a psychosocial expert.
- Childhood Cancer Survivor STS Dataset: A domain-specific STS dataset created by leveraging the inference QA dataset and applying data augmentation techniques (e.g., using Natural Language Inference pairs) and fine-tuning a cross-encoder model. This resulted in 31,456 sentence pairs.
- Inference QA Dataset: This serves as the bot's knowledge base, comprising:
- Expert QA: 3,500 question-answer pairs generated by GPT-4 Turbo, acting as various experts.
- Peer Survivor QA: 1,283 question-answer pairs extracted from academic articles and YouTube videos, representing real survivor experiences.
Key Insights and Ideas
- The Power of Domain-Specific Adaptation: The study strongly emphasizes that general-purpose language models and datasets are insufficient for specialized domains like childhood cancer survivorship. Domain-adaptive training and the creation of domain-specific datasets (like the CCS STS dataset) are crucial for achieving high performance.
- Bridging the Expert Gap: Recognizing the shortage of human experts, the MILD Bot leverages AI to provide accessible, multidisciplinary support, acting as a stigma-free platform for information and emotional guidance.
- The Value of Peer Survivor Input: Including responses from actual peer survivors is a key differentiator, offering empathetic and relatable insights that expert-generated answers might miss. This acknowledges the unique lived experiences of survivors.
- Intent Classification for Tailored Responses: The intent classifier is vital for ensuring that survivors receive information from the most relevant expert perspective, preventing information overload and providing more targeted support.
Key Results
- Intent Classifier Performance: The KLUE BERT model achieved the best performance for the intent classifier, with an Exact-Match Ratio (EMR) of 0.76 and F1-scores of 0.90 for precision and recall.
- Domain-Adaptive Training Effectiveness: Fine-tuning with the CCS STS dataset significantly improved the STS model's performance, with the best setting achieving a Cosine-pearson score of 0.93.
- Human Evaluation: In a human evaluation with 14 childhood cancer survivors, the MILD Bot's responses were rated highly on overall quality, factuality, completeness, ease of understanding, and empathy. Participants expressed a desire to reuse the bot. Experts also consistently preferred GPT-4 Turbo's responses over those from social media platforms.
Limitations and Future Work
- Dataset Size: The authors acknowledge that the CCS Domain Corpus, while substantial, is still relatively small compared to other domain-specific datasets. Future work includes augmenting the dataset with translated English data.
- Medical Expertise: While the bot covers multiple expert areas, the need for deeper medical expertise in addressing specific questions was noted. The plan is to expand the dataset with more cancer-related information, guided by pediatric oncologists.
- Retrieval-Only Approach: The current MILD Bot uses a retrieval-only approach for safety reasons, given the sensitive nature of the user population. Future work will explore Retrieval-Augmented Generation (RAG) systems to compare their performance and potential benefits against the current retrieval-only method.
- User Age Restriction: For the human evaluation, survivors under 20 were excluded to mitigate potential risks, indicating a need for careful consideration of age-appropriateness in future deployments.
Summary updated