TL;DR
This paper introduces an AI-powered "co-piloting" system designed to assist breastfeeding and maternity experts in responding to user queries. By integrating various NLP tasks into a pipeline, the system aims to reduce expert workload and improve response quality, ultimately enhancing the support provided to mothers.
What Problem Does This Paper Address?
Breastfeeding and maternity experts are a scarce resource, and engaging with mothers on such sensitive topics is time-consuming. Experts often spend significant time sifting through lengthy user messages and past conversations to gather crucial information. They also rely on informal methods like maintaining lists of frequently used sentences and helpful links. This manual process can be inefficient and lead to delays in providing support. The paper addresses the need for a system that can efficiently process user queries, extract relevant information, and assist experts in crafting timely and accurate responses, without compromising the trust and security essential in this domain.
Methods
The authors developed an AI-in-the-loop approach, meaning the AI assists experts rather than replacing them. The system is built as a pipeline of NLP/AI tasks, implemented as REST APIs using the Haystack Python API. Key tasks include:
- Conversation Detection: Identifies if a message starts a new conversation or continues an existing one.
- Text Summarization: Condenses complex user messages to highlight key information, using both abstractive and extractive methods.
- Intent Detection: Characterizes the user's query by identifying its main and secondary intents (e.g., compatibility issues, pain, general queries).
- Information Extraction: Extracts specific details like baby's age, mother's pregnancy status, or symptoms related to pain.
- Machine Translation: Translates queries to Spanish and generated responses back to the source language.
- Blog URL Recommendation: Suggests relevant blog posts based on query similarity.
- Image-to-Text Extraction: Extracts data from images, specifically for baby growth charts.
- Retrieval Augmented Generation (RAG): Generates responses by retrieving relevant Q&A pairs and using them to inform the LLM.
The system is designed to be modular, allowing for incremental deployment and testing of functionalities.
Datasets
The authors utilized a dataset of 100 randomly picked Q&A pairs for evaluating the answer generation task. This dataset was specifically curated to include "hard queries," which were either difficult for experts to classify or identified as "noisy" by the Cleanlab package. For intent detection, they manually annotated a subset of user messages from this dataset. The blog recommendation task uses a data store of 40,000 blog post chunks.
Key Insights and Ideas
- AI as a Co-Pilot, Not a Replacement: The core philosophy is to empower experts, not automate them out of the loop. This is crucial for maintaining trust and ensuring sensitive information is handled appropriately.
- Pipeline Approach for Complexity: Recognizing that not all queries are equal, a multi-task pipeline is essential to handle diverse user inputs effectively, from simple questions to complex, multi-faceted issues.
- The Importance of Context: Understanding the entire conversation history and extracting key information (like symptoms, history, and user sentiment) is vital for generating truly helpful and empathetic responses.
- Iterative Development is Key: The project started with a basic RAG approach and evolved based on expert feedback and monitoring, highlighting the need for continuous improvement and adaptation.
Key Results
- Improved Response Quality with RAG: Preliminary evaluation showed that RAG-generated responses had higher semantic similarity to expert responses compared to vanilla generation (e.g., 82.5% vs. 72.4% with mpnet2).
- Increased Conversion Rate: The deployment of the RAG system saw a gradual increase in the conversion rate (percentage of expert messages drafted from AI-generated responses), rising from 8% in February to 21% in May, indicating growing expert adoption.
- Intent Detection Accuracy: While only 37% of intent predictions fully matched ground truth, 91% had some level of match (partial or total), suggesting the system can identify relevant intents even if not perfectly.
- Summarization Benefits: Summarized inputs led to generated answers that were semantically similar to expert answers and closer in length to expert responses compared to using full inputs.
Limitations and Future Work
- Cost of LLMs: Proprietary LLMs are expensive, prompting consideration for developing in-house models for certain tasks in the future.
- Trust and Veracity: Ensuring experts can always verify information is paramount. The system prioritizes extractive understanding to maintain this trust.
- Handling Complex Queries: For highly complex queries involving personal history, retrieval might be poor, requiring more sophisticated methods to help experts understand the query first.
- UI and Feedback: The current UI needs further development to be fully usable by experts, including a robust feedback mechanism.
- Accuracy Improvements: Ongoing work is needed to improve accuracy in tasks like intent detection and information extraction, especially for nuanced medical scenarios.
- Expanding Functionality: Future work includes adding more templates for intent extraction, exploring hybrid retrieval, fine-tuning LLMs, and implementing fact-checking. They also plan to add message concatenation and rule-based question generation for specific intents.
Summary updated