Introduction
Health coaching serves as a critical intervention for the prevention and management of chronic diseases. However, the efficacy of health coaches often depends on their ability to provide immediate, evidence-based responses to complex client queries. While the rise of Large Language Models (LLMs) like ChatGPT offers potential for automation, these generative systems present significant risks in high-stakes healthcare environments. Specifically, their tendency to "hallucinate"—fabricating facts with high confidence—and their lack of transparent citations make them unsuitable for direct clinical use.
To address these challenges, this research focuses on the development of a domain-specific extractive Question Answering (QA) system. Unlike generative models, extractive systems are designed to identify and retrieve specific text spans from trusted, expert-vetted documents. This ensures that every answer provided is grounded in verifiable evidence, prioritizing reliability over conversational fluency.
The Human-AI Coaching Model
The proposed system is designed to function within a collaborative framework where the AI supports, rather than replaces, the human expert. In this model, the health coach acts as an intermediary between the client and the technology, ensuring that the advice remains personalized and empathetic while being backed by the AI's rapid retrieval capabilities.
As illustrated in the workflow above, the client poses questions to the health coach, who then queries the Q/A system. The system provides evidence-based answers, which the coach translates into actionable coaching advice for the client.
Methodology and Data-Centric Framework
The technical architecture follows a classic two-stage pipeline consisting of a Passage Retrieval component and a Passage Reader component. The retriever identifies relevant snippets from a knowledge base, while the reader extracts the precise answer from those snippets. To bridge the "domain gap" inherent in general-purpose models, the team fine-tuned specialized models: PubMedBERT for retrieval and BioBERT for reading.
A core contribution of this work is the implementation of a data-centric framework to enhance performance. Rather than focusing solely on model architecture, the researchers optimized the training data through two primary methods:
- Negative Passage Selection: Refining the model's ability to distinguish between relevant and irrelevant information during the retrieval phase.
- Question Reformulation: Utilizing techniques such as paraphrasing, word substitution, and back-translation to ensure the model is robust against the various ways a human might phrase a query.
The SleepQA Dataset
To facilitate training in a specialized field, the study introduces SleepQA, a domain-specific dataset. This resource contains 7,005 passages focused exclusively on sleep science and 5,000 expert-annotated question-answer pairs. SleepQA serves as a vital benchmark for evaluating a system's ability to handle nuances and terminology that general datasets, such as SQuAD, typically overlook.
Results and the "BM25 Paradox"
The evaluation of the system revealed a phenomenon the authors call the "BM25 Paradox." Initial results showed that a traditional keyword-based baseline (Lucene BM25) outperformed the neural models on automatic metrics like Exact Match (EM).
As shown in the table, the baseline system achieved an EM score of 0.30, while the initial fine-tuned PubMedBERT/BioBERT system trailed at 0.24. This suggests that in highly specialized domains, simple lexical matching remains a formidable baseline. However, by applying the data-centric enhancement framework—specifically word substitution and augmentation—the researchers achieved a 17% relative increase in performance, raising the EM score to 0.28.
Discussion and Human Evaluation
While the neural models did not fully surpass the BM25 baseline in automatic scores, human evaluation provided a different perspective. In real-world coaching scenarios, human experts found that the AI-supported system provided more appropriate and useful responses in nearly 40% of cases where the baseline failed. This highlights a disconnect between "lab metrics" and clinical utility; in healthcare, the relevance and context of an answer often outweigh a perfect string match.
Future Work
The researchers are currently analyzing data from a pilot Randomized Controlled Trial (RCT) involving 5,000 questions and participant interactions. This transition from laboratory evaluation to clinical outcome analysis will provide the definitive test of the system's value in improving health coaching efficiency and client outcomes. Future iterations will also explore whether more complex question reformulations can further enhance the reasoning capabilities of the neural models.
Summary updated