TL;DR
This paper introduces SMARTMiner, a framework for automatically extracting and evaluating SMART (Specific, Measurable, Attainable, Relevant, Time-bound) goals from unstructured health coaching notes. By developing a novel dataset (SMARTSpan) and a two-stage model, they enable more efficient and scalable goal tracking, bridging the gap between human-led coaching sessions.
What Problem Does This Paper Address?
Health coaching (HC) is a crucial intervention for promoting healthy behaviors and managing chronic diseases. A cornerstone of HC is setting actionable, short-term goals that drive long-term change. However, manually extracting and evaluating these goals from unstructured HC notes is time-consuming and challenging, especially in low-resource settings. Patients often struggle to recall goals, and the gap between bi-weekly or monthly coaching sessions can lead to a lack of timely feedback. This paper addresses the need for an automated system to identify and assess the "SMARTness" of goals discussed in HC sessions, making goal tracking more accessible and effective.
Methods
The SMARTMiner framework consists of two main components:
- Goal Extraction Module: This module identifies potential goal spans within the HC notes. The authors frame this as a multi-span reading comprehension (MSRC) task, using a fixed question ("What are the goals mentioned in the text?") to adapt existing MSRC models. They explore both extractive (e.g., SpanQualifier, CSS) and generative (e.g., LoRA fine-tuned LLMs, QASE) approaches.
- SMARTness Classification Module: Once a goal span is extracted, this module independently predicts binary values for the Specific (S), Measurable (M), and Attainable (A) components. These predictions are then aggregated to determine if the goal is SMART, Partially SMART, or Not SMART. This module uses transformer-based models.
A key methodological contribution is the two-stage fine-tuning strategy for extractive models: pre-training on a large, general MSRC dataset (MultiSpanQA) followed by fine-tuning on the domain-specific, low-resource SMARTSpan dataset.
Datasets
- SMARTSpan: A newly introduced dataset comprising 173 health coaching notes, annotated with 266 unique goal spans and their SMART attributes. This dataset is designed for low-resource, domain-specific scenarios and reflects the variability of real-world HC notes, including notes with no goals.
- MultiSpanQA: A widely used MSRC benchmark dataset used for pre-training and evaluating the generalizability of the goal extraction models.
Key Insights and Ideas
- Extractive models shine in low-resource, domain-specific settings: Unlike generative models that struggle with domain shift and hallucination, extractive models, especially when pre-trained on larger datasets, demonstrate superior performance in identifying precise goal spans in HC notes.
- Two-stage fine-tuning is crucial: Pre-training on a large, general dataset like MultiSpanQA significantly boosts the performance of extractive models on the smaller, specialized SMARTSpan dataset, highlighting the importance of task-aligned pretraining.
- Generative models are prone to hallucination and domain shift issues: While generative LLMs perform well on general MSRC tasks, they tend to generate irrelevant content or fail to accurately extract spans from the specialized HC notes.
- SMARTness classification is more than just a binary label: The authors' approach of predicting individual SMART components allows for more granular feedback, identifying why a goal might not be fully SMART.
- The "Relevant" and "Time-bound" components are tricky to annotate automatically: Due to their subjective nature or implicit definition within the intervention, these components were not directly annotated in the SMARTSpan dataset, presenting a challenge for full SMART goal evaluation.
Key Results
- Goal Extraction: Extractive models, particularly DeBERTa-v3-base fine-tuned sequentially on MultiSpanQA and SMARTSpan, achieved the highest performance, with EM F1 scores up to 90.11 and PM F1 scores up to 97.41 on SMARTSpan. Generative models performed significantly worse on SMARTSpan, with EM F1 scores as low as 17.79.
- SMARTness Classification: DeBERTa-v3-large achieved the highest scores, with an accuracy of 0.86, macro-average F1 of 0.85, and a SMART F1 score of 0.91. DeBERTa-v3-base also performed competitively.
- End-to-End Framework: The full SMARTMiner framework achieved 75.8% end-to-end accuracy, with the goal extractor correctly identifying 93.4% of gold goals and a precision of 91.2%.
Limitations and Future Work
- Anonymization Scalability: The manual anonymization process, while thorough, is not linearly scalable for larger datasets. Future work will involve more robust automated anonymization with human oversight.
- Limited SMART Components: The SMARTSpan dataset only annotates Specific, Measurable, and Attainable components. "Relevant" was excluded due to subjectivity, and "Time-bound" was often implicit. Future work could explore methods to infer or annotate these.
- GoalGuardian System: The authors envision extending SMARTMiner into "GoalGuardian," a fully autonomous system for weekly goal reviews, client check-ins, and progress reinforcement, aiming to sustain engagement between HC sessions.
- Improving Classification Nuance: Errors in SMARTness classification often occurred between adjacent categories (e.g., SMART vs. Partially SMART), suggesting a need for more training data that captures subtle variations in goal phrasing and clearer definitions for classification.
Summary updated