TL;DR
This paper introduces a novel annotation scheme for weight management dialogues, grounded in the Trans-Theoretical Model (TTM) of behavior change. They collected and annotated role-play chats between "seekers" and "helpers" to build a dataset that can train AI systems to provide more personalized and effective health advice.
What Problem Does This Paper Address?
Motivating individuals to change unhealthy behaviors, like those leading to obesity, is a significant challenge. While online forums offer peer support, real-time, personalized conversations could be more effective. However, building AI agents that can provide such support requires understanding the nuances of health behavior change conversations. Existing approaches often rely on expert-authored content, which is difficult to scale. This paper addresses the need for a data-driven approach by developing a method to annotate and analyze real human conversations about weight management, specifically leveraging established psychological models of behavior change.
Methods
The core of this work is a new annotation scheme designed to capture key aspects of health behavior change within dialogues. This scheme is inspired by the Trans-Theoretical Model (TTM), which describes behavior change as a process with distinct stages (Stages of Change - SOC) and specific activities or strategies (Processes of Change - POC) used to move through these stages.
The authors collected dialogues through a role-play scenario on Amazon Mechanical Turk (MTurk), where participants acted as either a "help seeker" (overweight and wanting to lose weight) or a "help provider" (offering advice). These dialogues were then annotated by experts using the TTM-based scheme, along with standard conversational labels like greetings and questions.
To evaluate the utility of their annotation scheme, they built three separate classifiers (for SOC, POC, and "other" labels) using logistic regression. They experimented with two approaches:
- Unsegmented: Classifying entire sentences.
- Segmented: Segmenting sentences into smaller units before classification, assuming perfect segmentation (oracle segmentation).
Datasets
The paper introduces a newly collected and annotated dataset of human-human role-play dialogues focused on weight management. This dataset consists of:
- 52 users
- 26 dialogues
- 309 turns
- 1230 word tokens
This dataset is unique in its focus on naturalistic, peer-to-peer conversations about health behavior change, annotated with a TTM-inspired scheme.
Key Insights and Ideas
- TTM as a Dialogue Framework: The TTM provides a robust theoretical foundation for understanding and annotating health behavior change conversations, moving beyond simple topic identification to capturing the underlying psychological processes.
- Peer-to-Peer Data is Valuable: Role-play dialogues on MTurk, despite initial skepticism, yielded high-quality interactions, suggesting crowdsourcing can be an effective method for collecting nuanced conversational data for health-related AI.
- Segmentation Matters: Automatically segmenting sentences before classification significantly improves accuracy, highlighting that the meaning and context relevant to behavior change often reside within specific phrases or clauses, not necessarily the entire utterance.
- Overlap is Natural: Health behavior change conversations naturally involve overlapping annotations (e.g., a sentence can relate to both a Stage of Change and a Process of Change), which poses a challenge for annotation and classification but reflects the complexity of real-world interactions.
Key Results
The paper reports classification accuracies for predicting SOC, POC, and "other" labels. The key finding is that the segmented approach significantly outperforms the unsegmented approach across all label types.
- SOC Prediction: Segmented accuracy of 0.48 vs. Unsegmented accuracy of 0.44.
- POC Prediction: Segmented accuracy of 0.49 vs. Unsegmented accuracy of 0.41.
- "Other" Label Prediction: Segmented accuracy of 0.67 vs. Unsegmented accuracy of 0.35.
All these differences were found to be statistically significant. The "other" labels saw the most dramatic improvement with segmentation, suggesting that these more general conversational acts are often embedded within longer utterances.
Limitations and Future Work
- Data Sparsity: The authors acknowledge that the sparsity of their dataset negatively impacts classification accuracy and plan to expand it.
- Expert Annotators: The reliance on expert annotators familiar with the TTM is a limitation for scaling annotation efforts.
- Crowdsourcing for Annotation: They propose exploring crowdsourcing methods, potentially using TTM-based questionnaires, to facilitate annotation of larger datasets.
- Automatic Segmentation: The current "segmented" approach uses oracle segmentation. Future work will focus on developing automatic segmentation techniques.
- Dialogue System Development: A significant future direction is to use this annotated data and classification models to build a dialogue system that can effectively play the role of a "helper" in weight management conversations.
Summary updated