TL;DR
This paper explores low-resource, real-time assessment of therapist empathy in counseling using pre-trained language models. They investigate whether empathy learned from general conversations can be transferred to the specialized domain of motivational interviewing (MI), finding that while some transfer exists, it's not yet reliable enough for direct application.
What Problem Does This Paper Address?
Accurately gauging therapist empathy is crucial for effective counseling, particularly in motivational interviewing (MI), as it directly impacts client rapport and outcomes. Traditional methods rely on manual annotation by trained experts, which is time-consuming, costly, and difficult to scale. While machine learning has been applied to session-level empathy assessment, it often requires large, well-annotated datasets of therapeutic dialogues, which are scarce. Furthermore, the potential to leverage empathy knowledge from readily available general conversation data for low-resource therapeutic settings has been largely unexplored. This paper aims to bridge this gap by investigating low-resource, utterance-level empathy assessment in MI, focusing on transferring knowledge from out-of-domain general conversations.
Methods
The core of the paper's approach involves training deep learning models, specifically BERT, on general conversation datasets to learn empathy patterns. They then apply these models to assess empathy in therapeutic dialogues. The key strategy is "out-of-domain (OOD) empathy contrast," where models are trained on pairs of empathetic vs. non-empathetic utterances from general conversations (e.g., from subreddits like r/Happy vs. r/CasualConversation). This is done at both the utterance level (considering only the therapist's utterance) and the two-utterance level (considering both client and therapist utterances for context). They also explore unsupervised baselines using Natural Language Inference (NLI) to frame empathy prediction as an entailment task.
Datasets
- General Conversation Datasets:
- Persona-based Empathetic Conversation (PEC): Comprises conversations from r/Happy, r/OffMyChest, and r/CasualConversation subreddits.
- EmpatheticDialogues (ED): A dataset of 23.1K general conversations from MTurker pairs, where speakers were given an emotion label to guide their conversation.
- Therapeutic Dialogue Dataset:
- ROLEPLAYMI: A publicly available dataset of transcripts from MI demonstrations, categorized as high-quality (MI-adherent) and low-quality (MI non-adherent). A subset of this dataset was manually annotated for utterance-level empathy, forming the ANNO benchmark.
Key Insights and Ideas
- The "Empathy Gap": There's a significant difference in how empathy is expressed in general conversations versus therapeutic settings. General conversations often use questions to express empathy, while MI-adherent therapists tend to use non-question statements to acknowledge client feelings. This domain difference poses a challenge for direct transfer learning.
- OOD Contrast is Better Than Nothing: Training models on OOD empathy contrast (general conversation empathy vs. non-empathy) yields better results on therapeutic data than training on OOD data without this contrast, even if the performance is still suboptimal.
- Brittleness of OOD Models: Models trained on general conversations show a significant drop in performance when applied to MI dialogues, indicating their "brittleness" and sensitivity to domain shifts.
- Questions as Noise: Removing question utterances from the therapeutic benchmark dataset (¬Q.ANNO) sometimes improves performance, suggesting that questions, often labeled as non-empathetic in MI, can introduce noise for OOD models.
- Client-Therapist Agreement as a Proxy: An unsupervised baseline using NLI to measure client-therapist agreement showed promising results, outperforming some OOD contrast models, suggesting a correlation between agreement and empathy.
Key Results
- Models trained on OOD empathy contrast achieved significantly better scores on their respective test sets (e.g., MCC > 0.9 for ED vs. r/CC (2)) compared to their performance on the manually annotated MI dataset (ANNO), where scores dropped to around 0.10 MCC.
- This performance drop on ANNO indicates that OOD empathy contrast models are not yet reliable predictors of therapeutic empathy.
- Among OOD models, r/OMC vs. r/CC (2) (2-utterance training) performed best on ANNO, achieving an average MCC of 0.17, suggesting that incorporating more context can be beneficial.
- Unsupervised NLI baselines scored around 0.05 MCC, while the client-therapist agreement NLI baseline achieved better results, outperforming half of the OOD contrast models.
Limitations and Future Work
- Heuristic Labeling: The heuristic empathy labels used for the general conversation datasets (PEC and ED) might not be perfectly accurate at the utterance level.
- Domain Shift: The inherent differences between general conversation and therapeutic dialogue, including the use of written chat vs. spoken transcripts, contribute to performance limitations.
- Lack of Punctuation: The ROLEPLAYMI dataset lacks punctuation, preventing fine-grained sentence-level analysis of empathy within utterances.
Future Work: * Investigate more fine-grained empathy annotation and prediction (e.g., at the sentence level) to reduce noise. * Explore few-shot learning methods for empathy classification, using OOD empathy contrast as a bootstrapping step for in-domain fine-tuning. * Further investigate the impact of the domain gap and explore methods to smooth it.
Summary updated