A Deep Learning Approach to Modeling Empathy in Addiction Counseling

James Gibson, Dogan Can, Bo Xiao, Zac E. Imel, David C. Atkins, Panayiotis G. Georgiou, Shrikanth S. NarayananInterspeech 2016 · DOI

TL;DR

This paper introduces a novel two-stage deep learning system to automatically predict counselor empathy in addiction counseling sessions from transcripts. By first learning to map conversational turns to local behavioral acts and then using this learned representation to predict global session-level empathy, the system significantly outperforms traditional end-to-end deep learning models. This work offers a promising avenue for objective and scalable assessment of therapeutic communication skills.

What Problem Does This Paper Address?

Motivational interviewing (MI) is a therapeutic approach crucial for helping individuals overcome addiction by exploring their ambivalence towards change. A key factor in MI's success is the counselor's ability to convey empathy, defined as understanding and grasping the client's perspective. Traditionally, assessing empathy requires costly and time-consuming manual analysis of therapy sessions. While previous research has explored machine learning for analyzing MI, this paper tackles the challenge of building a robust, automated system to predict counselor empathy directly from session transcripts, aiming to provide a more scalable and objective evaluation method.

Methods

The authors propose a two-stage deep learning approach.

  • Stage 1: Local Behavior Encoding (Encoder): A recurrent neural network (RNN), specifically a Long Short-Term Memory (LSTM) network, is trained to map sequences of conversational turns (represented by word embeddings) to a set of local behavioral acts. These acts are manually annotated utterance-level codes from the MISC manual. This stage acts as an "encoder," learning a low-dimensional representation of the dynamics of the client-counselor interaction.
  • Stage 2: Empathy Prediction (Decoder): The output of the encoder (prior to the sigmoid activation) is averaged across turns and fed into a feedforward layer. This layer predicts the global session-level empathy rating, which is binarized into "high" or "low" based on a Likert scale cutoff. This stage acts as a "decoder," using the learned local behavior representations to predict the final empathy score.

The system is trained in two stages: first, the encoder is trained to predict local behaviors. Then, the entire network (encoder and decoder) is trained end-to-end, using the pre-trained encoder weights to initialize the lower layers. This pre-training strategy is hypothesized to improve performance by providing a more informed starting point for the deep network.

A baseline model is also presented, which uses a single feedforward neural network to directly predict empathy from turn vectors without the intermediate local behavior encoding stage.

Datasets

The study utilizes a corpus of motivational interviews collected from six independent clinical studies focusing on addiction counseling (alcohol, marijuana, and other drug abuse). This corpus includes 148 sessions from real patients and an additional 200 sessions from the Context Tailored Training (CTT) dataset, which includes both real and standardized patients.

All data were manually transcribed and segmented. Utterances were assigned local-level behavioral codes according to the MISC manual (MISC28, with 19 counselor and 9 client codes). Session-level behavioral codes, including empathy, were rated on a 1-7 Likert scale and then binarized. The dataset was split into training (228 sessions) and testing (109 sessions) sets, with efforts made to balance empathy distributions.

Key Insights and Ideas

  • The Power of Intermediate Representations: The core insight is that modeling local conversational dynamics (behavioral acts) as an intermediate step significantly improves the prediction of global session-level empathy. This suggests that empathy is not just a direct response to immediate language but is built upon a sequence of nuanced interactions.
  • Pre-training for Deep Networks: Training the deep network in two stages, with the first stage supervised by local behavioral codes, acts as a form of pre-training. This approach provides a better initialization for the network, allowing it to learn more effectively than training the entire deep network end-to-end from scratch, especially with limited data.
  • MISC8 vs. MISC28: The study found that a reduced set of MISC codes (MISC8), focusing on the most frequent ones, yielded better prediction performance for local behaviors. This suggests that sparsity in the full MISC28 set poses a challenge, and a more robust representation can be achieved by focusing on the most informative codes. This MISC8 representation also provided a better initialization for the empathy prediction task.
  • Language Dynamics Matter: The paper highlights that while word embeddings capture semantic meaning, the sequential nature of conversations (turn dynamics) is crucial for understanding therapeutic communication. The RNN component explicitly models these dynamics.

Key Results

  • The proposed two-stage system with pre-training achieved a higher Unweighted Average Recall (UAR) for predicting session-level empathy compared to the baseline model and the end-to-end trained deep network.
  • The proposed system with an 8-dimensional encoding layer (MISC8) achieved a UAR of 78.6%, outperforming the baseline model (71.8%).
  • The "reference" model, which used the ground truth MISC labels as input to predict empathy, achieved the highest UAR (79.6%), demonstrating the strong predictive power of local behavioral codes for empathy.
  • The proposed system with pre-training using the MISC8 representation performed better than the reference MISC8 model, suggesting that the deep system can learn from both turn-level language dynamics and local behavioral acts.
  • The MISC8 representation was more robustly predicted than MISC28 in the first stage, achieving a higher F1-score (0.643 vs. 0.258).

Limitations and Future Work

  • Word Order Ignored: The current turn representation uses an average of word vectors, neglecting the order of words within a turn. The authors suggest modeling turns as sequences of word vectors to capture this important information.
  • Loss Function for Local Behaviors: The current loss function for the first stage does not explicitly account for correlations between local behavioral codes. Future work could explore multi-labeling loss functions to address this.
  • Attention Mechanisms: Incorporating an attention mechanism could allow the system to assign different weights to different turns, potentially improving performance and interpretability by focusing on behaviorally salient turns.
  • Audio Features: The authors are interested in extending the system to incorporate audio-derived features and using automatic speech recognition (ASR) to process audio data, which could enable analysis of sessions without manual transcription.
  • Larger Datasets: The paper mentions access to a larger corpus of sessions with global behavioral ratings but without transcripts or MISC annotations, which could be used to train and evaluate ASR-based systems.

Summary updated

BibTeX
@inproceedings{gibson2016deep,
  author = {James Gibson and Dogan Can and Bo Xiao and Zac E. Imel and David C. Atkins and Panayiotis G. Georgiou and Shrikanth S. Narayanan},
  title = {{A Deep Learning Approach to Modeling Empathy in Addiction Counseling}},
  booktitle = {Interspeech},
  year = {2016},
  doi = {10.21437/interspeech.2016-554},
}
Figure