TL;DR
This paper introduces RealCBT, a novel dataset of authentic Cognitive Behavioral Therapy (CBT) dialogues, and conducts the first comparative analysis of emotional arcs between real and LLM-generated therapy sessions. The findings reveal that while LLM-generated dialogues are fluent, they lack the nuanced emotional variability and authentic reactivity characteristic of real therapy, particularly from the client's perspective.
What Problem Does This Paper Address?
The increasing use of Large Language Models (LLMs) to simulate therapeutic interactions for training and supplementing mental health resources hinges on the quality of generated dialogues. However, a critical gap exists in understanding whether these synthetic conversations accurately capture the complex emotional dynamics of real therapy. This is particularly important in CBT, where emotional states are central to diagnosis, treatment, and client progress. The scarcity of real-world therapy data due to privacy concerns has led to a reliance on synthetic data, making it crucial to assess its emotional fidelity. This paper addresses this gap by directly comparing the emotional trajectories of real and LLM-generated CBT sessions.
Methods
The authors adapted the Utterance Emotion Dynamics (UED) framework, a lexicon-based approach, to analyze fine-grained affective trajectories across valence, arousal, and dominance dimensions. This involved processing dialogues at the utterance level and modeling how these emotional states evolve over time for both the counselor and the client. They computed several UED metrics, including:
- Emotion Mean: Average emotional score.
- Emotion Variability: Standard deviation of scores, indicating emotional richness.
- Displacement Length: Average number of emotion words spoken, indicating emotional intensity.
- Emotion Rise Rate: Speed of emotional shifts (reactivity).
- Emotion Recovery Rate: Speed of returning to baseline emotional state (regulation).
These metrics were then used to compare real and synthetic dialogues, as well as individual speaker roles.
Datasets
- RealCBT: A newly curated dataset of 76 authentic Cognitive Behavioral Therapy (CBT) dialogues transcribed from public video-sharing platforms (YouTube, Vimeo). This dataset is made publicly available.
- CACTUS: A synthetic multi-turn dialogue dataset generated by LLMs, designed to simulate CBT conversations based on therapeutic principles. This dataset was used for comparison.
Key Insights and Ideas
- Emotional Fidelity Gap: LLM-generated therapy dialogues, while fluent, fall short in capturing the nuanced emotional dynamics of real therapy, especially for the client role.
- Variability is Key: Real therapy sessions exhibit significantly greater emotional variability, particularly in arousal and valence, reflecting a more authentic and responsive emotional flow. Synthetic dialogues tend to be more uniform.
- Authentic Reactivity vs. Exaggeration: Synthetic speakers, especially clients, show faster and more intense emotional reactivity, which may signal an overemphasis on expressive language rather than genuine emotional regulation.
- Counselor vs. Client Differences: Synthetic counselors show closer alignment with real counselors in terms of emotional tone, suggesting LLMs are adept at simulating professional demeanor. However, synthetic clients diverge more significantly, impacting the authenticity of the simulated therapeutic process.
- Low Arc Similarity: Emotional arc alignment between real and synthetic speakers is consistently low, underscoring the difficulty LLMs face in replicating the dynamic, situated, and role-sensitive emotional flow of real therapy.
Key Results
- Emotion Mean: Synthetic speakers generally exhibited higher emotional intensity (arousal and valence) compared to real speakers, with exceptions for client valence.
- Emotion Variability: Real sessions showed significantly greater variability in arousal (median p-value < 0.0001, r = 0.84) and valence compared to synthetic ones.
- Displacement Length: Speakers in real dialogues used emotion-laden language more dynamically, particularly in arousal and dominance.
- Emotion Rise Rate: Synthetic speakers tended to respond more quickly and intensely to emotional situations in dominance and valence.
- Emotion Recovery Rate: While largely comparable, synthetic clients showed stronger emotion regulation in valence and dominance.
- Emotional Arc Similarity: Correlation analysis revealed consistently low alignment between real and synthetic sessions across all pairings (Real-Real, Syn-Syn, and Real-Syn), with Real-Syn correlations near zero.
Limitations and Future Work
- Dataset Limitations: The RealCBT dataset is limited in size and exhibits subgroup imbalance, which affects generalizability.
- Scope: The analysis is limited to CBT sessions and comparisons with a single synthetic dataset (CACTUS). Findings may not extend to other therapy modalities or contexts.
- Future Directions: The authors propose expanding annotations to include dialogue acts and empathy markers, training affect-aware LLMs, and analyzing therapeutic strategies. They also suggest broadening the scope to other therapeutic approaches and languages, and incorporating alternative synthetic datasets to test robustness. The goal is to strengthen the dataset's utility and support the development of more emotionally grounded and clinically meaningful LLM systems.
Summary updated