MIRROR: Multimodal Cognitive Reframing Therapy for Rolling with Resistance

Subin Kim, Hoonrae Kim, Jihyun Lee, Yejin Jeon, Gary LeeEMNLP 2025

TL;DR

This paper introduces MIRROR, a novel multimodal dataset and approach for AI-powered cognitive reframing therapy that incorporates nonverbal cues (facial expressions) to better manage client resistance. By moving beyond text-only models, MIRROR aims to improve therapeutic alliance and treatment efficacy, demonstrating significant gains in handling resistance compared to existing methods.

What Problem Does This Paper Address?

Traditional AI-driven cognitive behavioral therapy (CBT) models, primarily relying on text, struggle to effectively address client resistance. This resistance, a common therapeutic challenge, often manifests through nonverbal cues like facial expressions, sighs, or posture shifts, which text-only models cannot perceive. This limitation can lead to premature problem-solving rather than addressing deeper emotional needs, ultimately weakening the therapeutic alliance and reducing treatment efficacy. The authors also highlight the significant privacy risks associated with collecting real multimodal psychotherapy data, necessitating the creation of a synthetic dataset.

Methods

The core of this work is the MIRROR dataset, a synthetic multimodal dataset that pairs client statements with corresponding facial images. The dataset was constructed in three main stages:

  1. Multimodal Dialogue Design: This stage involved creating client profiles by combining existing text-only datasets (CACTUS) with facial identities from CelebA, augmented with four distinct resistance types (cognitive, emotional, behavioral, and non-resistant).
  2. Counseling Screenplay Generation: LLMs (GPT-40-MINI) were used to generate realistic dialogue screenplays, incorporating stage directions to explicitly represent nonverbal cues.
  3. Facial Expression Synthesis: A diffusion-based model (PhotoMaker) was employed to synthesize turn-level facial expressions that align with the dialogue and stage directions, preserving facial identity.

The authors also introduce "emotional captioning," an adaptation of chain-of-thought prompting, to interpret client emotional states from facial expressions and guide the AI therapist's responses. They evaluated their approach using a Vision-Language Model (VLM) fine-tuned on the MIRROR dataset.

Datasets

  • MIRROR (Multimodal Interactive Rolling with Resistance): A newly introduced synthetic multimodal dataset. It contains over 3,000 multimodal counseling dialogues, with each client turn annotated with a facial expression image reflecting their emotional state. It pairs client statements with corresponding facial images, designed to simulate real therapeutic interactions and address client resistance.
  • CelebA: Used for facial identity preservation.
  • CACTUS: Used as the source for text-based cognitive reframing therapy dialogues.
  • AnnoMI: Used for real-world counseling demonstrations.

Key Insights and Ideas

  • Nonverbal Cues are Crucial for Resistance Management: The study strongly suggests that incorporating nonverbal cues, particularly facial expressions, significantly enhances an AI therapist's ability to detect and manage client resistance, leading to better therapeutic outcomes.
  • Synthetic Data Can Bridge the Gap: The creation of the MIRROR dataset demonstrates that high-quality synthetic multimodal data can be generated to train AI therapists, overcoming the privacy and logistical challenges of using real-world data.
  • Emotional Captioning Improves Attunement: The proposed emotional captioning strategy, which interprets facial expressions to generate short textual descriptions of the client's emotional state, helps the AI therapist generate more empathetic and context-aware responses.
  • Multimodal Models Show Greater Resilience: Vision-enhanced models demonstrated greater resilience in maintaining performance when interacting with resistant clients compared to text-only models.

Key Results

  • MIRROR-LLAVAP+EC (the VLM fine-tuned on MIRROR with planning and emotional captioning) significantly outperformed baseline text-based LLMs and general-purpose VLMs in therapist skills assessment and client alliance building.
  • The model achieved higher scores in general counseling skills and CBT-specific skills, particularly in "Collaboration" and "Focus," when interacting with resistant clients.
  • Client alliance assessment showed substantial gains in "Approach" and "Affective Bond" for MIRROR-LLAVAP+EC, indicating improved rapport and empathy.
  • Domain expert evaluations further confirmed MIRROR-LLAVAP+EC's superiority in therapeutic alliance across all dimensions.
  • The response length analysis revealed that while longer responses might be favored by GPT-based evaluators, shorter, more focused responses from fine-tuned CBT models led to better performance and client understanding.

Limitations and Future Work

  • Biases in Image Generation: The use of LLMs and the CelebA dataset introduces potential cultural and demographic biases, which may underrepresent certain racial or ethnic groups. Future work aims to explore multimodal datasets with more balanced demographics.
  • Authenticity of Nonverbal Cues: Synthesized facial images may not fully capture the authenticity of real-world expressions, and the study primarily focused on facial expressions and utterances, neglecting other nonverbal cues like body posture and voice tone. Future work could incorporate a wider range of nonverbal signals.
  • Scope and Session Length: The dataset consists of relatively short, single-session dialogues, limiting the ability to capture longer-term therapeutic processes. Future work could extend the dataset to longer, multi-session interactions and explore multilingual settings.
  • Limited Expert Involvement: While rigorous filtering was applied, the dataset creation process did not involve direct human expert supervision, which remains critical for clinical fidelity. Future work should incorporate expert-driven criteria.
  • Model Selection: The evaluation was based on a single backbone model (LLAVA-v1.5-7B). Future research could compare multiple VLM architectures and benchmark against licensed human therapists.
  • GPT-Based Evaluation: While scalable, GPT-based evaluation has limitations, including length bias and potential under-recognition of relational qualities. Future work should continue to integrate human expert judgments.

Summary updated

BibTeX
@inproceedings{kim2025mirror,
  author = {Subin Kim and Hoonrae Kim and Jihyun Lee and Yejin Jeon and Gary Lee},
  title = {{MIRROR: Multimodal Cognitive Reframing Therapy for Rolling with Resistance}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2025},
}
Figure