eCBT-I dialogue system: a comparative evaluation of large language models and adaptation strategies for insomnia treatment

Xueying Bao, Xingyu Zhu, Dongren Yang, Hao Lou, Ruoyun Wang, Yutong Wu, Wenhui Li, Yu Xia, Li Zeng, Yingying Pan, Xiqin Wang, Xian Zhang, Ling Cheng, Yinghui Ling, Yan Zhang, Qi Zhao, Mei YangJournal of Translational Medicine 2025 · DOI

TL;DR

This paper introduces a specialized dialogue system for insomnia treatment (eCBT-I) that combines Retrieval-Augmented Generation (RAG) with a curated knowledge graph. By systematically evaluating eight mainstream Large Language Models (LLMs) across three adaptation strategies, the researchers found that Qwen2-7b using a "Freeze" adaptation strategy delivered the best balance of professional accuracy, reasoning, and safety.

What Problem Does This Paper Address?

Cognitive Behavioral Therapy for Insomnia (CBT-I) is the gold standard for treatment, but traditional face-to-face therapy is plagued by high costs, limited specialist availability, and low patient adherence. While digital "eCBT-I" versions exist, they often lack the interactive, personalized touch of a human therapist.

Generic LLMs (like base GPT models) show promise but struggle with "hallucinations" or lack the specific medical rigor required for psychiatric intervention. The authors address the challenge of creating a specialized, safe, and efficient dialogue agent that can operate effectively even when trained on relatively small, domain-specific datasets.

Methods

The authors built an intelligent dialogue system based on a RAG (Retrieval-Augmented Generation) architecture. This setup uses a dual-encoder retriever to pull relevant facts from a specialized CBT-I knowledge base (books, clinical guides, and therapist transcripts) before the LLM generates a response, ensuring the advice is grounded in medical reality.

The core of the research is a massive "grid search" for the best model-tuning combination:

  • 8 LLMs: Including ChatGLM (2 & 3), Baichuan (7b & 13b), Qwen (7b & 2-7b), and Llama-2 (7b & 13b).
  • 3 Adaptation Strategies:
    • LoRA: Low-Rank Adaptation (updates a small set of added weights).
  • QLoRA: Quantized LoRA (4-bit quantization to save memory).
  • Freeze: Freezing most pre-trained parameters and only updating specific top layers.

The models were evaluated using standard NLP metrics (BLEU-4, ROUGE) and a specialized Chinese reasoning benchmark (C-eval).

Datasets

  1. CBT-I Dialogue Dataset: A newly constructed dataset of 22,780 initial records sourced from medical databases, sleep guides, and real therapist-patient interview transcripts. After rigorous AI and manual cleaning (filtering for semantic coherence and CBT-I relevance), they finalized 2,387 high-quality dialogue pairs.
  2. C-eval: A comprehensive external benchmark containing ~14k multiple-choice questions across 52 disciplines to test the models' general reasoning and knowledge mastery.

Key Insights and Ideas

  • The "Freeze" Strategy Wins for Small Data: For most models in this study, the "Freeze" strategy outperformed LoRA and QLoRA. The authors suggest that on small, vertical datasets, freezing the majority of weights prevents overfitting and preserves the model's inherent linguistic capabilities while allowing it to adapt to the specific "tone" of therapy.
  • Architecture-Specific Tuning: Adaptation isn't "one size fits all." While Qwen and Llama models excelled with the Freeze method, Baichuan models actually performed better with LoRA/QLoRA. This highlights the need for researchers to match tuning strategies to specific model architectures.
  • Efficiency vs. Performance: The study proves that a well-tuned 7-billion parameter model (like Qwen2-7b) can significantly outperform 13-billion parameter models in specialized tasks, offering a "sweet spot" for deployment in resource-constrained medical environments.

Key Results

  • Top Performer: Qwen2-7b (Freeze) achieved the highest scores, notably reaching an average accuracy of 0.8076 on the C-eval benchmark, far outstripping competitors like Qwen-7b (0.6062).
  • Inference Speed: Baichuan-7b was the fastest (1.54s response time), but Qwen2-7b maintained a respectable 6.16s, well within the acceptable range for semi-synchronous human-machine interaction.
  • Safety and Harmfulness: Using a 5-point Likert scale, experts rated the system's safety. The average score was 4.89/5, with 91.2% of responses rated as "Strongly Disagree" regarding potential harmfulness.

Limitations and Future Work

The authors acknowledge several areas for improvement:

  • Model Recency: The study was limited to eight models available at the time of research; newer iterations like Qwen 3 or Llama 4 were not included.
  • Non-verbal Cues: Current eCBT-I is text-only. Future versions could integrate multimodal inputs like facial expression recognition or voice intonation to better mimic a human therapist's empathy.
  • Clinical Validation: While the technical evaluation is robust, the authors call for multi-center randomized controlled trials (RCTs) to prove the system's actual therapeutic efficacy in real-world patient outcomes.

Summary updated

Abstract

This study effectively integrates the CBT-I knowledge graph with the large language model through the RAG architecture, which improves the professionalism of the eCBT-I intelligent dialogue system. The systematic fine-tuning method selection process and the confirmation of the optimal model not only improve the adaptability of the large language model in the CBT-I task, but also provide a useful paradigm for AI applications in medical subfields with resource constraints and difficulties in data collection, laying a solid foundation for more accurate and efficient digital CBT-I clinical practice in the future.
BibTeX
@article{bao2025ecbt,
  author = {Xueying Bao and Xingyu Zhu and Dongren Yang and Hao Lou and Ruoyun Wang and Yutong Wu and Wenhui Li and Yu Xia and Li Zeng and Yingying Pan and Xiqin Wang and Xian Zhang and Ling Cheng and Yinghui Ling and Yan Zhang and Qi Zhao and Mei Yang},
  title = {{eCBT-I dialogue system: a comparative evaluation of large language models and adaptation strategies for insomnia treatment}},
  journal = {Journal of Translational Medicine},
  year = {2025},
  doi = {10.1186/s12967-025-06871-y},
}
Figure