CPsyExam: A Chinese Benchmark for Evaluating Psychology using Examinations

Jiahao Zhao, Jingwei Zhu, Minghuan Tan, Min Yang, Di Yang, Chenhao Zhang, Guancheng Ye, Chengming Li, Xiping Hu

Abstract

As Large Language Models (LLMs) are increasingly integrated into mental health services, assessing their domain-specific competence becomes paramount. This paper presents CPsyExam, the first comprehensive benchmark designed to evaluate psychological expertise within the Chinese context. Comprising over 22,000 questions sourced from professional certification and academic entrance exams, CPsyExam evaluates models across two primary tracks: factual knowledge and practical case analysis. Our evaluation of leading LLMs reveals a significant "reasoning gap," where models proficient in factual recall struggle with the complex logical application required for clinical scenarios.

Introduction

The deployment of LLMs for psychological counseling and mental health support necessitates a rigorous evaluation of their professional knowledge. General benchmarks often oversimplify psychology, treating it as a broad category without accounting for the nuances of clinical practice or regional examination standards. Existing specialized benchmarks are frequently limited in scope, focusing on narrow sub-fields. CPsyExam addresses this gap by providing a "gold standard" benchmark that spans the entire Chinese psychological examination system, including teacher qualifications, professional counselor licenses, and graduate entrance exams.

Methodology and Framework

The construction of CPsyExam followed a multi-stage pipeline involving data collection, cleaning, and taxonomy design.

Overview of the CPsyExam construction pipeline and data sources.

Data Collection and Processing

The researchers gathered 22,000 questions through web crawling and OCR from four major Chinese examination systems:

  • Graduate Entrance Examinations (GEE)
  • Psychological Counselor Examination (PCE)
  • Teachers’ Qualification Examination (TQE)
  • Self-study Examination (SSE)

Benchmark Design

The benchmark is bifurcated into two tracks:

  1. Knowledge (KG): Focuses on fact-based recall across 39 sub-disciplines (e.g., Developmental, Abnormal, and Social Psychology).
  2. Case Analysis (CA): Scenario-based questions that require models to identify psychological states, reason through symptoms, and apply clinical theories.

To ensure robustness, the benchmark includes diverse question formats: Single-Choice Questions (SCQ), Multiple-Response Questions (MAQ), and Open-ended QA. MAQs are particularly challenging as they require a 100% correct selection of all valid options, preventing models from relying on simple elimination strategies.

Examples of Knowledge and Case Analysis questions from the benchmark.

Key Insights and Results

Evaluation of proprietary and open-source models yielded several critical findings:

The Reasoning Gap

While top-tier models like GPT-4 demonstrate strong performance in the Knowledge track (achieving ~78% accuracy), their performance declines significantly in Case Analysis. For instance, GPT-4's accuracy dropped from 76.56% on Knowledge SCQs to 60.33% on Case Analysis SCQs. This suggests that while LLMs can memorize psychological facts, they struggle with the "Reasoning" and "Application" steps essential for clinical work.

The Fine-Tuning Paradox

A surprising result was that models specifically fine-tuned for psychological dialogue (e.g., MeChat, MindChat) often underperformed compared to their general-purpose base models. This suggests that current Supervised Fine-Tuning (SFT) approaches may lead to "catastrophic forgetting" or prioritize conversational fluency over rigorous logical reasoning.

Persona and Prompting

Experiments with role-playing prompts showed that a model's "persona" significantly impacts its performance. When ChatGLM2-6B was prompted to act as an "expert," its accuracy improved by approximately 14% compared to a "layperson" prompt, indicating that latent domain knowledge can be unlocked through effective context setting.

Conclusion

CPsyExam provides a critical tool for assessing the readiness of LLMs for psychological applications. The benchmark highlights the disparity between factual recall and clinical reasoning, serving as a roadmap for future model development. Future work will focus on expanding the benchmark to cross-cultural frameworks and refining automated evaluation metrics with human-in-the-loop verification.

Summary updated

BibTeX
@inproceedings{zhao2025cpsyexam,
  author = {Jiahao Zhao and Jingwei Zhu and Minghuan Tan and Min Yang and Di Yang and Chenhao Zhang and Guancheng Ye and Chengming Li and Xiping Hu},
  title = {{CPsyExam: A Chinese Benchmark for Evaluating Psychology using Examinations}},
  booktitle = {International Conference on Computational Linguistics},
  year = {2025},
  url = {https://openreview.net/forum?id=nQbQnl3LtA},
}
Figure