Abstract
As Large Language Models (LLMs) are increasingly integrated into mental health services, assessing their domain-specific competence becomes paramount. This paper presents CPsyExam, the first comprehensive benchmark designed to evaluate psychological expertise within the Chinese context. Comprising over 22,000 questions sourced from professional certification and academic entrance exams, CPsyExam evaluates models across two primary tracks: factual knowledge and practical case analysis. Our evaluation of leading LLMs reveals a significant "reasoning gap," where models proficient in factual recall struggle with the complex logical application required for clinical scenarios.
Introduction
The deployment of LLMs for psychological counseling and mental health support necessitates a rigorous evaluation of their professional knowledge. General benchmarks often oversimplify psychology, treating it as a broad category without accounting for the nuances of clinical practice or regional examination standards. Existing specialized benchmarks are frequently limited in scope, focusing on narrow sub-fields. CPsyExam addresses this gap by providing a "gold standard" benchmark that spans the entire Chinese psychological examination system, including teacher qualifications, professional counselor licenses, and graduate entrance exams.
Methodology and Framework
The construction of CPsyExam followed a multi-stage pipeline involving data collection, cleaning, and taxonomy design.
Data Collection and Processing
The researchers gathered 22,000 questions through web crawling and OCR from four major Chinese examination systems:
- Graduate Entrance Examinations (GEE)
- Psychological Counselor Examination (PCE)
- Teachers’ Qualification Examination (TQE)
- Self-study Examination (SSE)
Benchmark Design
The benchmark is bifurcated into two tracks:
- Knowledge (KG): Focuses on fact-based recall across 39 sub-disciplines (e.g., Developmental, Abnormal, and Social Psychology).
- Case Analysis (CA): Scenario-based questions that require models to identify psychological states, reason through symptoms, and apply clinical theories.
To ensure robustness, the benchmark includes diverse question formats: Single-Choice Questions (SCQ), Multiple-Response Questions (MAQ), and Open-ended QA. MAQs are particularly challenging as they require a 100% correct selection of all valid options, preventing models from relying on simple elimination strategies.
Key Insights and Results
Evaluation of proprietary and open-source models yielded several critical findings:
The Reasoning Gap
While top-tier models like GPT-4 demonstrate strong performance in the Knowledge track (achieving ~78% accuracy), their performance declines significantly in Case Analysis. For instance, GPT-4's accuracy dropped from 76.56% on Knowledge SCQs to 60.33% on Case Analysis SCQs. This suggests that while LLMs can memorize psychological facts, they struggle with the "Reasoning" and "Application" steps essential for clinical work.
The Fine-Tuning Paradox
A surprising result was that models specifically fine-tuned for psychological dialogue (e.g., MeChat, MindChat) often underperformed compared to their general-purpose base models. This suggests that current Supervised Fine-Tuning (SFT) approaches may lead to "catastrophic forgetting" or prioritize conversational fluency over rigorous logical reasoning.
Persona and Prompting
Experiments with role-playing prompts showed that a model's "persona" significantly impacts its performance. When ChatGLM2-6B was prompted to act as an "expert," its accuracy improved by approximately 14% compared to a "layperson" prompt, indicating that latent domain knowledge can be unlocked through effective context setting.
Conclusion
CPsyExam provides a critical tool for assessing the readiness of LLMs for psychological applications. The benchmark highlights the disparity between factual recall and clinical reasoning, serving as a roadmap for future model development. Future work will focus on expanding the benchmark to cross-cultural frameworks and refining automated evaluation metrics with human-in-the-loop verification.
Summary updated