Introduction
The global shortage of mental health professionals—particularly in regions like India where only 0.75 professionals exist per 100,000 people—has led to an increased reliance on online forums for support. However, the volume of data on these platforms necessitates automated screening tools. Traditional Natural Language Processing (NLP) approaches often rely on massive, uncurated datasets from the web, which are frequently noisy and lack the theoretical depth required for clinical accuracy.
This paper introduces CASE-BERT, a model designed to bridge this gap by prioritizing data quality over quantity. By pre-training on a curated "curriculum" of psychology textbooks, CASE-BERT achieves state-of-the-art performance while using up to 2,300x less data than existing domain-specific models.
The CASE Framework
The authors propose the CASE (Curricular Assistive Psychology Expert) framework, which draws an analogy between the training of a machine learning model and the professional development of a human psychologist. This process occurs in three distinct stages:
- Base Model (The High School Graduate): The framework begins with a standard, off-the-shelf BERT model possessing general linguistic capabilities.
- Curricular Pre-training (The Psychology Student): The model is pre-trained on a specialized "Curricular Dataset" consisting of 110 psychology textbooks and journals (approx. 7.57 million words). Using Masked Language Modeling (MLM), the model learns the theoretical foundations of the field.
- Fine-tuning (The Practitioner): Finally, the model is fine-tuned on labeled social media data to perform specific diagnostic tasks, such as identifying signs of depression or anxiety.
Experimental Results and Data Efficiency
The primary breakthrough of CASE-BERT lies in its extreme data efficiency. While models like ClinicalBERT and BioBERT are pre-trained on over 18 billion words, CASE-BERT utilizes only 7.57 million words. Despite this massive reduction in scale, CASE-BERT consistently outperforms its larger counterparts on mental health screening benchmarks.
On the CounselChat dataset, CASE-BERT-Base achieved an F1 score of 0.909 for identifying depression, compared to 0.853 for MentalBERT and 0.754 for ClinicalBERT. Similar performance gains were observed on social media benchmarks like Depression_Reddit, where the model reached a 0.965 F1 score.
The Generative Gap
A notable finding of this research is the performance gap between specialized discriminative models and large generative models (LLMs) like Llama 3 and GPT-3.5. While LLMs showed high recall, they suffered from low precision, frequently generating false positives. In contrast, CASE-BERT’s structured pre-training allowed for a more balanced and precise classification, suggesting that "textbooks are all you need" for building reliable discriminative experts in specialized domains.
Conclusion and Future Work
CASE-BERT demonstrates that high-quality, structured knowledge is more effective than uncurated big data for specialized classification tasks. While the current model focuses on depression and anxiety, future work aims to expand the curriculum to a broader range of disorders and explore generative diagnostic summaries. This framework provides a blueprint for developing efficient, privacy-conscious expert models in fields where data is scarce or sensitive.
Summary updated