CASE: Efficient Curricular Data Pre-training for Building Assistive Psychology Expert Models

Sarthak Harne, Monjoy Narayan Choudhury, Madhav Rao, T. K. Srikanth, Seema Mehrotra, Apoorva Vashisht, Aarushi Basu, Manjit SodhiEMNLP 2024 · DOI

Introduction

The global shortage of mental health professionals—particularly in regions like India where only 0.75 professionals exist per 100,000 people—has led to an increased reliance on online forums for support. However, the volume of data on these platforms necessitates automated screening tools. Traditional Natural Language Processing (NLP) approaches often rely on massive, uncurated datasets from the web, which are frequently noisy and lack the theoretical depth required for clinical accuracy.

This paper introduces CASE-BERT, a model designed to bridge this gap by prioritizing data quality over quantity. By pre-training on a curated "curriculum" of psychology textbooks, CASE-BERT achieves state-of-the-art performance while using up to 2,300x less data than existing domain-specific models.

The CASE Framework

The authors propose the CASE (Curricular Assistive Psychology Expert) framework, which draws an analogy between the training of a machine learning model and the professional development of a human psychologist. This process occurs in three distinct stages:

  1. Base Model (The High School Graduate): The framework begins with a standard, off-the-shelf BERT model possessing general linguistic capabilities.
  2. Curricular Pre-training (The Psychology Student): The model is pre-trained on a specialized "Curricular Dataset" consisting of 110 psychology textbooks and journals (approx. 7.57 million words). Using Masked Language Modeling (MLM), the model learns the theoretical foundations of the field.
  3. Fine-tuning (The Practitioner): Finally, the model is fine-tuned on labeled social media data to perform specific diagnostic tasks, such as identifying signs of depression or anxiety.
The CASE framework mimics the professional development of a human expert through curated data stages.

Experimental Results and Data Efficiency

The primary breakthrough of CASE-BERT lies in its extreme data efficiency. While models like ClinicalBERT and BioBERT are pre-trained on over 18 billion words, CASE-BERT utilizes only 7.57 million words. Despite this massive reduction in scale, CASE-BERT consistently outperforms its larger counterparts on mental health screening benchmarks.

CASE-BERT achieves superior F1 scores for depression detection despite using significantly less pre-training data than baselines.

On the CounselChat dataset, CASE-BERT-Base achieved an F1 score of 0.909 for identifying depression, compared to 0.853 for MentalBERT and 0.754 for ClinicalBERT. Similar performance gains were observed on social media benchmarks like Depression_Reddit, where the model reached a 0.965 F1 score.

The Generative Gap

A notable finding of this research is the performance gap between specialized discriminative models and large generative models (LLMs) like Llama 3 and GPT-3.5. While LLMs showed high recall, they suffered from low precision, frequently generating false positives. In contrast, CASE-BERT’s structured pre-training allowed for a more balanced and precise classification, suggesting that "textbooks are all you need" for building reliable discriminative experts in specialized domains.

Conclusion and Future Work

CASE-BERT demonstrates that high-quality, structured knowledge is more effective than uncurated big data for specialized classification tasks. While the current model focuses on depression and anxiety, future work aims to expand the curriculum to a broader range of disorders and explore generative diagnostic summaries. This framework provides a blueprint for developing efficient, privacy-conscious expert models in fields where data is scarce or sensitive.

Summary updated

BibTeX
@inproceedings{harne2024case,
  author = {Sarthak Harne and Monjoy Narayan Choudhury and Madhav Rao and T. K. Srikanth and Seema Mehrotra and Apoorva Vashisht and Aarushi Basu and Manjit Sodhi},
  title = {{CASE: Efficient Curricular Data Pre-training for Building Assistive Psychology Expert Models}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2024},
  doi = {10.18653/v1/2024.findings-emnlp.925},
}
Figure