Predicting Psychological Health from Childhood Essays. The UGent-IDLab CLPsych 2018 Shared Task System

Klim Zaporojets, Lucas Sterckx, Johannes Deleu, Thomas Demeester, Chris Develder

Abstract

This paper describes the system developed by the UGent-IDLab team for Task A of the CLPsych 2018 Shared Task. The objective was to predict psychological health scores (Anxiety, Depression, and Total) of 11-year-old children based on handwritten essays from the National Child Development Study (NCDS). Our approach leverages a diverse ensemble of models, including traditional linear regressors, gradient boosting, and deep learning architectures like CNNs and RNNs. By utilizing a greedy forward model selection strategy, we combined lexical features, domain-specific lexicons, and textual statistics to achieve second place in the competition with a disattenuated Pearson correlation of 0.5667 for the Total score.

Introduction

Identifying psychological distress in early childhood is critical for long-term well-being. The CLPsych 2018 Shared Task (Task A) challenged participants to predict mental health markers from essays written by 11-year-olds in 1958 as part of the National Child Development Study (NCDS). These children were asked to imagine their lives at age 25. The dataset consists of 9,217 short essays (average 225 characters) that were transcribed from handwriting, often containing spelling errors and transcription artifacts. Psychological ground truth was provided by teachers using the Bristol Social Adjustment Guides (BSAG), yielding scores for anxiety, depression, and total adjustment.

Methodology

Our system employs a "strength in numbers" strategy, integrating various feature sets and model architectures.

Feature Engineering

We extracted several types of features to capture different aspects of the text:

  • Lexical Features: Bag-of-words and bag-of-character n-grams. Character-level features were particularly effective at handling spelling variations common in children's writing.
  • Domain-Specific Lexicons: LIWC (Linguistic Inquiry and Word Count) and DepecheMood were used to capture emotional and mental state categories.
  • Textual Statistics: Metrics such as sentence length, spelling mistake ratios, and the frequency of "illegible" tokens from the transcription process.
  • Language Model Features: Perplexity scores from a pre-trained AWD-LSTM were used to measure linguistic coherence.

Model Architectures

We trained a variety of regressors to ensure diversity in the ensemble:

  • Linear Models: Ridge Regression and Support Vector Machines (SVMs).
  • Gradient Boosting: XGBoost models trained on engineered features.
  • Deep Learning: Feed-Forward Neural Networks (FFNN) using designed features, Convolutional Neural Networks (CNN) for local pattern recognition, and Recurrent Neural Networks (RNN) with self-attention for sequential dependencies.

Ensembling Strategy

The final predictions were generated using Forward Model Selection. This greedy algorithm iteratively adds individual models to the ensemble only if they improve the disattenuated Pearson correlation on the validation set.

Results and Discussion

The ensemble approach significantly outperformed individual models. A key insight was that model diversity was more important than individual accuracy; models with different "viewpoints" (e.g., character-level CNNs vs. lexicon-based linear models) complemented each other effectively. Additionally, using Huber loss for neural networks improved robustness against the skewed distribution of psychological scores, where extreme distress is rare but critical to identify.

Performance comparison of individual models and ensembles on development and test sets.

As shown in the results table, the ensemble achieved a substantial boost over single models. On the official test set, our system reached a disattenuated Pearson correlation of 0.5667 for the Total score, securing second place overall. Interestingly, features like spelling mistake ratios and transcription artifacts proved to be predictive, suggesting that the mechanics of writing and the difficulty of transcription may correlate with psychological markers.

Conclusion

Our results demonstrate that combining traditional NLP features with deep learning through a greedy ensemble is a robust strategy for clinical NLP tasks. While the system performed well, future work could explore topic modeling (e.g., LDA) and clinical-specific pre-training for word embeddings to further enhance predictive performance and generalizability across different cohorts.

Summary updated

Abstract

This paper describes the IDLab system submitted to Task A of the CLPsych 2018 shared task. The goal of this task is predicting psychological health of children based on language used in hand-written essays and socio-demographic control variables. Our entry uses word- and character-based features as well as lexicon-based features and features derived from the essays such as the quality of the language. We apply linear models, gradient boosting as well as neural-network based regressors (feed-forward, CNNs and RNNs) to predict scores. We then make ensembles of our best performing models using a weighted average.
BibTeX
@inproceedings{zaporojets2018predicting,
  author = {Klim Zaporojets and Lucas Sterckx and Johannes Deleu and Thomas Demeester and Chris Develder},
  title = {{Predicting Psychological Health from Childhood Essays. The UGent-IDLab CLPsych 2018 Shared Task System}},
  booktitle = {CLPsych @ NAACL},
  year = {2018},
  doi = {10.18653/v1/w18-0613},
}
Figure