Predicting Psychological Health from Childhood Essays with Convolutional Neural Networks for the CLPsych 2018 Shared Task (Team UKNLP)

Anthony Rios, Tung Tran, Ramakanth KavuluruNAACL 2018

TL;DR

This paper presents two convolutional neural network (CNN) based systems, UKNLPA and UKNLPT, for predicting psychological health from childhood essays. The systems achieved competitive results in the CLPsych 2018 Shared Task, ranking third and fourth respectively, demonstrating the potential of deep learning for analyzing textual data to understand long-term mental well-being.

What Problem Does This Paper Address?

The paper tackles the challenge of predicting long-term psychological health outcomes using early-life textual data. Specifically, it addresses two tasks within the CLPsych 2018 Shared Task:

  1. Task A: Predicting current behavioral health scores (anxiety and depression) at age 11 from essays written by 11-year-olds.
  2. Task B: Predicting future psychological distress at ages 23, 33, 42, and 50, also using the age 11 essays.

This is a significant problem because early identification of potential mental health issues can lead to timely interventions, improving long-term well-being. The challenge lies in extracting meaningful predictive signals from unstructured text written by children, which can be noisy and contain various linguistic idiosyncrasies.

Methods

The authors developed two main systems, UKNLPA and UKNLPT, both based on a Convolutional Neural Network (CNN) architecture.

  • Core CNN Architecture: The CNN processes essays by treating them as sequences of word vectors. Convolution filters are applied to n-grams (sequences of words) to extract features, which are then pooled (max-over-time pooling) to create a fixed-size vector representation of the essay. This is a standard approach for text classification and regression tasks.
  • UKNLPA: This model concatenates the CNN-derived feature vector with auxiliary features: 59 Linguistic Inquiry and Word Count (LIWC) features and the child's gender. This combined vector is then fed into an affine output layer to predict the six psychological health scores. The model is trained using a Huber loss, which is a robust loss function less sensitive to outliers than Mean Squared Error. For Task B, since age 50 ground truth scores were unavailable, the age 42 predictions were used as a proxy. UKNLPA also includes an ensemble of multiple CNNs and a linear model (ridge regression) for improved performance.
  • UKNLPT: This model also uses a CNN architecture but incorporates gender and social class as auxiliary features. A key innovation here is a "switch" mechanism: the network outputs both a regression prediction and a binary classification prediction. The binary output acts as a switch, determining whether the regression output is passed forward. This approach aims to jointly learn to discriminate between zero and non-zero cases, which is relevant given the zero-inflated nature of some psychological distress scores. For Task B, age 50 predictions are made by averaging age 33 and 42 predictions.

Datasets

The primary dataset used is from the National Child Development Study (NCDS).

  • Training Data: 9217 essays written at age 11 and 4235 essays written at age 50. Ground truth training scores were provided for ages 11 (total, anxiety, depression) and 23, 33, and 42 (distress).
  • Testing Data: 1000 essays written at age 11.

The paper does not introduce a new benchmark dataset but rather utilizes an existing one for a shared task.

Key Insights and Ideas

  • Auxiliary Features Matter: Combining CNN features with established linguistic features (LIWC) and demographic information (gender, social class) significantly enhances predictive performance. This highlights that while deep learning can learn complex patterns, incorporating domain-specific knowledge is crucial.
  • Huber Loss Robustness: The use of Huber loss proved beneficial, demonstrating its effectiveness in handling potential outliers in psychological health data, which often exhibit skewed distributions.
  • Zero-Inflated Data Challenges: The paper acknowledges that many psychological health scores are zero-inflated. This observation explains why metrics like MAE might favor models that simply predict zero more often, while DPC (disattenuated Pearson correlation) better captures linear relationships with the ground truth.
  • Ensembling for Performance: The success of UKNLPA as an ensemble of CNNs and a linear model underscores the benefit of combining diverse modeling approaches to capture different aspects of the data.
  • Proxy for Missing Data: Using age 42 predictions as a proxy for age 50 distress scores in Task B is a practical approach to handle missing ground truth, though it introduces a potential source of error.

Key Results

  • Task A (Age 11 Prediction): UKNLPA ranked third among seven teams with a disattenuated Pearson correlation (DPC) score of 0.5587. UKNLPT ranked fourth.
  • Task B (Future Distress Prediction): UKNLPA ranked third with an average DPC score of 0.3062. UKNLPT ranked seventh.
  • The paper also reports results for variants like uk_linear (ridge regression) and uk_cnn (CNN ensemble), showing that simple linear models can be strong baselines and that ensembles like uk_ens2 (combining linear and CNN) can outperform individual models.

Limitations and Future Work

  • Data Cleaning: The authors hypothesize that improving results by implementing more robust data cleaning techniques, such as using a spell checker to correct misspellings and handling illegible words, could be beneficial.
  • Method Specialization: They suggest that since no single method performs best across all psychological health categories, using different methods for each category and developing more sophisticated ensemble approaches could lead to further improvements.
  • Proxy for Age 50 Data: The reliance on age 42 predictions as a proxy for age 50 distress in Task B is a limitation due to the absence of actual age 50 ground truth for training.

Summary updated

BibTeX
@inproceedings{rios2018predicting,
  author = {Anthony Rios and Tung Tran and Ramakanth Kavuluru},
  title = {{Predicting Psychological Health from Childhood Essays with Convolutional Neural Networks for the CLPsych 2018 Shared Task (Team UKNLP)}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2018},
}
Figure