An Approach to the CLPsych 2018 Shared Task Using Top-Down Text Representation and Simple Bottom-Up Model Selection

Micah Iserman, Molly Ireland, Andrew Littlefield, Tyler Davis, Sage MaliepaardNAACL 2018

TL;DR

This paper explores predicting mental health outcomes from 11-year-old children's essays about their future lives. The authors found that simple, theory-driven dictionary-based text analysis, combined with data-driven model selection, yielded interpretable results, particularly for future distress, even if overall accuracy was modest.

What Problem Does This Paper Address?

The CLPsych 2018 Shared Task aimed to predict current and future mental health (anxiety, depression, general distress) from childhood essays. This is a challenging problem because mental health is complex and can change over time, making early prediction difficult. The authors highlight the need for methods that are not only accurate but also interpretable for clinicians, allowing for practical application in understanding and potentially intervening with children's well-being. They aim to bridge the gap between computational linguistics and clinical psychology by exploring how linguistic features in children's writing can serve as signals for psychological states.

Methods

The core of their approach involved a combination of "top-down" and "bottom-up" methods.

  • Top-Down Text Representation: They primarily used theory-driven dictionaries, most notably the Linguistic Inquiry and Word Count (LIWC) dictionaries. These dictionaries categorize words based on psychological constructs (e.g., emotion, cognitive processes). They also incorporated Latent Dirichlet Allocation (LDA) to identify latent topics within the text.
  • Simple Bottom-Up Model Selection: After extracting linguistic features, they employed a data-driven approach to select the best predictors for each outcome. This involved calculating correlations between features and outcomes and including variables with a correlation above a certain threshold in linear mixed-effects regression models. They also implemented an edit-distance based approach to account for misspellings in the text, aiming to improve dictionary matching.

Datasets

  • National Child Development Study (NCDS): The primary dataset used was a subsample from the NCDS, specifically essays written by 11-year-old children about their imagined lives at age 25. This dataset was prepared for the CLPsych Shared Task 2018.
  • CLPsych Shared Task 2018 Benchmark: The paper implicitly contributes to the benchmark established by the CLPsych Shared Task, providing their results and methodology for comparison with other participating teams.

Key Insights and Ideas

  • Interpretability is Key for Clinical Application: The authors emphasize that for computational linguistics to be useful in clinical settings, the methods and results must be transparent and interpretable. This guided their preference for dictionary-based approaches over more opaque "black box" models.
  • Misspellings as a Signal: Surprisingly, the percentage of words captured by dictionaries before automated spelling correction (i.e., including misspellings) was a robust predictor of distress. This suggests that general psychological distress might manifest as difficulties with academic performance or cognitive load, impacting spelling accuracy.
  • Future Focus vs. Present Tense: The prediction that focusing on the future (using future tense) would be negatively related to distress was partially supported, suggesting that adherence to the writing prompt might be linked to better psychological states.
  • Word Count as a Predictor: Similar to other research, they found that children who wrote fewer words in their essays tended to have more severe behavioral and psychological symptoms. This aligns with theories suggesting inhibition and less expressive language can be linked to distress.
  • Conjunctions and Academic Engagement: The positive correlation between conjunctions and psychological distress (except for anxiety) suggests that a more "dynamic" or conversational writing style might be associated with academic disengagement or conformity issues, which in turn relate to distress.

Key Results

  • Modest Out-of-Sample Accuracy: While the approach yielded interpretable findings, the out-of-sample accuracy for predictions was modest.
  • Better Future Predictions: Their models performed better for predicting future psychological distress (ages 23, 33, and 42) compared to concurrent (age 11) predictions.
  • Misspellings and Distress: The percentage of words captured by dictionaries before text cleaning was a significant predictor of less anxiety and depression at age 11 and less general distress at ages 23 and 33.
  • Word Count and Distress: Fewer words written correlated with more severe behavioral and psychological symptoms.
  • Conjunctions and Distress: Conjunctions were positively correlated with psychological distress (except anxiety).
  • Sex Differences: For age 23 distress, their model predicted psychological distress for women better than for men. For age 11 BSAG total scores, the model performed equally well for both sexes.

Limitations and Future Work

  • Modest Effect Sizes: The authors acknowledge that the effect sizes of their predictors were generally small, partly due to the low base-rates of distress in the NCDS sample. This limits the clinical utility of their findings, as it could lead to a high number of false positives.
  • Limited Predictive Power for Age 42 Distress: They struggled to identify robust predictors for psychological distress at age 42, attributing this to the increasing difficulty of predicting mental states over longer time spans.
  • Future Work: They suggest that future approaches could benefit from more seamlessly integrating top-down (theory-driven) and bottom-up (data-driven) methods. They also highlight the potential of improved theory-driven dictionaries using data-driven methods (like distributed representations) to enhance computational linguists' ability to provide actionable insights for clinicians. They also propose finer-grained approaches to measuring temporal orientation and prospection.

Summary updated

BibTeX
@inproceedings{iserman2018approach,
  author = {Micah Iserman and Molly Ireland and Andrew Littlefield and Tyler Davis and Sage Maliepaard},
  title = {{An Approach to the CLPsych 2018 Shared Task Using Top-Down Text Representation and Simple Bottom-Up Model Selection}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2018},
}
Figure