Overcoming the bottleneck in traditional assessments of verbal memory: Modeling human ratings and classifying clinical group membership

Chelsea Chandler, Peter W. Foltz, Jian Cheng, Jared C. Bernstein, Elizabeth P. Rosenfeld, Alex S. Cohen, Terje B. Holmlund, Brita ElvevågNAACL 2019

TL;DR

This paper tackles the bottleneck in traditional verbal memory assessments, which rely on time-consuming human review. They developed an automated pipeline using speech recognition and NLP to transcribe, analyze, and score story recalls, achieving human-level accuracy in predicting recall quality and distinguishing between healthy individuals and psychiatric patients.

What Problem Does This Paper Address?

Assessing verbal memory is crucial for understanding neurocognitive function, especially in conditions like dementia, aphasia, autism, and mental illness. However, traditional methods, like the Wechsler Memory Scale's Logical Memory subtest, are hampered by a significant bottleneck: they require expert human review for scoring. This process is time-consuming, limits the frequency of assessments, and restricts their application, especially in large-scale or remote settings. Furthermore, traditional tests often have limited versions, further hindering frequent use. This paper aims to overcome this bottleneck by automating the entire process, from administration and transcription to analysis and scoring, making verbal memory assessment more accessible, frequent, and objective.

Methods

The authors developed a multi-stage computational pipeline:

  • Automated Administration & Transcription: Participants interacted with a mobile app to hear stories and then retell them. The audio recordings were transcribed using Automatic Speech Recognition (ASR). They compared two ASR systems: Google's Speech API and a custom Deep Neural Network-Hidden Markov Model.
  • Feature Engineering: A range of linguistic features were extracted from the transcribed retells. These included:
  • Surface Features: Word counts, unique word counts, n-grams, and part-of-speech tags.
  • Semantic Features: They leveraged pre-trained word embeddings (specifically word2vec) to capture meaning. Key metrics included:
  • Cosine Distance: A standard measure of similarity between vector representations.
  • Word Mover's Distance (WMD): A more sophisticated metric that measures the "distance" between two documents by finding the optimal mapping of words from one to the other, capturing semantic similarity more effectively than cosine distance.
  • Cohesion/Coherence: Measures of how well the retell flows and stays on topic.
  • Human Rating Prediction: A ridge regression model was trained to predict human expert ratings of story recall quality. They identified three key features (common types, word type count, and WMD) that best predicted these ratings.
  • Clinical Group Classification: The same features were used in an ensemble of logistic regression classifiers to distinguish between psychiatric patients and healthy participants. This ensemble approach allowed for predictions even when participants didn't complete all parts of the task.

Datasets

The study utilized data collected from:

  • 105 stable patients with mental illness from a substance use treatment program.
  • 120 undergraduate students from Louisiana State University, presumed to be healthy.

Participants interacted with a mobile application that administered verbal memory tasks. The stories used were structurally similar to those in the Wechsler Memory Scale-III, with 24 different stories developed to allow for frequent administration.

Key Insights and Ideas

  • WMD is a powerful semantic similarity metric for recall: The Word Mover's Distance proved to be a highly effective feature for capturing semantic similarity in retells, outperforming simpler metrics like cosine distance and correlating strongly with human ratings. This highlights the importance of semantic understanding over exact word matching.
  • Automated assessment can rival human performance: The automated pipeline achieved a correlation of 0.88 with human ratings for predicting recall quality, very close to the inter-rater reliability of human raters (0.92). This suggests that automated systems can be as reliable and objective as human experts.
  • ASR is robust enough for clinical prediction: Even with a significant word error rate (10-20%), the automated pipeline's performance in predicting ratings and classifying groups only slightly decreased compared to using human transcriptions. This demonstrates the viability of using ASR in real-world, potentially noisy environments.
  • Ensemble classifiers handle incomplete data well: The use of an ensemble of classifiers allowed the system to make accurate predictions even when participants provided only immediate or delayed retells, or missed responses, which is crucial for real-world data collection.
  • "Fooling" the model is a limitation: The regression model could be "fooled" by participants who rambled about a topic (e.g., balloons) without recalling key details, indicating that while semantic features are powerful, they might not capture all nuances of recall quality.

Key Results

  • Human Rating Prediction: The ridge regression model achieved an average correlation of 0.88 with human ratings for story recall quality over 10-fold cross-validation.
  • Clinical Group Classification: The ensemble classifier achieved an overall accuracy of 76% in distinguishing between psychiatric patients and healthy participants.
  • Model accuracy: 76%
    • Precision: 73%
    • Recall: 80%
    • F1 score: 76%
  • ASR Performance:
    • Google ASR: Average word error rate of 20.90% (26.51% for patients, 16.38% for healthy).
  • Custom ASR: Average word error rate of 10.79% (13.36% for patients, 5.90% for healthy).
  • ASR Pipeline Performance:
    • Regression model correlation with human ratings using ASR transcripts: 0.86 (Google ASR) and 0.87 (custom ASR).
  • Classification model accuracy using ASR transcripts: 74% for both ASR systems.

Limitations and Future Work

  • "Fooling" the Regression Model: The authors acknowledge that their regression model could be fooled by participants who provide lengthy but irrelevant retells, indicating a need for more nuanced feature extraction to capture true recall quality.
  • Clinical and Behavioral Implications: While the study validates the metrics and pipeline, establishing their clinical and behavioral implications requires further complex empirical work, including modeling multiple dynamic data streams and correctly calibrating new metrics.
  • Semi-Immediate Feedback: The authors suggest that future studies should "close the triage" by providing semi-immediate feedback from the assessment to relevant entities.
  • Longitudinal Tracking: The study's design was geared towards longitudinal tracking, and future work could further explore how these automated assessments can be used for continuous monitoring.

Summary updated

BibTeX
@inproceedings{chandler2019overcoming,
  author = {Chelsea Chandler and Peter W. Foltz and Jian Cheng and Jared C. Bernstein and Elizabeth P. Rosenfeld and Alex S. Cohen and Terje B. Holmlund and Brita Elvevåg},
  title = {{Overcoming the bottleneck in traditional assessments of verbal memory: Modeling human ratings and classifying clinical group membership}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2019},
}
Figure