Modeling performance differences on cognitive tests using LSTMs and skip-thought vectors trained on reported media consumption.

Maury Courtland, Aida Davani, Melissa Reyes, Leigh Yeh, Jun Leung, Brendan Kennedy, Morteza Dehghani, Jason ZevinNAACL 2019

TL;DR

This paper explores how individual language experiences, derived from reported media consumption, can explain performance differences on cognitive tests. By training language models (LSTMs and skip-thought vectors) on participant-specific media corpora, they found these models could predict performance on a speech perception task, suggesting that personalized language models can offer a more equitable and valid approach to cognitive assessment than standardized methods.

What Problem Does This Paper Address?

Cognitive tests often standardize language materials to ensure fairness and equality. However, this approach overlooks the significant impact of participants' diverse language experiences, which can systematically influence test outcomes and introduce bias. Creating natural, confound-free stimuli is laborious, and existing automated methods often rely on language statistics from corpora that don't reflect individual language use. This mismatch between standardized stimuli and individual experience creates a persistent confound, leading to biased testing. The authors aim to address this by developing a method to normalize test difficulty based on an individual's reported language consumption habits, moving towards a more equitable assessment paradigm.

Methods

The researchers employed two primary neural network-based approaches to model language and predict cognitive test performance:

  • Neural Cloze Model (LSTM-based): They trained vanilla Long Short-Term Memory (LSTM) networks and attention-based LSTMs on participant-specific media corpora. These models learn the joint probability of word sequences, specifically focusing on predicting the final word of a sentence given the preceding words. This "cloze probability" is hypothesized to reflect the predictability of sentence-final words, which are crucial in the cognitive tasks used. The models were trained to capture long-distance dependencies within sentences, an advantage over traditional n-gram models.
  • Skip-thought Vectors: To capture semantic similarity, they used skip-thought vectors, which encode sentences into high-dimensional vectors. These vectors are generated by RNNs (GRUs) and are designed to represent sentence-level semantics such that similar sentences have close vector representations. The distance between a test item's embedding and embeddings of sentences from a participant's corpus was used as a measure of semantic typicality.

Datasets

  • Media Consumption Data: Participants completed an online survey detailing their current and formative media consumption habits across various categories (movies, books, TV, etc.).
  • Corpora: Language data from reported media sources were aggregated into corpora. These corpora were derived from online repositories of television scripts and movie subtitles, including complete series and individual movie subtitles. The data was cleaned and tokenized into sentences for model training.
  • Behavioral Data: Participants completed two cognitive tests:
  • Reading Span Task: Assesses verbal working memory by requiring participants to recall the last word of sentence sets.
  • Speech Perception in Noise (SPiN) Task: Assesses functional hearing by requiring participants to identify spoken words presented amidst background noise.

Key Insights and Ideas

  • Language Experience Matters: The study strongly suggests that an individual's language experience, as reflected in their media consumption, significantly impacts their performance on certain cognitive tasks.
  • Personalized Models for Fairer Assessment: By modeling language based on individual media habits, the researchers demonstrate a path towards creating cognitive assessments that are more equitable and less prone to bias.
  • Predictability vs. Semantic Similarity: The study highlights different ways language models can relate to cognitive performance. Cloze probability (predictability of the last word) showed a positive correlation with performance, while semantic similarity (measured by skip-thought vectors) showed a more complex, inverse relationship.
  • Task-Specific Sensitivity: The models were more successful in predicting performance on the SPiN task than on the reading span task, suggesting that the influence of language experience might be more pronounced or detectable in certain types of cognitive assessments.

Key Results

  • SPiN Task Performance:
    • LSTM Models: Significant rank correlations were found between the cloze probability predictions of both vanilla and attention-based LSTMs and participants' performance on SPiN items for both clusters. For example, for Cluster 2, the vanilla LSTM activation showed a correlation of ρ = .46 (p < .005).
  • Skip-thought Vectors: Significant rank correlations were also observed between the distance metrics (Taxicab and standardized Euclidean) derived from skip-thought vectors and SPiN task performance for both clusters. Interestingly, the direction of this correlation indicated that as the distance from corpus neighbors increased, performance on the item increased.
  • Reading Span Task Performance: No significant rank correlations were found between any of the models' activations or distances and performance on the reading span task for either cluster.

Limitations and Future Work

  • Reading Span Task Limitations: The authors acknowledge that the reading span task's scoring procedure might be problematic for item-level analysis, potentially making it uninformative or invalid compared to the SPiN task.
  • Skip-thought Interpretation: The inverse correlation observed with skip-thought vectors (increased distance leading to better performance) is noted as curious. The authors suggest that this might indicate that "unusual" items, which are semantically distant from typical corpus content, might elicit better performance, possibly due to increased attentional resources. This warrants further investigation.
  • Future Directions: The authors propose testing their models' ability to create test stimuli that are equitable across diverse language communities. They also plan to construct dynamic corpora to mitigate sampling error inherent in static corpora, aiming for more robust and representative language models. The ultimate goal is to develop methods for identifying or synthesizing fair test items and enabling personalized test item creation.

Summary updated

BibTeX
@inproceedings{courtland2019modeling,
  author = {Maury Courtland and Aida Davani and Melissa Reyes and Leigh Yeh and Jun Leung and Brendan Kennedy and Morteza Dehghani and Jason Zevin},
  title = {{Modeling performance differences on cognitive tests using LSTMs and skip-thought vectors trained on reported media consumption.}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2019},
}
Figure