CDRNN: Discovering Complex Dynamics in Human Language Processing

Cory ShainACL 2021

TL;DR

This paper introduces CDRNN, a novel deep learning model that extends existing methods for analyzing human language processing. CDRNN can capture complex, time-varying, and non-linear relationships between linguistic input and cognitive responses, offering a more flexible and interpretable approach than previous techniques.

What Problem Does This Paper Address?

Understanding how the human mind processes language incrementally is a central question in psycholinguistics. Traditional methods often rely on regression models that struggle to account for the dynamic and complex nature of cognitive processes. Specifically, these methods often fail to capture:

  • Delayed effects: The influence of a word on our understanding might not be immediate but can manifest with a delay.
  • Non-linear relationships: The effect of a linguistic feature (like word surprisal) might not be a simple linear increase or decrease.
  • Time-varying influences: The way a word affects processing might change over time.
  • Interactions between factors: Multiple linguistic features might interact in complex ways to influence processing.

While deep neural networks (DNNs) excel at capturing complex patterns, they are often criticized for being "black boxes," making it difficult to extract scientific insights. This paper aims to bridge this gap by developing a DNN-based model that is both powerful and interpretable for psycholinguistic analysis.

Methods

The core innovation is the Continuous-time Deconvolutional Regressive Neural Network (CDRNN). CDRNN builds upon Continuous-time Deconvolutional Regression (CDR) by reimplementing the impulse response function (IRF) as a deep neural network.

  • Deep Neural IRF: Instead of fitting parametric functions, CDRNN uses a neural network to learn the IRF. This allows it to capture non-linear and time-varying influences of predictors on responses.
  • Relaxed Assumptions: CDRNN relaxes several simplifying assumptions of CDR, including linearity, additivity, homoskedasticity (constant variance), stationarity (time-invariant effects), and context-independence.
  • Recurrent Component (Optional): CDRNN can optionally incorporate a recurrent neural network (RNN) to model how the context of preceding words influences the current processing dynamics.
  • Distributional Regression: CDRNN models not just the mean of the response (e.g., reading time) but also its variance, providing a more complete picture of the processing dynamics.
  • Perturbation Analysis for Interpretation: To interpret the model, the authors use perturbation analysis. This involves systematically changing input features and observing the model's output to understand the effect of specific predictors.

Datasets

The study uses existing, publicly available datasets for training and evaluation:

  • Natural Stories Corpus: A self-paced reading (SPR) corpus (~1M instances).
  • Dundee Corpus: An eye-tracking corpus (~200K instances).
  • Natural Stories fMRI Corpus: An fMRI dataset (~200K instances).

These datasets were used with train/dev/test splits defined in prior work.

Key Insights and Ideas

  • DNNs for Psycholinguistics: CDRNN demonstrates that deep learning can be effectively adapted for psycholinguistic research, offering significant advantages over traditional regression models.
  • Interpretable Dynamics: The model's ability to estimate IRFs allows researchers to visualize and understand the temporal dynamics of cognitive processes, revealing patterns that were previously hidden.
  • Modality-Specific Processing: CDRNN reveals differences in how language is processed across modalities (e.g., reading vs. fMRI), suggesting that the underlying cognitive dynamics are not identical.
  • Beyond Linearity: The study highlights that assuming linear relationships can be overly restrictive, especially for fMRI data where saturation effects are prominent. CDRNN's flexibility allows it to capture these non-linearities.
  • Inertia Effects in Reading: CDRNN supports the interpretation of "rate effects" in reading as an "inertia effect," where faster reading in the recent past leads to faster current reading, even when accounting for non-linear habituation.

Key Results

  • Improved Predictive Performance: CDRNN models consistently outperformed baseline models (LME, GAM, CDR) in terms of mean squared error across reading and fMRI datasets. For example, CDRNN-RNN achieved significantly lower error than all baselines on fMRI data.
  • Generalization: The model's estimates generalized well to unseen data, supporting the trustworthiness of the learned IRFs.
  • Capturing Non-Linearity: CDRNN revealed strong non-linearities in fMRI responses to surprisal, which were not captured by linear models or even CDR.
  • Modality Differences: CDRNN showed that surprisal effects decay more quickly in eye-tracking than in reading, and that fMRI responses to surprisal are more attenuated and show different temporal dynamics compared to reading.
  • Interaction Effects: The model could explore interactions between predictors, revealing that factors like sound power have a relatively flat influence on the response, suggesting that the brain's language system is less sensitive to low-level perceptual effects.

Limitations and Future Work

  • Computational Cost: While CDRNN offers flexibility, the optional RNN component can increase computational cost, especially when dealing with asynchronously measured predictors.
  • Interpretability of Individual Parameters: Like other DNNs, individual weights and biases in CDRNN are not directly interpretable. The authors rely on perturbation analysis for understanding predictor effects.
  • Future Work: The authors suggest that CDRNN could be used to explore a wider range of linguistic phenomena and cognitive dynamics. Further investigation into the variance IRFs and their relationship to mean IRFs across different modalities is also a promising direction. The flexibility of CDRNN opens up new avenues for testing hypotheses about the functional form and temporal dynamics of language processing.

Summary updated

BibTeX
@inproceedings{shain2021cdrnn,
  author = {Cory Shain},
  title = {{CDRNN: Discovering Complex Dynamics in Human Language Processing}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2021},
}
Figure