Adaptive stretching of representations across brain regions and deep learning model layers

Xin-Ya Zhang, Sebastian Bobadilla-Suarez, Xiaoliang Luo, Marilena Lemonari, Scott L. Brincat, Markus Siegel, Earl K. Miller, Bradley C. LovebioRxiv 2023 · DOI

TL;DR

This paper investigates how the brain and deep learning models adapt their internal representations to focus on task-relevant information. They found that both monkeys and a CNN-LSTM model "stretch" their representations along the relevant dimension (color or motion) to improve performance, suggesting this is a fundamental adaptive strategy for learning.

What Problem Does This Paper Address?

The prefrontal cortex (PFC) is known to influence visual processing to prioritize goal-relevant information. However, it's unclear if the brain broadly reconfigures its entire visual system to achieve this, or if certain areas remain specialized. This paper explores whether visual representations are "stretched" along task-relevant dimensions across various brain regions and deep learning model layers. This is crucial for understanding how attention and learning mechanisms work, especially in systems that learn through trial-and-error, like non-human animals and artificial neural networks.

Methods

The researchers used a combination of neurophysiological recordings from monkeys and a deep learning model (CNN-LSTM) trained on the same task.

  • Monkey Task: Monkeys performed a visual categorization task where they had to attend to either color or motion direction based on a trial-by-trial cue.
  • Deep Learning Model: A CNN-LSTM model was trained on the same visual stimuli and reward structure as the monkeys. The CNN processed visual input, and the stacked LSTM simulated processing across multiple brain regions.
  • Representational Similarity Analysis (RSA): This technique was used to compare how stimuli were represented in the brain and the model. They constructed dissimilarity matrices (RDMs) to quantify these representations.
  • Spike Train Dissimilarity Measures: They evaluated various measures (rate coding, ISI distance, SPIKE distance) to find the best way to capture neural similarity, finding that spike timing measures (ISI and SPIKE) were superior.
  • Cognitive Model Fitting: A cognitive model was used to estimate the "attention" allocated to each dimension, quantifying representational stretching.

Datasets

The study used data from a previously published experiment [17] involving two rhesus monkeys performing a visual categorization task. The same visual stimuli (colored, moving dots) were used to train the CNN-LSTM model.

Key Insights and Ideas

  • Universal "Stretching" Mechanism: The finding that both biological brains and artificial neural networks exhibit "dimensional stretching" suggests this is a fundamental principle of adaptive learning and representation optimization, not unique to biological systems.
  • Spike Timing is Crucial: The study highlights the importance of spike timing, not just firing rate, in neural coding. This offers a more nuanced understanding of how information is processed in the brain.
  • Modularity vs. Flexibility: While some brain regions (MT for motion, V4 for color) appear to be modality-bound, others (PFC, FEF, LIP, IT) and the LSTM model show remarkable flexibility, reconfiguring their representations based on task demands. This points to a trade-off between specialized processing and general adaptability.
  • Learning Through Error Minimization: The fact that the ANN model, without explicit attention mechanisms, learned to stretch representations simply by minimizing errors, underscores how task performance optimization can drive the emergence of complex cognitive strategies.

Key Results

  • Brain Regions: Lateral PFC, FEF, LIP, and IT showed significant stretching of representations along task-relevant dimensions. MT and V4, however, were found to be modality-bound, prioritizing their preferred dimension regardless of the task.
  • Deep Learning Model: The CNN-LSTM model exhibited task-relevant stretching across its LSTM layers, mirroring the brain's adaptive behavior. More advanced layers showed greater attention to the relevant dimension.
  • Spike Timing: ISI and SPIKE distance measures were significantly better at capturing the experimenter-intended stimulus coordinates compared to rate-based measures.
  • Model Performance: The CNN-LSTM model achieved high accuracy (over 98%) on the task, comparable to or exceeding the monkeys' performance.

Limitations and Future Work

  • Modality-Bound Regions: The study identified MT and V4 as modality-bound. Future work could explore the neural mechanisms underlying this specialization and how it interacts with more flexible regions.
  • ANN vs. Brain Differences: While the ANN showed impressive task-relevant stretching, it was more flexible than the brain, not exhibiting modality-bound layers. This suggests biological systems may have inherent constraints or benefits for localized processing.
  • Spike Timing in Models: The current ANN model used rate-based coding for its units. Future research should explore computational models that explicitly incorporate spike timing to better understand its functional role in decision-making.
  • Generalizability: The study's findings are based on a specific task. Further research is needed to determine if this "stretching" phenomenon generalizes to other tasks and cognitive domains.

Summary updated

Abstract

Abstract Prefrontal cortex (PFC) is known to modulate the visual system to favor goal-relevant information by accentuating task-relevant stimulus dimensions. Does the brain broadly re-configures itself to optimize performance by stretching visual representations along task-relevant dimensions? We considered a task that required monkeys to selectively attend on a trial-by-trial basis to one of two dimensions (color or motion direction) to make a decision. Except for V4 (color bound) and MT (motion bound), the brain radically re-configured itself to stretch representations along task-relevant dimensions in lateral PFC, frontal eye fields (FEF), lateral intraparietal cortex (LIP), and inferotemporal cortex (IT). Spike timing was crucial to this code. A deep learning model was trained on the same visual input and rewards as the monkeys. Despite lacking an explicit selective attention or other control mechanism, the model displayed task-relevant stretching as a consequence of error minimization, indicating that stretching is an adaptive strategy.
BibTeX
@misc{zhang2023adaptive,
  author = {Xin-Ya Zhang and Sebastian Bobadilla-Suarez and Xiaoliang Luo and Marilena Lemonari and Scott L. Brincat and Markus Siegel and Earl K. Miller and Bradley C. Love},
  title = {{Adaptive stretching of representations across brain regions and deep learning model layers}},
  year = {2023},
  doi = {10.1101/2023.12.01.569615},
  howpublished = {bioRxiv},
}
Figure