TL;DR
This paper demonstrates how traveling waves, a fundamental neural activity pattern, can emerge in artificial neural networks through biologically plausible learning rules, rather than being explicitly programmed. These emergent waves are shown to be crucial for robust working memory, even in the presence of distractions, and their dynamics closely mirror those observed in primate brains.
What Problem Does This Paper Address?
Artificial neural networks (ANNs), despite their impressive performance, often lack a key characteristic of biological brains: structured spatiotemporal dynamics like traveling waves (TWs). These waves are ubiquitous in the cortex and are thought to play vital roles in information processing, perception, and memory. Previous attempts to model TWs in ANNs have relied on hand-crafted connectivity or imposed dynamics, limiting our understanding of how these waves naturally arise in biological circuits. This paper tackles the question of whether TWs can emerge organically within a trainable recurrent neural network (RNN) under biologically realistic constraints.
Methods
The researchers developed a data-driven framework to induce traveling waves in RNNs. Key methods include:
- Spatially Embedded RNNs: Units were arranged on a 2D sheet, mimicking the cortical surface, with synaptic strengths decaying with inter-neuronal distance. This enforces a "local connectivity" prior.
- Manifold Alignment: The RNN's latent activity space was aligned with the neural manifold observed in non-human primate (NHP) prefrontal cortex (PFC) recordings during a working memory task. This constraint guides the network's learning towards biologically plausible dynamics.
- Forward-Biased Parameterization: To explicitly test the role of directional connectivity, a novel parameterization was introduced that embeds a latent feed-forward structure within the recurrent weight matrix, while still respecting spatial locality.
- Optical Flow Analysis: This technique was used to quantify wave propagation by estimating the instantaneous velocity field of neural activity across the 2D network.
- Working Memory Task (DMS): The trained networks were evaluated on a delayed match-to-sample task, including a distractor stimulus, to assess their ability to maintain information and handle interference.
- Neural Decoding and Selectivity Analysis: Time-resolved measures were used to track how sample identity and distractor information were represented and propagated through the network and in NHP recordings.
Datasets
- Non-human primate (NHP) prefrontal cortex (PFC) recordings: Used to derive the empirical neural manifold for the manifold alignment constraint.
- Simulated data: Generated by the trained RNN models for evaluation on the DMS task.
Key Insights and Ideas
- Emergence over Imposition: Traveling waves can emerge naturally in RNNs through learning, driven by biologically plausible constraints, rather than needing to be explicitly programmed.
- Manifold Alignment is Key: Aligning the network's latent dynamics to empirical neural data is crucial for inducing persistent traveling waves during tasks like working memory.
- Directionality Matters: The emergence of traveling waves is strongly linked to a shift from reciprocal (bidirectional) connectivity to more unidirectional, feed-forward motifs.
- Segregated Pathways for Memory and Distraction: Traveling waves provide a mechanism for maintaining working memory by routing information along distinct, stable pathways, preventing interference from distractors.
Key Results
- Sustained Traveling Waves: The "Manifold-Aligned" RNN, constrained by NHP PFC data, exhibited persistent traveling waves during the working memory delay period, unlike a baseline "Locally Connected" RNN where wave strength decayed.
- Forward-Biased Connectivity: The Manifold-Aligned network developed a connectivity structure characterized by more unidirectional connections and feed-forward loops, which was further validated by the "Forward-Biased" architecture.
- Robust Working Memory: The Forward-Biased RNN maintained sample information during a distractor, with information propagating along distinct, temporally ordered sequences for the sample and the distractor.
- Biological Plausibility: The temporal cascade of information propagation observed in the model during distraction was mirrored in NHP PFC recordings, supporting the biological relevance of the findings.
- Rank Stability: Both the model and NHP data showed stable ordering of neurons for processing sample and distractor information across different delay lengths.
Limitations and Future Work
- Hand-crafted anatomical prior: While the model incorporates distance-dependent connectivity, the initial spatial arrangement of units is still a form of prior. Future work could explore learning this spatial embedding.
- Simplifications in the model: The model uses a leaky continuous-time rate RNN, which is a simplification of biological neurons. Exploring more biologically detailed neuron models could reveal further insights.
- Generalizability: While the DMS task is a good benchmark, testing the emergent traveling wave dynamics on a wider range of cognitive tasks would be valuable.
- Mechanistic understanding: The paper provides a framework for understanding how traveling waves support cognition. Further research could delve deeper into the specific computational roles of these waves in different cognitive functions.
Summary updated