Deep recurrent spiking neural networks capture both static and dynamic representations of the visual cortex under movie stimuli

Liwei Huang, Zhengyu Ma, Huihui Zhou, Yonghong TianarXiv 2023 · arXiv · OpenReview

Abstract

In the real world, visual stimuli received by the biological visual system are predominantly dynamic. Understanding how the visual cortex represents movie stimuli provides insight into information processing mechanisms. We design a recurrent module based on the hierarchy of the mouse cortex and integrate it into Deep Spiking Neural Networks (SNNs). Using Time-Series Representational Similarity Analysis (TSRSA), we measure the similarity between networks and mouse cortical regions under natural movie stimuli. Trained on video action recognition, our recurrent SNN achieves the highest representational similarity, significantly outperforming feedforward SNNs and those trained on image classification. Our work establishes that deep recurrent SNNs are competent to capture both static and dynamic representations, contributing to our understanding of movie information processing in the visual cortex.

Introduction

While Deep Neural Networks (DNNs) are the standard for modeling the visual system, most rely on feedforward architectures where information flows unidirectionally. In contrast, the biological brain is characterized by abundant recurrent and feedback connections. Furthermore, traditional vision research often focuses on static images, whereas biological vision is a continuous stream. Existing models frequently fail to capture the dynamic information—the temporal context—that allows for understanding motion. This research bridges the gap by creating a model that is both biologically plausible, using spiking neurons and structural recurrence.

Methods: LoRaFB-SNet and TSRSA

The researchers introduced the Long-Range Feedback Spiking Network (LoRaFB-SNet). Its architecture follows two core principles:

  1. Spiking Neurons: Utilizing Leaky Integrate-and-Fire (LIF) neurons that communicate via discrete "spikes," mimicking electrochemical signaling.
  2. Long-Range Feedback: A recurrent module sends information from higher stages back to lower stages, implemented via depthwise transposed convolutions to fuse high-level features with low-level inputs.
Architecture of LoRaFB-SNet, showing biological inspiration from mouse cortical regions and the corresponding network stages with long-range feedback loops.

To evaluate the model, the authors developed Time-Series Representational Similarity Analysis (TSRSA). Unlike traditional RSA, which compares responses to independent images, TSRSA measures the correlation of neural responses across time, capturing how sequences of frames influence representations.

Key Results

LoRaFB-SNet achieved the highest representational similarity to the mouse visual cortex across all tested paradigms. It significantly outperformed CORnet (a recurrent non-spiking model) and SEW-ResNet (a feedforward spiking model).

TSRSA scores for Movie 1 and Movie 2, comparing LoRaFB-SNet against CORnet and ResNet-2p-CPC, demonstrating superior brain-matching performance.

Key findings include:

  • Video Training is Essential: Feedback connections only significantly improved brain-similarity when the model was trained on video data (UCF101). When trained on static images (ImageNet), the benefit of feedback loops diminished, suggesting these structures are tuned for temporal "flow."
  • Temporal Accumulation: The model's advantage increases with the length of movie clips, suggesting it is better at accumulating information over time, similar to the biological visual system.
  • Sensitivity to Temporal Order: Shuffling movie frames caused a sharp drop in similarity, proving the model is sensitive to chronological structure rather than just individual frames.

Conclusion

LoRaFB-SNet represents a significant step toward biologically realistic neural modeling. By combining spiking dynamics with long-range feedback and training on temporal data, the model captures the essence of how the mouse visual cortex processes dynamic scenes. Future work will explore more complex feedback mechanisms and whether these findings extend to primate visual systems and higher-level cognitive tasks.

Summary updated

Abstract

In the real world, visual stimuli received by the biological visual system are predominantly dynamic rather than static. A better understanding of how the visual cortex represents movie stimuli could provide deeper insight into the information processing mechanisms of the visual system. Although some progress has been made in modeling neural responses to natural movies with deep neural networks, the visual representations of static and dynamic information under such time-series visual stimuli remain to be further explored. In this work, considering abundant recurrent connections in the mouse visual system, we design a recurrent module based on the hierarchy of the mouse cortex and add it into Deep Spiking Neural Networks, which have been demonstrated to be a more compelling computational model for the visual cortex. Using Time-Series Representational Similarity Analysis, we measure the representational similarity between networks and mouse cortical regions under natural movie stimuli. Subsequently, we conduct a comparison of the representational similarity across recurrent/feedforward networks and image/video training tasks. Trained on the video action recognition task, recurrent SNN achieves the highest representational similarity and significantly outperforms feedforward SNN trained on the same task by 15% and the recurrent SNN trained on the image classification task by 8%. We investigate how static and dynamic representations of SNNs influence the similarity, as a way to explain the importance of these two forms of representations in biological neural coding. Taken together, our work is the first to apply deep recurrent SNNs to model the mouse visual cortex under movie stimuli and we establish that these networks are competent to capture both static and dynamic representations and make contributions to understanding the movie information processing mechanisms of the visual cortex.
BibTeX
@misc{huang2023deep,
  author = {Liwei Huang and Zhengyu Ma and Huihui Zhou and Yonghong Tian},
  title = {{Deep recurrent spiking neural networks capture both static and dynamic representations of the visual cortex under movie stimuli}},
  year = {2023},
  eprint = {2306.01354},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2306.01354},
}
Figure