Abstract
In the real world, visual stimuli received by the biological visual system are predominantly dynamic. Understanding how the visual cortex represents movie stimuli provides insight into information processing mechanisms. We design a recurrent module based on the hierarchy of the mouse cortex and integrate it into Deep Spiking Neural Networks (SNNs). Using Time-Series Representational Similarity Analysis (TSRSA), we measure the similarity between networks and mouse cortical regions under natural movie stimuli. Trained on video action recognition, our recurrent SNN achieves the highest representational similarity, significantly outperforming feedforward SNNs and those trained on image classification. Our work establishes that deep recurrent SNNs are competent to capture both static and dynamic representations, contributing to our understanding of movie information processing in the visual cortex.
Introduction
While Deep Neural Networks (DNNs) are the standard for modeling the visual system, most rely on feedforward architectures where information flows unidirectionally. In contrast, the biological brain is characterized by abundant recurrent and feedback connections. Furthermore, traditional vision research often focuses on static images, whereas biological vision is a continuous stream. Existing models frequently fail to capture the dynamic information—the temporal context—that allows for understanding motion. This research bridges the gap by creating a model that is both biologically plausible, using spiking neurons and structural recurrence.
Methods: LoRaFB-SNet and TSRSA
The researchers introduced the Long-Range Feedback Spiking Network (LoRaFB-SNet). Its architecture follows two core principles:
- Spiking Neurons: Utilizing Leaky Integrate-and-Fire (LIF) neurons that communicate via discrete "spikes," mimicking electrochemical signaling.
- Long-Range Feedback: A recurrent module sends information from higher stages back to lower stages, implemented via depthwise transposed convolutions to fuse high-level features with low-level inputs.
To evaluate the model, the authors developed Time-Series Representational Similarity Analysis (TSRSA). Unlike traditional RSA, which compares responses to independent images, TSRSA measures the correlation of neural responses across time, capturing how sequences of frames influence representations.
Key Results
LoRaFB-SNet achieved the highest representational similarity to the mouse visual cortex across all tested paradigms. It significantly outperformed CORnet (a recurrent non-spiking model) and SEW-ResNet (a feedforward spiking model).
Key findings include:
- Video Training is Essential: Feedback connections only significantly improved brain-similarity when the model was trained on video data (UCF101). When trained on static images (ImageNet), the benefit of feedback loops diminished, suggesting these structures are tuned for temporal "flow."
- Temporal Accumulation: The model's advantage increases with the length of movie clips, suggesting it is better at accumulating information over time, similar to the biological visual system.
- Sensitivity to Temporal Order: Shuffling movie frames caused a sharp drop in similarity, proving the model is sensitive to chronological structure rather than just individual frames.
Conclusion
LoRaFB-SNet represents a significant step toward biologically realistic neural modeling. By combining spiking dynamics with long-range feedback and training on temporal data, the model captures the essence of how the mouse visual cortex processes dynamic scenes. Future work will explore more complex feedback mechanisms and whether these findings extend to primate visual systems and higher-level cognitive tasks.
Summary updated