Abstract
Prior to the onset of vision, neurons in the developing mammalian retina spontaneously fire in correlated activity patterns known as retinal waves. Experimental evidence suggests that retinal waves strongly influence the emergence of sensory representations before visual experience. We aim to model this early stage of functional development by using movies of neurally active developing retinas as pre-training data for neural networks. Specifically, we pre-train a ResNet-18 with an unsupervised contrastive learning objective (SimCLR) on both simulated and experimentally-obtained movies of retinal waves, then evaluate its performance on image classification tasks. We find that pre-training on retinal waves significantly improves performance on tasks that test object invariance to spatial translation, while slightly improving performance on more complex tasks like image classification. Notably, these performance boosts are realized on held-out natural images even though the pre-training procedure does not include any natural image data. We then propose a geometrical explanation for the increase in network performance, namely that the spatiotemporal characteristics of retinal waves facilitate the formation of separable feature representations. In particular, we demonstrate that networks pre-trained on retinal waves are more effective at separating image manifolds than randomly initialized networks, especially for manifolds defined by sets of spatial translations. These findings indicate that the broad spatiotemporal properties of retinal waves prepare networks for higher order feature extraction.
Introduction
In deep learning, the prevailing assumption is that models require vast quantities of labeled data, such as ImageNet, to develop robust object recognition capabilities. However, biological systems challenge this notion; newborn mammals can perform fundamental visual tasks almost immediately after birth, suggesting that the brain is not a "blank slate" at the onset of visual experience.
During late-stage gestation and early postnatal development, the mammalian retina generates spontaneous, correlated activity patterns called retinal waves. While these waves are known to guide the physical wiring of the visual system (e.g., retinotopic mapping), their role in shaping the functional "software"—the internal representations required for object recognition—remains less understood. This study investigates whether this innate activity provides a structured pre-training signal that "bootstraps" the geometry of the visual system, preparing it for the complex spatial properties of the natural world.
Methods
The researchers employed a ResNet-18 architecture trained via SimCLR, a self-supervised contrastive learning algorithm. The core of the approach was to replace standard natural image pre-training with movies of retinal waves.
Data and Training
The pre-training data consisted of two sources:
- Biological Data: Calcium imaging movies of whole-mount retinas from postnatal mice (P8–P11).
- Simulated Data: A reaction-diffusion model mimicking cholinergic retinal wave propagation.
The SimCLR objective tasked the network with "compressing" temporally consecutive frames into similar representations while distinguishing them from temporally distant frames. To isolate the importance of the waves' inherent structure, the researchers compared performance against three control datasets: spatially shuffled (permuted pixels), temporally shuffled (permuted frames), and spatiotemporally shuffled (both).
Evaluation
Following pre-training, the hidden layers of the ResNet-18 were frozen. A linear readout layer was then trained on three specific tasks using CIFAR-10 and CIFAR-100:
- Standard Classification: Identifying object categories.
- Spatial Translation: Recognizing an object regardless of its position in the frame.
- Color Change: Recognizing an object despite shifts in color.
Results
Innate Translation Invariance
The most significant finding was that retinal waves serve as a natural teacher for spatial translation invariance. Because waves propagate across the retina, they provide a temporal signal that links shifted versions of similar activity patterns. Networks pre-trained on unshuffled retinal waves (both real and simulated) achieved 85-90% accuracy on translation tasks, significantly outperforming randomly initialized networks (~70%) and all shuffled controls.
Untangling Image Manifolds
Using manifold analysis, the authors demonstrated that retinal wave pre-training effectively "untangles" data representations. In randomly initialized networks, different translations of the same object are often tangled in high-dimensional space. Pre-training on waves increases manifold capacity, making it easier for a linear classifier to separate distinct objects. This was evidenced by a measurable decrease in manifold radius and dimension for translated images.
Generalization to Natural Images
Remarkably, although the network never encountered natural images during pre-training, the features learned from blob-like retinal waves generalized to natural objects. This suggests that the spatiotemporal "rules" governing retinal waves share a fundamental geometry with the natural visual environment. The performance boost was lost when temporal order was shuffled, confirming that the sequential movement of waves is the critical information source for learning spatial invariance.
Discussion
This work suggests that the brain uses spontaneous activity to prepare its internal representation geometry before the first photon hits the retina. While retinal waves do not provide semantic information (e.g., the difference between a cat and a dog), they establish a spatial grid that facilitates subsequent learning.
Limitations and Future Directions
The study utilizes standard deep learning architectures (ResNet-18) and objectives (SimCLR) which, while effective, are not strictly biologically plausible. Future research could explore whether more brain-like architectures, such as spiking neural networks or local learning rules like Spike-Timing-Dependent Plasticity (STDP), yield similar geometric bootstrapping. Additionally, investigating spontaneous activity in other sensory modalities, such as the auditory system, could reveal if this is a universal principle of neural development.
Summary updated