For years, the gold standard for computationally modeling the biological visual system was the deep convolutional neural network trained on massive labeled datasets like ImageNet. Early benchmarking demonstrated that supervised models captured the visuo-semantic organization of the primate inferior temporal (IT) cortex far better than hand-engineered alternatives (Khaligh-Razavi). By mapping the layers of deep networks to human fMRI data, researchers revealed a striking hierarchical complexity gradient that closely mirrors the ventral stream (van Gerven et al.).
Yet, this success introduces a profound biological paradox. Human infants do not develop their visual systems by being fed millions of explicit category labels. Biological learning is fundamentally unsupervised. This discrepancy creates a critical theoretical gap: if the brain lacks a global teacher signal, how does it arrive at internal representations that look so much like those engineered via supervised learning?
A growing body of research suggests the answer lies in self-supervised learning (SSL). By optimizing for intrinsic statistical regularities within natural sensory streams, self-supervised networks naturally untangle complex environmental data, yielding internal structures that align remarkably well with biological brains.
The Intuition of Self-Supervision: Exploiting Natural Regularities
The core insight behind self-supervised learning is that the physical world is not random; it contains rich, structured regularities that can act as a proxy for supervision. Instead of relying on external semantic labels, a system can use one part of the sensory input to predict another.
Consider the property of temporal coherence—the physical reality that the world changes slowly and continuously. Objects do not instantly transform; they move smoothly across our field of view. Becker demonstrated that by maximizing the mutual information between a network's outputs at consecutive time steps, a model can learn to group different viewpoints of the same object together without any external labels.
Extending this to the spatial domain, Becker et al. showed that maximizing mutual information across different spatial regions allows a network to discover coherent underlying structures like depth, even while adapting to sudden surface discontinuities like object edges.
[Sensory Stream] ---> Spatial & Temporal Coherence ---> [Internal Representation]
(Continuous) (Maximize Mutual Info) (Untangled Manifolds)
Beyond passive observation, an organism's own actions and multi-sensory experiences provide powerful self-supervised signals:
- Egomotion: Carreira showed that an agent can learn robust visual features simply by predicting how its visual input changes as a function of its own physical movement through an environment.
- Multimodal Correlation: The correlation between different sensory modalities can drive category discovery. Coen demonstrated that a system can cross-reference auditory speech and visual lip movements to spontaneously discover phonetic vowel categories without explicit linguistic instruction.
- Spontaneous Internal Activity: Even before birth, biological systems leverage internal regularities. Ligeralde et al. showed that unsupervised pre-training on spontaneous, non-natural retinal waves establishes an efficient representational geometry that prepares the visual cortex for downstream tasks before the eyes ever open.
Why Self-Supervision Replicates Brain Geometry
Why do these label-free objectives yield representations that align so closely with the biological brain? The answer lies in the concept of manifold untangling. The brain's task is to take highly complex, non-linear sensory inputs (like pixels or sound waves) and separate them into useful, linearly separable concepts (like distinguishing objects or tracking motion). Self-supervised objectives exert a computational pressure that naturally mirrors this biological untangling.
Recent large-scale benchmarking has confirmed this alignment across species and modalities. Conwell et al. evaluated over 110 neural network architectures against thousands of neurons in the rodent visual cortex, finding that self-supervised models predict biological neural responses nearly as well as category-supervised ones, successfully recapitulating the brain's hierarchy.
In humans, Konkle et al. demonstrated that self-supervised frameworks optimizing for view-invariance predict ventral stream activity across the entire visual cortex with the same fidelity as models trained on millions of human-curated labels. This functional specialization can even emerge organically by training parallel pathways with self-supervised predictive learning (The functional specialization of visual cortex emerges from training parallel pathways with self-supervised predictive learning).
This alignment is not restricted to vision. In the auditory domain, a study on speech processing (Self-Supervised Models of Audio Effectively Explain Human Cortical Responses to Speech) revealed that self-supervised models naturally develop an internal hierarchy—from raw acoustics to high-level semantics—that maps cleanly onto the human auditory cortex.
Furthermore, Ozcelik et al. showed that features extracted from contrastive self-supervised networks align so tightly with human neural activity that they can be used to reconstruct highly realistic natural images directly from fMRI scans. As Doerig et al. argue within the "neuroconnectionist" framework, these findings strongly imply that the functional organization of the brain emerges not from hardcoded semantic templates, but from autonomous optimization against the statistical structure of the environment.
Grounding Self-Supervision in Biological Constraints
While modern self-supervised algorithms achieve impressive neural alignment, they often rely on non-biological mechanisms. For example, contrastive learning models frequently depend on artificial data augmentations (like extreme color jittering) or complex negative sampling. As Kataoka et al. point out, the next frontier requires identifying learning objectives that achieve high-level semantic organization using only naturalistic visual shifts and temporal predictions.
To achieve true biological plausibility, researchers are increasingly constraining these models with realistic data streams and local learning rules:
- Naturalistic Data Streams: Orhan et al. addressed the data gap by training self-supervised models on raw, unmanipulated head-camera footage from infants, proving that generic learning mechanisms can extract sophisticated visual categories from the messy, continuous sensory stream of real life.
- Local Learning Rules: The brain operates via discrete spikes and local synaptic plasticity rather than global backpropagation. To bridge this gap, Wang developed unsupervised Spiking Convolutional Neural Networks (SCNNs) that successfully predict fMRI responses using local learning rules.
- Cellular Heterogeneity: Chakraborty et al. demonstrated that when recurrent spiking networks are endowed with cellular heterogeneity, they naturally self-organize to produce internal data manifolds that are topologically identical to those generated by globally optimized supervised networks.
By combining self-supervised objectives with biologically realistic constraints, cognitive science is uncovering how local, label-free interactions give rise to the global functional specialization of the mind.
Key Foundational Work
Learning to Categorize Objects Using Temporal Coherence (1992) — contributed 1 concept elements to this direction.
Learning to Make Coherent Predictions in Domains with Discontinuities (1991) — contributed 1 concept elements to this direction.