TL;DR
This paper introduces a novel deep predictive coding network trained with biologically plausible Hebbian learning. The model successfully replicates emergent neural properties observed in the visual cortex, such as increasing selectivity and sparseness along the hierarchy, without explicit programming of these features. This work offers a computational framework for understanding how complex perception arises from unsupervised learning in deep neural architectures.
What Problem Does This Paper Address?
Understanding how the brain constructs our perception of the world is a fundamental challenge. Traditional deep learning models often rely on supervised learning with massive labeled datasets, which is biologically implausible. The brain learns from unlabeled sensory input through complex, hierarchical processing. While predictive coding offers a promising computational framework for modeling this, previous models have been limited in scale and biological plausibility, often hand-crafted and lacking emergent properties. This paper addresses the need for a scalable, biologically plausible deep predictive coding model that can learn unsupervisedly and exhibit emergent neural response properties seen in the visual cortex.
Methods
The authors developed a Deep Hebbian Predictive Coding Network (DHPC). Key methodological aspects include:
- Deep Architecture: The network is structured into multiple hierarchical layers, mimicking cortical areas.
- Predictive Coding Framework: Each layer infers causes of sensory input and generates predictions for lower layers, with errors propagated back up the hierarchy.
- Unsupervised Hebbian Learning: Instead of backpropagation, the network uses a Hebbian learning rule for updating synaptic strengths, which is more biologically plausible.
- Emergent Properties: The model is designed to learn without explicit constraints for specific neural properties (like orientation selectivity). These properties are expected to emerge naturally from the learning process.
- Receptive Fields: The model incorporates receptive fields that increase in size with hierarchical depth, mirroring biological visual processing.
- Inference and Learning Steps: The model employs distinct inference (estimating causes) and learning (updating synaptic weights) steps, both utilizing locally available information.
Datasets
The model was trained using 2000 images from the CIFAR-10 dataset, specifically focusing on airplanes and automobiles. These images were used as sensory input. The evaluation also involved using novel stimuli from these classes and other object classes not seen during training.
Key Insights and Ideas
- Biologically Plausible Unsupervised Learning: The DHPC network demonstrates that complex neural properties can emerge from simple, biologically plausible learning rules (Hebbian learning) and unsupervised processing, without needing explicit programming.
- Hierarchical Emergence: Neural properties like image selectivity and sparseness systematically increase with hierarchical depth, mirroring observations in the biological visual system. This suggests a fundamental principle of how information is processed and represented across cortical areas.
- Reconstruction Fidelity: The model can reconstruct sensory input from inferred causes, with lower areas providing sharper reconstructions. This highlights the network's ability to capture and represent sensory information effectively.
- Object Classification Generalization: Representations learned in higher areas enable better generalization for object classification, even for unfamiliar exemplars, suggesting that higher-level representations capture more abstract and robust features.
Key Results
- Emergent Orientation Selectivity: Neurons in the lowest area (Area 1) of the model with receptive fields exhibited orientation selectivity, similar to V1 neurons, which emerged without being explicitly programmed.
- Increasing Image Selectivity: Mean image selectivity (measured by kurtosis) increased systematically from lower to higher areas, with statistical significance (p < 5.10–15).
- Increasing Sparseness: Average sparseness also increased systematically from lower to higher areas, with high statistical significance (p < 5.10–34).
- Object Classification Performance: A Support Vector Machine (SVM) classifier trained on representations from higher areas achieved significantly better classification accuracy for distinguishing object classes (airplanes vs. cars) compared to classifiers trained on lower-area representations. For example, Area 4 representations showed significantly higher performance than other areas.
- Negative Correlation between Selectivity and Activity: Neurons with high selectivity tended to have low average activity, a finding consistent with experimental data.
Limitations and Future Work
- Anatomical Specificity: The current DHPC networks do not explicitly model different cell types and their specific locations within neocortical layers, which is a direction for future research to bridge theoretical models with biological reality.
- Ergodicity: While there's a close relationship between selectivity and sparseness, the model's average selectivity and sparseness did not perfectly satisfy the condition for weak ergodicity. Further exploration of hyperparameter space is suggested.
- Temporal Dynamics: The paper notes that incorporating recurrent representational units to capture temporal dynamics of input would be an interesting direction for future research.
- Generalizability of Sparseness Findings: The authors suggest that inconsistent results regarding sparseness in experimental data might be explained by the complex interplay of factors like regularization and hierarchical position, and that findings from studies comparing only a few brain regions may not generalize broadly.
Summary updated