Deep predictive coding accounts for emergence of complex neural response properties along the visual cortical hierarchy

Shirin Dora, Sander M. Bohté, Cyriel M. A. PennartzbioRxiv 2020 · DOI

TL;DR

This paper introduces a novel deep predictive coding network trained with biologically plausible Hebbian learning. The model successfully replicates emergent neural properties observed in the visual cortex, such as increasing selectivity and sparseness along the hierarchy, without explicit programming of these features. This work offers a computational framework for understanding how complex perception arises from unsupervised learning in deep neural architectures.

What Problem Does This Paper Address?

Understanding how the brain constructs our perception of the world is a fundamental challenge. Traditional deep learning models often rely on supervised learning with massive labeled datasets, which is biologically implausible. The brain learns from unlabeled sensory input through complex, hierarchical processing. While predictive coding offers a promising computational framework for modeling this, previous models have been limited in scale and biological plausibility, often hand-crafted and lacking emergent properties. This paper addresses the need for a scalable, biologically plausible deep predictive coding model that can learn unsupervisedly and exhibit emergent neural response properties seen in the visual cortex.

Methods

The authors developed a Deep Hebbian Predictive Coding Network (DHPC). Key methodological aspects include:

  • Deep Architecture: The network is structured into multiple hierarchical layers, mimicking cortical areas.
  • Predictive Coding Framework: Each layer infers causes of sensory input and generates predictions for lower layers, with errors propagated back up the hierarchy.
  • Unsupervised Hebbian Learning: Instead of backpropagation, the network uses a Hebbian learning rule for updating synaptic strengths, which is more biologically plausible.
  • Emergent Properties: The model is designed to learn without explicit constraints for specific neural properties (like orientation selectivity). These properties are expected to emerge naturally from the learning process.
  • Receptive Fields: The model incorporates receptive fields that increase in size with hierarchical depth, mirroring biological visual processing.
  • Inference and Learning Steps: The model employs distinct inference (estimating causes) and learning (updating synaptic weights) steps, both utilizing locally available information.

Datasets

The model was trained using 2000 images from the CIFAR-10 dataset, specifically focusing on airplanes and automobiles. These images were used as sensory input. The evaluation also involved using novel stimuli from these classes and other object classes not seen during training.

Key Insights and Ideas

  • Biologically Plausible Unsupervised Learning: The DHPC network demonstrates that complex neural properties can emerge from simple, biologically plausible learning rules (Hebbian learning) and unsupervised processing, without needing explicit programming.
  • Hierarchical Emergence: Neural properties like image selectivity and sparseness systematically increase with hierarchical depth, mirroring observations in the biological visual system. This suggests a fundamental principle of how information is processed and represented across cortical areas.
  • Reconstruction Fidelity: The model can reconstruct sensory input from inferred causes, with lower areas providing sharper reconstructions. This highlights the network's ability to capture and represent sensory information effectively.
  • Object Classification Generalization: Representations learned in higher areas enable better generalization for object classification, even for unfamiliar exemplars, suggesting that higher-level representations capture more abstract and robust features.

Key Results

  • Emergent Orientation Selectivity: Neurons in the lowest area (Area 1) of the model with receptive fields exhibited orientation selectivity, similar to V1 neurons, which emerged without being explicitly programmed.
  • Increasing Image Selectivity: Mean image selectivity (measured by kurtosis) increased systematically from lower to higher areas, with statistical significance (p < 5.10–15).
  • Increasing Sparseness: Average sparseness also increased systematically from lower to higher areas, with high statistical significance (p < 5.10–34).
  • Object Classification Performance: A Support Vector Machine (SVM) classifier trained on representations from higher areas achieved significantly better classification accuracy for distinguishing object classes (airplanes vs. cars) compared to classifiers trained on lower-area representations. For example, Area 4 representations showed significantly higher performance than other areas.
  • Negative Correlation between Selectivity and Activity: Neurons with high selectivity tended to have low average activity, a finding consistent with experimental data.

Limitations and Future Work

  • Anatomical Specificity: The current DHPC networks do not explicitly model different cell types and their specific locations within neocortical layers, which is a direction for future research to bridge theoretical models with biological reality.
  • Ergodicity: While there's a close relationship between selectivity and sparseness, the model's average selectivity and sparseness did not perfectly satisfy the condition for weak ergodicity. Further exploration of hyperparameter space is suggested.
  • Temporal Dynamics: The paper notes that incorporating recurrent representational units to capture temporal dynamics of input would be an interesting direction for future research.
  • Generalizability of Sparseness Findings: The authors suggest that inconsistent results regarding sparseness in experimental data might be explained by the complex interplay of factors like regularization and hierarchical position, and that findings from studies comparing only a few brain regions may not generalize broadly.

Summary updated

Abstract

Abstract Predictive coding provides a computational paradigm for modelling perceptual processing as the construction of representations accounting for causes of sensory inputs. Here, we develop a scalable, deep predictive coding network that is trained using a Hebbian learning rule. Without a priori constraints that would force model neurons to respond like biological neurons, the model exhibits properties similar to those reported in experimental studies. We analyze low- and high-level properties such as orientation selectivity, object selectivity and sparseness of neuronal populations in the model. As reported experimentally, image selectivity increases systematically across ascending areas in the model hierarchy. A further emergent network property is that representations for different object classes become more distinguishable from lower to higher areas. Thus, deep predictive coding networks can be effectively trained using biologically plausible principles and exhibit emergent properties that have been experimentally identified along the visual cortical hierarchy. Significance Statement Understanding brain mechanisms of perception requires a computational approach based on neurobiological principles. Many deep learning architectures are trained by supervised learning from large sets of labeled data, whereas biological brains must learn from unlabeled sensory inputs. We developed a Predictive Coding methodology for building scalable networks that mimic deep sensory cortical hierarchies, perform inference on the causes of sensory inputs and are trained by unsupervised, Hebbian learning. The network models are well-behaved in that they faithfully reproduce visual images based on high-level, latent representations. When ascending the sensory hierarchy, we find increasing image selectivity, sparseness and generalizability for object classification. These models show how a complex neuronal phenomenology emerges from biologically plausible, deep networks for unsupervised perceptual representation.
BibTeX
@misc{dora2020deep,
  author = {Shirin Dora and Sander M. Bohté and Cyriel M. A. Pennartz},
  title = {{Deep predictive coding accounts for emergence of complex neural response properties along the visual cortical hierarchy}},
  year = {2020},
  doi = {10.1101/2020.02.07.937292},
  howpublished = {bioRxiv},
}
Figure