A Deep Predictive Coding Network for Learning Latent Representations

Shirin Dora, Cyriel M. A. Pennartz, Sander M. BohtébioRxiv 2018 · DOI

TL;DR

This paper presents a novel deep neural network architecture inspired by predictive coding principles in the brain. It demonstrates how this network can learn hierarchical latent representations from real-world images in an unsupervised manner, showing promising generalization capabilities even for unseen object categories.

What Problem Does This Paper Address?

The brain is thought to operate as a prediction machine, constantly trying to anticipate incoming sensory information. Predictive coding is a leading theory for how this might be achieved, suggesting that the brain builds internal models to predict stimuli and uses prediction errors to update these models. While this theory is compelling, a significant challenge has been translating it into practical deep neural network architectures that are both biologically plausible and effective for learning complex representations. Specifically, it's unclear how to build deep networks that adhere to the brain's architectural constraints (like local receptive fields) while still learning rich, hierarchical representations. This paper aims to bridge this gap by proposing a predictive coding-based algorithm for training deep neural networks that respects these biological constraints.

Methods

The authors propose a deep generative model based on predictive coding. The network architecture is inspired by Convolutional Neural Networks (CNNs) but with a key difference: neurons in a layer only project to a local receptive field in the layer below, mimicking the retinotopic arrangement of receptive fields in sensory cortices. This local connectivity allows for independent learning of filters across different receptive fields.

The learning algorithm is an implementation of predictive coding. For a given input image, latent representations are learned at each layer such that they can predict the representations in the layer below. This process is iterative, with prediction errors driving updates to both latent representations and synaptic weights (filters). A unique aspect is the parallel update of latent representations and filters within each layer, which is enabled by the local connectivity and the predictive coding framework. The algorithm alternates between updating latent representations and then updating filters, a strategy designed to improve convergence speed.

Datasets

The primary dataset used for training and evaluation is a subset of the CIFAR-10 dataset, specifically 1000 images of horses and ships. The paper also mentions using images of other objects (like frogs, cars, trucks, sparrows) that were not part of the training set to evaluate the model's generalization capabilities.

Key Insights and Ideas

  • Biologically Plausible Deep Learning: The paper demonstrates a concrete way to build deep neural networks that are more aligned with biological constraints (local connectivity, hierarchical processing) than standard CNNs, using predictive coding as the learning mechanism.
  • Unsupervised Learning of Hierarchical Representations: The network learns to infer latent representations at multiple levels of abstraction without explicit labels, suggesting that predictive coding can be a powerful unsupervised learning paradigm.
  • Generalization through Local Regularities: The local receptive field architecture allows the network to capture granular, common regularities in images at lower layers. These regularities are then combined in higher layers to form more abstract representations, leading to good generalization even for novel object categories.
  • Parallelization Potential: The predictive coding framework, combined with the local architecture, allows for significant parallelization in both feedforward and feedback processing, as well as in updating latent representations and filters, potentially speeding up training.

Key Results

  • Image Reconstruction: The trained generative model can reconstruct original images from inferred latent representations. While the reconstructions are noted to be somewhat blurry (attributed to mean squared error loss), they preserve important details of the objects.
  • Translation Invariance: The model shows a degree of translation invariance, meaning it can infer reasonable latent representations for images where objects have been shifted.
  • Generalization to Unseen Objects: Crucially, the model demonstrates the ability to infer latent representations for images of object categories (e.g., frogs, cars, trucks) that were not present in the training set. This suggests the learned representations capture fundamental visual statistics rather than just memorizing training examples.

Limitations and Future Work

  • Reconstruction Blurriness: The paper acknowledges that reconstructions are blurry, suggesting that using an L1-norm loss might yield visually better results.
  • Computational Implications: While the algorithm is amenable to parallelization, the paper briefly touches upon computational implications and its relationship to other machine learning approaches, suggesting further analysis is warranted.
  • Future Directions: The authors imply that this approach can be extended to other modalities beyond vision and can be scaled to model more complex brain processes like attention and memory. The architecture's ability to capture granular regularities is highlighted as a key factor for generalization, suggesting further exploration of how these regularities are learned and utilized.

Summary updated

Abstract

Abstract It has been argued that the brain is a prediction machine that continuously learns how to make better predictions about the stimuli received from the external environment. It builds a model of the world around us and uses this model to infer the external stimulus. Predictive coding has been proposed as a mechanism through which the brain might be able to build such a model of the external environment. However, it is not clear how predictive coding can be used to build deep neural network models of the brain while complying with the architectural constraints imposed by the brain. In this paper, we describe an algorithm to build a deep generative model using predictive coding that can be used to infer latent representations about the stimuli received from external environment. Specifically, we used predictive coding to train a deep neural network on real-world images in a unsupervised learning paradigm. To understand the capacity of the network with regards to modeling the external environment, we studied the latent representations generated by the model on images of objects that are never presented to the model during training. Despite the novel features of these objects the model is able to infer the latent representations for them. Furthermore, the reconstructions of the original images obtained from these latent representations preserve the important details of these objects.
BibTeX
@misc{dora2018deep,
  author = {Shirin Dora and Cyriel M. A. Pennartz and Sander M. Bohté},
  title = {{A Deep Predictive Coding Network for Learning Latent Representations}},
  year = {2018},
  doi = {10.1101/278218},
  howpublished = {bioRxiv},
}
Figure