Predictive Coding is a Consequence of Energy Efficiency in Recurrent Neural Networks

Abdullahi Ali, Nasir Ahmad, E. de Groot, Marcel van Gerven, Tim C. KietzmannSSRN 2021 · DOI

TL;DR

This paper demonstrates that predictive coding, a prominent theory of brain function, can emerge in recurrent neural networks (RNNs) simply by optimizing for energy efficiency. By training RNNs to minimize energy consumption in predictable environments, the networks spontaneously develop prediction and error units, mimicking key aspects of biological predictive coding without explicit architectural constraints.

What Problem Does This Paper Address?

Predictive coding theory posits that the brain constantly predicts incoming sensory information and only processes deviations from these predictions (prediction errors). This is thought to be an efficient way to handle the vast amount of sensory data. While computational models have successfully implemented predictive coding by hard-wiring specific hierarchical and inhibitory connections, a key question remained: could these mechanisms arise naturally from more fundamental principles, rather than being explicitly designed? This paper tackles this by exploring whether energy efficiency, a critical constraint for biological systems, could be the driving force behind the emergence of predictive coding principles in artificial neural networks.

Methods

The authors trained recurrent neural networks (RNNs) to minimize their energy consumption. This energy cost was defined as the sum of action potentials and synaptic transmission, mirroring biological energy expenditure. The RNNs were fed sequences of predictable visual input (MNIST and CIFAR10 digits/images). Instead of explicitly programming predictive coding mechanisms, the networks were trained using an L1 cost function on the preactivations of their units. This objective, combined with the predictable input, encouraged the networks to learn to suppress predictable input, effectively developing internal models to predict upcoming stimuli. They also performed "virtual lesion" experiments, disabling specific units to test their functional role.

Datasets

  • MNIST: A dataset of handwritten digits, used for training and testing the RNNs. Sequences of digits were generated in numerical order.
  • CIFAR10: A dataset of colored images from 10 classes (airplanes, cars, birds, etc.), used to test the generality of the findings. Sequences of images were generated in ascending order of class index.

Key Insights and Ideas

  • Energy efficiency as a driver for prediction: The core insight is that optimizing for energy efficiency in a predictable environment naturally leads to the emergence of predictive coding mechanisms, rather than requiring explicit architectural design.
  • Emergence of prediction and error units: The trained RNNs spontaneously developed distinct populations of units that appear to function as "prediction units" (generating expectations) and "error units" (signaling deviations from expectations), mirroring biological neural organization.
  • Two timescales of prediction: The networks exhibited two distinct modes of prediction: a fast, lateral inhibition among "error" units and a slower, recurrent loop involving "prediction" units that allowed for integration of evidence over time.
  • Functional role of prediction units: Lesioning experiments confirmed that the identified "prediction units" are causally responsible for the network's ability to integrate evidence and improve predictions over time.

Key Results

  • Trained RNNs showed a significant reduction in "energy consumption" (mean absolute preactivation) over time as they processed predictable sequences, converging towards a theoretical lower bound.
  • The networks learned to inhibit predictable sensory input, with the "internal drive" (representing predictions) becoming more targeted and stronger as sequences progressed.
  • Analysis revealed distinct populations of units: those with consistently non-zero median preactivation (potential prediction units) and those with near-zero preactivation (potential error units).
  • Lesioning prediction units severely impaired the network's ability to reduce energy consumption over time and integrate evidence, while lesioning error units had minimal impact, confirming the functional role of prediction units.
  • These results were replicated on the more complex CIFAR10 dataset, demonstrating the generality of the findings.

Limitations and Future Work

  • Proof of concept: The authors acknowledge that the current work, relying on smaller datasets like MNIST and CIFAR10, represents a proof of concept.
  • Future work: Future research could explore larger-scale recurrent systems and their interaction with more natural image statistics. Additionally, investigating energy efficiency in spiking neural network models, which consider membrane voltages and spike times, is suggested. The paper also points to the need to explore how predictive systems optimally process differing spatial and temporal scales and handle unpredictable inputs.

Summary updated

BibTeX
@misc{ali2021predictive,
  author = {Abdullahi Ali and Nasir Ahmad and E. de Groot and Marcel van Gerven and Tim C. Kietzmann},
  title = {{Predictive Coding is a Consequence of Energy Efficiency in Recurrent Neural Networks}},
  year = {2021},
  doi = {10.2139/ssrn.3976481},
  howpublished = {SSRN},
}
Figure