Network Plasticity as Bayesian Inference

David Kappel, Stefan Habenschuss, Robert Legenstein, Wolfgang Maass

TL;DR

This paper proposes a new framework for understanding how neural networks learn and adapt, suggesting that plasticity should be viewed as Bayesian inference rather than just maximizing likelihood. This "synaptic sampling" approach explains observed biological phenomena like trial-to-trial variability and offers a more robust way for networks to generalize and adapt to changing environments.

What Problem Does This Paper Address?

Traditional models of neural network learning, particularly in neuroscience, often assume that synaptic plasticity aims to optimize network parameters to maximize the likelihood of observed data. This is analogous to maximum likelihood (ML) learning in machine learning. However, this ML approach faces several challenges:

  • Generalization: ML models can overfit to training data, leading to poor performance on new, unseen data.
  • Robustness: Biological neural networks are constantly exposed to internal and external perturbations (e.g., noise, lesions). They need to maintain stable function and adapt quickly.
  • Biological Plausibility: Experimental data shows significant trial-to-trial variability in synaptic parameters (e.g., spine volume fluctuations) and structural changes (e.g., spine formation/depletion) that are difficult to reconcile with deterministic ML learning. These phenomena are often treated as noise or artifacts rather than functional components.

The authors aim to bridge this gap by proposing a normative model that explains these biological observations and offers a more theoretically sound and robust learning mechanism.

Methods

The core of the paper introduces a new learning framework called synaptic sampling. Instead of converging to a single optimal parameter value (like in ML), synaptic sampling proposes that network parameters sample from a posterior distribution. This posterior distribution is defined by both the prior knowledge (structural constraints, biological rules) and the likelihood of the observed data.

Key aspects of the method include:

  • Bayesian Inference: The goal is to learn the posterior distribution $p^*(\theta|x)$ of network parameters $\theta$ given inputs $x$. This is achieved by combining a prior $p_s(\theta)$ (encoding structural rules) with the likelihood $p_N(x|\theta)$.
  • Stochastic Differential Equations (SDEs): The dynamics of network parameters are described by SDEs. The proposed rule (Eq. 3) has three components:
  • A drift term that pushes parameters towards regions of high posterior probability (related to ML).
  • A prior term that incorporates structural constraints.
  • A stochastic diffusion term (driven by a Wiener process) that allows parameters to explore the parameter space and sample from the posterior.
  • "Synaptic Sampling": This term refers to the process where network parameters (e.g., synaptic weights) fluctuate stochastically, allowing the network to sample from the posterior distribution. This explains observed trial-to-trial variability as a functional aspect of learning.
  • Online Learning: The framework is extended to an online learning setting where updates occur sequentially based on individual inputs, making it more biologically plausible and computationally efficient.
  • Integration of Structural and Synaptic Plasticity: The model naturally incorporates both changes in synaptic weights (synaptic plasticity) and changes in connectivity (structural plasticity, e.g., spine formation/depletion) within a single unified framework.

Datasets

The paper uses standard, publicly available datasets for evaluation:

  • MNIST Dataset: Used for training and testing handwritten digit recognition tasks with Restricted Boltzmann Machines (RBMs) and spike-based Winner-Take-All (WTA) circuits.
  • TI 46 Dataset: Used for spoken digit recognition with cochleagram representations.

No new benchmark datasets are introduced.

Key Insights and Ideas

  • Trial-to-Trial Variability is Functional: The paper reframes biological variability in synaptic parameters not as noise, but as a crucial mechanism for sampling from a posterior distribution, enabling robust learning and generalization.
  • Bayesian Inference as the Goal: Instead of maximizing likelihood, neural networks learn by approximating a posterior distribution over their parameters. This naturally incorporates prior knowledge and leads to better generalization.
  • Unified Model for Structural and Synaptic Plasticity: The synaptic sampling framework provides a single, principled rule that governs both changes in synaptic strength and the formation/elimination of connections.
  • Inherent Robustness and Compensation: The continuous stochastic sampling process makes networks inherently more resilient to perturbations (like lesions) and allows them to automatically compensate for changes in input statistics.
  • Priors Matter: The model highlights the importance of priors in shaping network structure and function, explaining observed biological constraints like sparse connectivity and heavy-tailed weight distributions.

Key Results

  • Improved Generalization: Simulations with RBMs show that synaptic sampling with appropriate priors significantly improves generalization performance on unseen data compared to standard maximum likelihood learning, preventing overfitting.
  • Adaptation to Changing Input Statistics: The model demonstrates that networks can adapt their structure and synaptic weights to changing input distributions, exhibiting network rewiring that maintains functionality.
  • Compensation for Perturbations: Experiments with spike-based networks show that the synaptic sampling mechanism allows networks to recover performance after significant lesions (removal of neurons or connections), showcasing inherent compensation capabilities.
  • Biological Plausibility: The model provides a theoretical explanation for various experimental observations, including dendritic spine dynamics, trial-to-trial variability in synaptic efficacy, and the emergence of sparse connectivity.

Limitations and Future Work

  • Parameter Tuning: While the framework is general, specific implementations might still require careful tuning of parameters like the learning rate $b$ and temperature $T$.
  • Molecular Implementation of Priors: The paper suggests that priors might be implemented through gene regulation or other molecular mechanisms, but this remains an open area for experimental investigation.
  • Consolidation of Synaptic Weights: The authors propose that synaptic weight consolidation could be modeled as a modification of the prior, which is an attractive avenue for future research.
  • Non-local Learning Rules: The current model primarily focuses on local plasticity rules. Investigating whether more complex, non-local learning rules can be approximated by synaptic sampling is an open question.
  • Experimental Verification of Priors: Further experimental work is needed to directly measure and understand the nature and role of priors in biological neural networks.

Summary updated

Abstract

General results from statistical learning theory suggest to understand not only brain computations, but also brain plasticity as probabilistic inference. But a model for that has been missing. We propose that inherently stochastic features of synaptic plasticity and spine motility enable cortical networks of neurons to carry out probabilistic inference by sampling from a posterior distribution of network configurations. This model provides a viable alternative to existing models that propose convergence of parameters to maximum likelihood values. It explains how priors on weight distributions and connection probabilities can be merged optimally with learned experience, how cortical networks can generalize learned information so well to novel experiences, and how they can compensate continuously for unforeseen disturbances of the network. The resulting new theory of network plasticity explains from a functional perspective a number of experimental data on stochastic aspects of synaptic plasticity that previously appeared to be quite puzzling.
BibTeX
@article{kappel2015network,
  author = {David Kappel and Stefan Habenschuss and Robert Legenstein and Wolfgang Maass},
  title = {{Network Plasticity as Bayesian Inference}},
  journal = {PLOS Computational Biology},
  year = {2015},
  doi = {10.1371/journal.pcbi.1004485},
}
Figure