Neural Implementation of Hierarchical Bayesian Inference by Importance Sampling

Lei Shi, Thomas L. GriffithsNeurIPS 2009 · OpenReview

The "Bayesian Brain" hypothesis suggests that our minds are remarkably adept at dealing with uncertainty by combining prior knowledge with noisy sensory data. However, there is a massive gap between the elegant mathematics of Bayesian inference and the messy reality of biological neurons. Standard Bayesian models often require calculating integrals over high-dimensional spaces—a computationally expensive task that is difficult to implement in biological wetware.

Previous theories, such as "Bayesian decoding," suggested neurons encode probability distributions in their firing patterns, but these often require complex, biologically questionable operations to extract the result. This paper asks: Is there a simpler, more "natural" way for neurons to perform these calculations? The authors propose that the brain performs complex Bayesian inference not through complicated mathematical decoding, but through a simple "importance sampling" mechanism.

The Importance Sampling Mechanism

The authors propose a mechanism based on Importance Sampling, a classic Monte Carlo method used in computer science and statistics. Instead of calculating a full probability distribution, the system takes samples from a "surrogate" distribution (the prior) and weights them by how well they explain the current data (the likelihood).

This mathematical concept maps elegantly onto neural architecture:

  • Priors as Population Density: The "prior" is physically encoded by the number of neurons tuned to a specific stimulus. For example, if vertical lines are common in nature, more neurons are dedicated to detecting them.
  • Likelihood as Firing Rate: When a stimulus appears, a neuron's firing rate is determined by how closely the stimulus matches its "preferred" feature (its tuning curve).
  • Normalization: Inhibitory neurons provide "divisive normalization," effectively dividing the activity of individual neurons by the total population activity.
Neural architecture for importance sampling

Because Radial Basis Function (RBF) neurons only fire when a stimulus is close to their preference, the model naturally produces "sparse" activity. This is highly energy-efficient and consistent with actual brain observations. Furthermore, the authors extend this to Hierarchical Bayesian Inference. By stacking these layers, the output of one "importance sampler" becomes the input for the next, allowing the brain to process increasingly abstract variables (e.g., moving from simple edges in visual cortex area V1 to complex objects in V4). They also demonstrate that this math holds up even when using Poisson spiking neurons, which are more biologically realistic than continuous-rate models.

Validating the Model: Key Results

The paper validates its model against two landmark human psychophysical findings, demonstrating that a relatively small number of neurons (as few as 20-40 per population) are sufficient to produce a close match to human behavioral data.

Haptic-Visual Cue Combination

The model was tested against classic data regarding how humans integrate touch (haptic) and sight to estimate the height of a bar. The model successfully replicated human behavior: when visual noise increased, the model automatically shifted its "weight" toward haptic information, perfectly matching the statistically optimal Bayesian prediction.

Visual-haptic cue combination experiment and model simulation

The Oblique Effect

Humans are known to be more sensitive to horizontal and vertical orientations than diagonal ones (the "oblique effect"). By setting the population density of neurons in the model to match the "prior" of the natural world—where horizontal and vertical lines are statistically more frequent—the model naturally exhibited higher sensitivity at these cardinal orientations.

Limitations and Future Directions

While the model offers a compelling and simplified view of Bayesian inference in the brain, it primarily explains how the system functions once set up. It does not detail the learning rules (such as synaptic plasticity) required to ensure neuron densities and weights match environmental statistics over time. Additionally, the current model focuses on static inference. Future work could explore how this sampling approach handles tracking stimuli that change rapidly over time, potentially utilizing methods like Particle Filtering. Finally, while the authors point to interneurons and shunting inhibition as candidates for the normalization step, the exact biological circuit remains a hypothesis to be tested by electrophysiologists.

Summary updated

Abstract

The goal of perception is to infer the hidden states in the hierarchical process by which sensory data are generated. Human behavior is consistent with the optimal statistical solution to this problem in many tasks, including cue combination and orientation detection. Understanding the neural mechanisms underlying this behavior is of particular importance, since probabilistic computations are notoriously challenging. Here we propose a simple mechanism for Bayesian inference which involves averaging over a few feature detection neurons which fire at a rate determined by their similarity to a sensory stimulus. This mechanism is based on a Monte Carlo method known as importance sampling, commonly used in computer science and statistics. Moreover, a simple extension to recursive importance sampling can be used to perform hierarchical Bayesian inference. We identify a scheme for implementing importance sampling with spiking neurons, and show that this scheme can account for human behavior in cue combination and oblique effect.
BibTeX
@inproceedings{shi2009neural,
  author = {Lei Shi and Thomas L. Griffiths},
  title = {{Neural Implementation of Hierarchical Bayesian Inference by Importance Sampling}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2009},
  url = {https://openreview.net/forum?id=euv_cmWC74c},
}
Figure