Biasing restricted Boltzmann machines to manipulate latent selectivity and sparsity

Hanlin Goh, Nicolas Thome, Matthieu Cord2010

TL;DR

This paper introduces a novel way to train Restricted Boltzmann Machines (RBMs) by incorporating "inductive biases" that mimic biological neural coding principles like selectivity and sparsity. This allows for more controlled and interpretable latent representations, leading to improved performance on discriminative tasks.

What Problem Does This Paper Address?

Restricted Boltzmann Machines (RBMs) are powerful generative models for learning distributed representations of data. However, standard RBM training, often based on maximum likelihood, doesn't explicitly guide the learned representations to possess specific desirable properties. The authors argue that for many tasks, it's beneficial to incorporate "inductive biases" – prior assumptions about the data or task – into the learning process. Specifically, they focus on two key properties observed in biological neural coding: selectivity (a neuron responding strongly to only a few stimuli) and sparsity (a population of neurons having few active units for a given stimulus). The challenge is to effectively inject these properties into RBMs to improve their representational capabilities and downstream task performance.

Methods

The core contribution is a modification to the RBM learning algorithm that incorporates "latent activation biases." Instead of solely relying on the data to determine hidden unit activations, the authors introduce a target activation pattern, denoted by $P$, which encodes the desired selectivity and sparsity. This bias $P$ is a spatiotemporal matrix where each element $p_{jk}^{(k)}$ represents the desired activation of hidden unit $j$ for input example $k$.

The modification involves adjusting the RBM's energy function and, consequently, the parameter update rules (specifically for weights $W$ and biases $b$). The key idea is to interpolate the original hidden unit activations $h^+$ with the latent activation biases $P$ using a parameter $\phi$. This creates a new effective activation $S$ that is used in the positive phase of the contrastive divergence update. The authors derive new update rules for $W$ and $b$ that incorporate this bias, leading to an asymmetry in the learning signal that guides the RBM towards the desired properties. They also propose a specific method for designing $P$ based on the rank of activations within a lifetime (for selectivity) or population (for sparsity) domain, inspired by biological distributions.

Datasets

  • Kyoto natural images dataset: Used for training RBMs on image patches (14x14 pixels) to visualize learned filters.
  • MNIST handwritten digits database: Used for both training RBMs and evaluating their discriminative performance. The authors note that they used a shallow architecture for evaluation, training RBMs with 1000 hidden units.

Key Insights and Ideas

  • Biasing RBMs for biological plausibility: By incorporating latent activation biases inspired by neural selectivity and sparsity, RBMs can learn more interpretable and potentially more efficient representations.
  • Fine-grained control over latent representations: The proposed method allows for precise manipulation of both individual unit selectivity and population sparsity, offering a level of control not present in standard RBMs.
  • Transfer of properties from bias to parameters: The paper demonstrates that the desired properties encoded in the latent activation biases are successfully transferred to the learned parameters (filters) of the RBM.
  • Synergy of selectivity and sparsity: Combining both selectivity and sparsity is shown to be crucial for reducing the number of overly active or silent units and improving the representation of examples, outperforming biasing for only one property.
  • Improved discriminative performance: Biasing RBMs with selectivity and sparsity leads to significant improvements in classification tasks, suggesting that these properties are beneficial for downstream applications.

Key Results

  • Natural Images: When trained on natural image patches, the biased RBM learns Gabor-like filters, consistent with other sparse coding methods. These filters also showed utility in reconstructing MNIST digits, yielding a 43% decrease in reconstruction error compared to a standard RBM.
  • Handwritten Digits: For MNIST, when the target mean $\mu$ of activation biases was set low, the learned filters resembled localized handwritten strokes.
  • Activity Ratio Analysis: The paper quantifies selectivity and sparsity using an "activity ratio" $\alpha$. They show that as the target mean $\mu$ of activation biases decreases, the activity ratios for both selectivity and sparsity decrease, confirming the transfer of properties. Biasing for both selectivity and sparsity together led to better results than biasing for only one.
  • Discriminative Performance: On MNIST, the biased RBM achieved a significant improvement in classification error rate, outperforming the standard RBM by 29% (a 1.25% absolute improvement in error rate) in a semi-supervised setting.

Limitations and Future Work

  • Limitations:
    • The paper notes that while selectivity and sparsity are often correlated in biology, they are not strictly dependent. Their method aims to control both, but the interplay and potential trade-offs are complex.
  • The authors mention that "grandmother cells" (units that respond to only one specific input) can arise if $\mu$ is too low, leading to poor generalization. This suggests a need for careful tuning of the bias parameters.
  • Future Work:
    • The authors express interest in modeling other latent activation biases inspired by neural phenomena, such as topographical organization and explicitness.
  • They suggest exploring the application of this biasing technique to other deep learning architectures and tasks.

Summary updated

Abstract

This paper proposes a modification to the restricted Boltzmann machine (RBM) learning algorithm to incorporate inductive biases. These latent activation biases are ideal solutions of the latent activity and may be designed either by modeling neural phenomenon or inductive principles of the task. In this paper, we design ac-tivation biases for sparseness and selectivity based on the activation distributions of biological neurons. With this model, one can manipulate the selectivity of indi-vidual hidden units and the sparsity of population codes. The biased RBM yields a filter bank of Gabor-like filters when trained on natural images, while model-ing handwritten digits results in filters with stroke-like features. We quantitatively verify that the latent representations assume the properties of the activation bi-ases. We further demonstrate that RBMs biased with selectivity and sparsity can significantly outperform standard RBMs for discriminative tasks. 1
BibTeX
@misc{goh2010biasing,
  author = {Hanlin Goh and Nicolas Thome and Matthieu Cord},
  title = {{Biasing restricted Boltzmann machines to manipulate latent selectivity and sparsity}},
  year = {2010},
}
Figure