Diversity priors for learning early visual features

Hanchen Xiong, Antonio Jose Rodríguez-Sánchez, Sándor Szedmák, Justus H. Piater

Abstract

Traditional models of the primary visual cortex (V1) often prioritize sparsity and selectivity to replicate biological neural activity. However, these models frequently suffer from redundancy and "dead neurons." This paper proposes that sparsity and selectivity are emergent properties of a more fundamental principle: diversity. By implementing a "diversity prior" in a Restricted Boltzmann Machine (RBM), the authors demonstrate the development of Gabor-like visual features that align with biological data while ensuring full utilization of the neural population.

Introduction

In the study of the visual cortex, two properties of "simple cells" in area V1 are legendary: selectivity and sparsity. Selectivity refers to a neuron's specific response to particular stimuli (e.g., a vertical line at a specific location), while sparsity describes the phenomenon where only a tiny fraction of the total neuron population is active at any given moment.

While classic models like Independent Component Analysis (ICA) and Sparse Coding have tried to optimize for these properties directly, they often struggle to achieve both simultaneously. Furthermore, in neural networks like RBMs, neurons are "conditionally independent," meaning they do not coordinate which features they learn. This often results in redundant features or "dead neurons" that never respond to stimuli. The authors argue that instead of optimizing for sparsity or selectivity directly, we should force neurons to be diverse.

Methods and Data

The researchers utilized a Restricted Boltzmann Machine (RBM) and introduced a novel Diversity Prior during the learning phase. This prior works by minimizing the square cosine similarity between the weight vectors (receptive fields) of the hidden units, pushing the learned "basis images" as far apart as possible in mathematical space. Biologically, this mimics the effect of lateral inhibition, where active neurons suppress their neighbors to sharpen signal contrast.

Training was conducted using the Olshausen and Field (1996) benchmark database of natural images. These images were "whitened" to mimic retinal processing and broken into 100,000 small $14 \times 14$ pixel patches. To validate the results, the learned features were compared against physiological data from macaque monkey V1 neurons.

Key Insights: Diversity as a Primary Driver

The most striking insight of this work is that population sparsity and individual selectivity emerge naturally as "epiphenomena" when neurons are forced to represent diverse, non-overlapping features. Even though the RBM architecture lacks physical lateral connections, the diversity prior acts as a "virtual inhibitory interneuron" during training.

Conceptual framework of sparsity, selectivity, and population diversity across different coding regimes.

By penalizing similarity, the model ensures that the "real estate" of possible visual features is covered efficiently. This prevents multiple neurons from wasting energy learning the exact same edge orientation, leading to a more robust and biologically plausible representation.

Key Results

  • Biological Fidelity: The learned receptive fields were Gabor-like filters that showed a high degree of similarity to actual macaque V1 data in terms of shape profiles and aspect ratios.
  • Eliminating "Dead" Neurons: In a population of 200 neurons, the Diverse RBM had 0 dead neurons. In contrast, a standard "Sparse RBM" had 68 dead neurons, and a "Selective RBM" had 88. This proves that diversity is a much more robust way to ensure every part of the model is doing useful work.
  • Generative Success: The model could reconstruct test images with high qualitative accuracy, proving the learned features captured the essential statistics of natural scenes.

Limitations and Future Work

The authors acknowledge that an RBM with a diversity prior is not a complete biological model, as it lacks features like contrast normalization and temporal adaptation. Future work could explore how diversity priors behave in deep, multi-layer stacks (Deep Belief Networks) or apply these priors to video data to see if "temporal diversity" leads to the discovery of motion-sensitive features similar to those in area V5/MT.

Summary updated

Abstract

This paper investigates how utilizing diversity priors can discover early visual features that resemble their biological counterparts. The study is mainly motivated by the sparsity and selectivity of activations of visual neurons in area V1. Most previous work on computational modeling emphasizes selectivity or sparsity independently. However, we argue that selectivity and sparsity are just two epiphenomena of the diversity of receptive fields, which has been rarely exploited in learning. In this paper, to verify our hypothesis, restricted Boltzmann machines (RBMs) are employed to learn early visual features by modeling the statistics of natural images. Considering RBMs as neural networks, the receptive fields of neurons are formed by the inter-weights between hidden and visible nodes. Due to the conditional independence in RBMs, there is no mechanism to coordinate the activations of individual neurons or the whole population. A diversity prior is introduced in this paper for training RBMs. We find that the diversity prior indeed can assure simultaneously sparsity and selectivity of neuron activations. The learned receptive fields yield a high degree of biological similarity in comparison to physiological data. Also, corresponding visual features display a good generative capability in image reconstruction.
BibTeX
@article{xiong2015diversity,
  author = {Hanchen Xiong and Antonio Jose Rodríguez-Sánchez and Sándor Szedmák and Justus H. Piater},
  title = {{Diversity priors for learning early visual features}},
  journal = {Frontiers Comput. Neurosci.},
  year = {2015},
  doi = {10.3389/fncom.2015.00104},
}
Figure