Abstract
Traditional models of the primary visual cortex (V1) often prioritize sparsity and selectivity to replicate biological neural activity. However, these models frequently suffer from redundancy and "dead neurons." This paper proposes that sparsity and selectivity are emergent properties of a more fundamental principle: diversity. By implementing a "diversity prior" in a Restricted Boltzmann Machine (RBM), the authors demonstrate the development of Gabor-like visual features that align with biological data while ensuring full utilization of the neural population.
Introduction
In the study of the visual cortex, two properties of "simple cells" in area V1 are legendary: selectivity and sparsity. Selectivity refers to a neuron's specific response to particular stimuli (e.g., a vertical line at a specific location), while sparsity describes the phenomenon where only a tiny fraction of the total neuron population is active at any given moment.
While classic models like Independent Component Analysis (ICA) and Sparse Coding have tried to optimize for these properties directly, they often struggle to achieve both simultaneously. Furthermore, in neural networks like RBMs, neurons are "conditionally independent," meaning they do not coordinate which features they learn. This often results in redundant features or "dead neurons" that never respond to stimuli. The authors argue that instead of optimizing for sparsity or selectivity directly, we should force neurons to be diverse.
Methods and Data
The researchers utilized a Restricted Boltzmann Machine (RBM) and introduced a novel Diversity Prior during the learning phase. This prior works by minimizing the square cosine similarity between the weight vectors (receptive fields) of the hidden units, pushing the learned "basis images" as far apart as possible in mathematical space. Biologically, this mimics the effect of lateral inhibition, where active neurons suppress their neighbors to sharpen signal contrast.
Training was conducted using the Olshausen and Field (1996) benchmark database of natural images. These images were "whitened" to mimic retinal processing and broken into 100,000 small $14 \times 14$ pixel patches. To validate the results, the learned features were compared against physiological data from macaque monkey V1 neurons.
Key Insights: Diversity as a Primary Driver
The most striking insight of this work is that population sparsity and individual selectivity emerge naturally as "epiphenomena" when neurons are forced to represent diverse, non-overlapping features. Even though the RBM architecture lacks physical lateral connections, the diversity prior acts as a "virtual inhibitory interneuron" during training.
By penalizing similarity, the model ensures that the "real estate" of possible visual features is covered efficiently. This prevents multiple neurons from wasting energy learning the exact same edge orientation, leading to a more robust and biologically plausible representation.
Key Results
- Biological Fidelity: The learned receptive fields were Gabor-like filters that showed a high degree of similarity to actual macaque V1 data in terms of shape profiles and aspect ratios.
- Eliminating "Dead" Neurons: In a population of 200 neurons, the Diverse RBM had 0 dead neurons. In contrast, a standard "Sparse RBM" had 68 dead neurons, and a "Selective RBM" had 88. This proves that diversity is a much more robust way to ensure every part of the model is doing useful work.
- Generative Success: The model could reconstruct test images with high qualitative accuracy, proving the learned features captured the essential statistics of natural scenes.
Limitations and Future Work
The authors acknowledge that an RBM with a diversity prior is not a complete biological model, as it lacks features like contrast normalization and temporal adaptation. Future work could explore how diversity priors behave in deep, multi-layer stacks (Deep Belief Networks) or apply these priors to video data to see if "temporal diversity" leads to the discovery of motion-sensitive features similar to those in area V5/MT.
Summary updated