Learning sparse codes from compressed representations with biologically plausible local wiring constraints

Kion Fallah, Adam Willats, Ninghao Liu, Christopher Rozell

Introduction

This paper demonstrates that the brain can learn efficient, sparse representations of visual data even when that data is compressed through biologically realistic "local wiring" bottlenecks. By using structured random matrices to model the pathway from the retina to the cortex, the authors show—both mathematically and through simulations—that the resulting neural "receptive fields" closely match those found in actual primate brains.

The Information Bottleneck Problem

In theoretical neuroscience, sparse coding is a leading model for how the primary visual cortex (V1) represents information. The idea is that the brain uses a "dictionary" of features to represent images using as few active neurons as possible.

However, there is a major gap in these models: the information bottleneck. The human retina has about 150 million photoreceptors, but only 1.5 million fibers in the optic nerve carry that data to the brain. This means the visual signal is heavily compressed. While recent mathematical advances show that sparse codes can be learned from compressed data, those models relied on "dense" random matrices—mathematical constructs where every input connects to every output. Biologically, this is impossible; a single retinal ganglion cell only receives input from a small, localized patch of photoreceptors.

The authors ask a critical question: Can we still learn effective sparse codes if the compression is restricted by biologically plausible local wiring constraints?

Methods: Modeling Local Wiring

The researchers combined sparse coding algorithms with structured random matrices to model the compression between the retina and the cortex. They tested two specific types of local wiring:

  1. Block Diagonal Matrices (BDM): The input is divided into independent groups, and each group is compressed separately.
  2. Banded Random Matrices (BRM): A more realistic model where compression "windows" overlap as they move across the input.
The retina as an information bottleneck modeled using structured random matrices (Block Diagonal and Banded Random Matrices).

To make these 1D mathematical matrices work with 2D images, they used Hilbert space-filling curves to unwrap images into vectors while preserving the proximity of neighboring pixels. The core of their approach involves learning a dictionary directly in the compressed space. To see what the neurons "see," they used a reconstruction technique (cross-correlation) to estimate the effective receptive fields (RFs) in the original pixel space.

Datasets and Validation

  • Natural Images: The model was trained on 80,000 whitened patches ($16 \times 16$ pixels) extracted from a standard set of natural images.
  • Biological Data: To validate the model, the authors compared their results against measured receptive field data from the V1 neurons of macaque monkeys (specifically, a dataset of 169 simple cells).

Key Insights and Results

  • Local Wiring is Sufficient: The most significant finding is that global, dense connections are not needed to preserve the information necessary for sparse coding. Localized, randomized projections are entirely sufficient.
  • Performance Stability: Training and generalization errors remained remarkably consistent regardless of the degree of localization. Even when connections were restricted to $1/16$th of the input space, the model performed as well as a dense model.
  • The "Noise" is Meaningful: When looking at the learned dictionary in the compressed space, it appears to be random noise. However, when mapped back to the original image space, it reveals structured, Gabor-like filters (edge detectors) that qualitatively resemble biological receptive fields.
  • Better than the Standard Model: Surprisingly, the Banded Random Matrix (BRM) model produced receptive fields with "asymmetry" statistics that matched biological data better than traditional, uncompressed sparse coding models. Using a Kruskal-Wallis analysis, the authors showed that the BRM model's distribution of receptive field asymmetry was significantly closer to macaque V1 data ($P < 0.05$). This suggests that the compression bottleneck might actually be a feature, not just a limitation, of the visual system.

Limitations and Future Work

  • Basis Constraints: The analytic proof in the paper is currently specific to an orthonormal Fourier basis. While empirical results suggest the approach works for other dictionaries, a more general mathematical proof is still needed.
  • Receptive Field Size: The model's recovered receptive fields were generally smaller than those measured in real monkeys. The authors suggest this might be resolved by using "overcomplete" dictionaries (more neurons than inputs) or adding non-linear processing stages.
  • Developmental Modeling: Future work could explore how these randomized, localized connections might grow during biological development, potentially guided by chemical gradients.

Summary updated

Abstract

Sparse coding is an important method for unsupervised learning of task-independent features in theoretical neuroscience models of neural coding. While a number of algorithms exist to learn these representations from the statistics of a dataset, they largely ignore the information bottlenecks present in fiber pathways connecting cortical areas. For example, the visual pathway has many fewer neurons transmitting visual information to cortex than the number of photoreceptors. Both empirical and analytic results have recently shown that sparse representations can be learned effectively after performing dimensionality reduction with randomized linear operators, producing latent coefficients that preserve information. Unfortunately,current proposals for sparse coding in the compressed space require a centralized compression process (i.e., dense random matrix) that is biologically unrealistic due to local wiring constraints observed in neural circuits. The main contribution of this paper is to leverage recent results on structured random matrices to propose a theoretical neuroscience model of randomized projections for communication between cortical areas that is consistent with the local wiring constraints observed in neuroanatomy. We show analytically and empirically that unsupervised learning of sparse representations can be performed in the compressed space despite significant local wiring constraints in compression matrices of varying forms (corresponding to different local wiring patterns). Our analysis verifies that even with significant local wiring constraints, the learned representations remain qualitatively similar,have similar quantitative performance in both training and generalization error, and are consistent across many measures with measured macaque V1 receptive fields.
BibTeX
@inproceedings{fallah2020learning,
  author = {Kion Fallah and Adam Willats and Ninghao Liu and Christopher Rozell},
  title = {{Learning sparse codes from compressed representations with biologically plausible local wiring constraints}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2020},
  url = {https://openreview.net/forum?id=NMTDVhcajAH},
}
Figure