Learning sparse codes from compressed representations with biologically plausible local wiring constraints

Kion Fallah, Adam A. Willats, Ninghao Liu, Christopher J. Rozell

TL;DR

This paper introduces a novel computational model for unsupervised learning of neural representations that accounts for the "information bottlenecks" in biological sensory pathways. By using structured random matrices that mimic local neural wiring, the model can learn sparse representations from compressed data, yielding results comparable to learning from uncompressed data and aligning with observed receptive field properties in the visual cortex.

What Problem Does This Paper Address?

Biological sensory systems, particularly the visual pathway, are known to compress information significantly before it reaches higher processing areas. This compression is not a centralized, dense process but rather a distributed one with local connectivity constraints. Traditional sparse coding models, while successful in learning meaningful representations from data statistics, often overlook these biological constraints by assuming dense compression matrices. This paper addresses the gap of learning sparse representations in a compressed space that is biologically plausible, specifically by incorporating local wiring constraints that reflect how neurons are connected.

Methods

The core of this work is the development of a sparse coding framework that operates on data compressed by structured random matrices. Instead of a dense random matrix, they propose using matrices that mimic local wiring, specifically:

  • Block Diagonal Matrices (BDM): These represent groups of neurons receiving input from localized sets of inputs.
  • Banded Random Matrices (BRM): These model a more gradual shift in input connectivity between successive neurons.

These structured matrices are designed to satisfy "local wiring constraints" observed in neuroanatomy. The learning process involves performing sparse coding directly on the compressed data, and then using a reconstruction matrix to infer the original receptive fields. The paper also provides an analytic guarantee for the recovery of unique sparse representations under these structured compression schemes.

Datasets

The experiments were conducted using:

  • Whitened natural image patches: 80,000 16x16 patches were used for training, and 20,000 patches from different images were used for validation.
  • Macaque V1 receptive field measurements: The learned receptive field characteristics were compared against existing experimental data from macaque primary visual cortex.

No new datasets were introduced.

Key Insights and Ideas

  • Biologically Plausible Compression is Key: The paper demonstrates that incorporating realistic local wiring constraints into the compression process doesn't hinder, and can even improve, the learning of meaningful sparse representations.
  • Structured Random Matrices Work: Unlike dense random matrices, structured matrices (BDM, BRM) that respect local connectivity can still facilitate effective sparse coding, suggesting a more biologically realistic mechanism for information compression.
  • Receptive Fields Remain Similar: Despite learning in a compressed space with significant wiring constraints, the learned receptive fields qualitatively resemble those learned from uncompressed data and, importantly, match observed macaque V1 receptive fields.
  • Analytic Guarantees for Structured Compression: The work provides theoretical backing, showing that under certain conditions, unique sparse representations can be recovered even with these structured, localized compression matrices.

Key Results

  • Qualitative Similarity: Learned dictionaries (representing receptive fields) from compressed data with local wiring constraints (BDM, BRM) are visually similar to those learned from uncompressed data.
  • Quantitative Performance: Training and generalization errors are comparable between the proposed model and conventional sparse coding.
  • Receptive Field Matching: The statistical distribution of learned receptive field shapes (width, height, asymmetry, eccentricity) closely matches experimentally observed macaque V1 receptive fields, with the BRM model showing a particularly good fit for asymmetry.
  • Analytic Guarantee: Theorem 1 provides conditions under which sparse representations can be uniquely recovered from compressed data using BDM or BRM matrices.

Limitations and Future Work

  • Idealized Models: The BDM and BRM models are simplifications of actual retinal wiring and do not perfectly capture the complex, non-uniform density of neurons in the retina.
  • Gabor Fit Limitations: While the learned receptive fields are qualitatively similar, a precise match to receptive field size was not a primary goal. Future work could explore more overcomplete dictionaries or incorporate nonlinear output stages for a better fit.
  • Generalization to Other Bases: While the analytic results are strong for Fourier bases, extensions to other sparsity bases for BRMs are noted as an area for future research.
  • Algorithm Performance: The paper acknowledges that while analytic guarantees exist, the performance of specific numerical algorithms in finding these ideal solutions is not guaranteed. Empirical results, however, suggest good performance.

Summary updated

Abstract

Abstract Sparse coding is an important method for unsupervised learning of task-independent features in theoretical neuroscience models of neural coding. While a number of algorithms exist to learn these representations from the statistics of a dataset, they largely ignore the information bottlenecks present in fiber pathways connecting cortical areas. For example, the visual pathway has many fewer neurons transmitting visual information to cortex than the number of photoreceptors. Both empirical and analytic results have recently shown that sparse representations can be learned effectively after performing dimensionality reduction with randomized linear operators, producing latent coefficients that preserve information. Unfortunately, current proposals for sparse coding in the compressed space require a centralized compression process (i.e., dense random matrix) that is biologically unrealistic due to local wiring constraints observed in neural circuits. The main contribution of this paper is to leverage recent results on structured random matrices to propose a theoretical neuroscience model of randomized projections for communication between cortical areas that is consistent with the local wiring constraints observed in neuroanatomy. We show analytically and empirically that unsupervised learning of sparse representations can be performed in the compressed space despite significant local wiring constraints in compression matrices of varying forms (corresponding to different local wiring patterns). Our analysis verifies that even with significant local wiring constraints, the learned representations remain qualitatively similar, have similar quantitative performance in both training and generalization error, and are consistent across many measures with measured macaque V1 receptive fields.
BibTeX
@inproceedings{fallah2020learning,
  author = {Kion Fallah and Adam A. Willats and Ninghao Liu and Christopher J. Rozell},
  title = {{Learning sparse codes from compressed representations with biologically plausible local wiring constraints}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2020},
  url = {https://openreview.net/forum?id=NMTDVhcajAH},
}
Figure