Adaptive compressed sensing - A new class of self-organizing coding models for neuroscience

William K. Coulter, Christopher J. Hillar, Guy Isley, Friedrich T. SommerICASSP 2010 · DOI

TL;DR

This paper introduces Adaptive Compressed Sensing (ACS), a novel neural network model that addresses limitations of traditional sparse coding in neuroscience. ACS learns biologically realistic receptive fields even when receiving subsampled sensory input, making it a more plausible model for how sensory information is processed in the brain, particularly in cortico-cortical communication.

What Problem Does This Paper Address?

Neuroscience models aiming to explain how the brain processes sensory information, like visual input, often rely on "sparse coding." These models assume that each neuron in a network has direct, one-to-one access to the full sensory data (e.g., all pixels in an image patch) to learn efficient representations. This assumption, however, clashes with the known anatomical wiring of the brain, where connections between different brain areas are often sparse and incomplete. Specifically, the thalamo-cortical projection to the primary visual cortex and cortico-cortical projections are known to subsample the input data. This poses a challenge: how can neural circuits learn meaningful representations if they don't have access to the complete sensory information? This paper tackles this problem by proposing a new learning framework that works with subsampled data.

Methods

The core innovation is Adaptive Compressed Sensing (ACS). This approach combines ideas from compressed sensing (a technique for efficiently representing signals with fewer measurements) and adaptive learning.

  • Compressed Sensing Foundation: Instead of learning a dictionary that directly reconstructs the original high-dimensional data, ACS learns a dictionary for a compressed version of the data. This compression is achieved using a fixed, random projection matrix ($\Phi$).
  • Adaptive Learning: The "adaptive" part means the network learns its internal representation (the dictionary $\Theta$) to best decode this compressed signal. This learning is driven by an error-based energy function that is minimized using gradient descent, similar to conventional sparse coding.
  • Neural Network Implementation: The ACS learning rule is translated into a neural network architecture. Crucially, the feedforward weights in this network subsample and mix the original data, unlike traditional sparse coding where they are assumed to match the receptive fields directly.
  • Cascaded Processing: The paper also explores how ACS can model communication between brain areas. A "second stage" network receives a compressed representation from a "first stage" network and learns to decode it into another sparse representation.

Datasets

The simulations used natural scene images. These images were preprocessed by "whitening" (a standard technique in neuroscience modeling). The experiments involved coding 12x12 pixel patches from these images. No new benchmarks were introduced; the focus was on evaluating the ACS model's performance against existing sparse coding models using natural image data.

Key Insights and Ideas

  • Receptive Fields Emerge from Subsampled Data: The most striking finding is that ACS networks can learn biologically realistic, spatially smooth receptive fields even when their feedforward connections only receive a subsampled and thus impoverished version of the original sensory input. This directly addresses the anatomical limitations of previous models.
  • ACS as a Generic Cortical Communication Scheme: The model demonstrates that ACS can form meaningful representations in a secondary sensory area, even when the input from the primary area is also subsampled. This suggests ACS could be a fundamental building block for hierarchical processing in the brain, involving cycles of compression and expansion.
  • Recurrent Connections are Essential for Efficiency: The paper mathematically proves that in ACS, where input is subsampled, recurrent connections (or feedback mechanisms) become essential for shaping receptive fields. This contrasts with conventional sparse coding where direct feedforward connections can suffice.

Key Results

  • Qualitative Reconstruction: Figure 2 shows that ACS can reconstruct images from learned representations, though the quality is not as high as conventional sparse coding. However, the learned receptive fields (ACS RF) are smooth and resemble those found in biological systems, unlike the more amorphous feedforward weights (ACS FF).
  • Receptive Field Similarity: Figure 1(b) visually demonstrates that while the feedforward weights in ACS are "amorphous and noisy" due to subsampling, the learned receptive fields are smooth and resemble those of traditional sparse coding.
  • Performance Variance: Figure 3 indicates that ACS reconstructions have a lower variance compared to conventional compressed sensing, suggesting more stable learning.
  • Mathematical Proofs: Theorems 1-4 establish that in ACS, receptive fields are generally not scalar multiples of feedforward weights, especially when the input is subsampled, highlighting the crucial role of feedback in shaping these fields.

Limitations and Future Work

  • Reconstruction Quality: The paper acknowledges that the reconstruction quality from ACS is not as high as that achieved by conventional sparse coding.
  • Future Work: The authors suggest that ACS could serve as a general scheme for cortical communication, involving repeated cycles of compression and expansion. They also note that the role of recurrent circuitry in shaping receptive fields becomes essential when input is subsampled, opening avenues for further research into the interplay of feedback and feedforward processing.

Summary updated

BibTeX
@inproceedings{coulter2010adaptive,
  author = {William K. Coulter and Christopher J. Hillar and Guy Isley and Friedrich T. Sommer},
  title = {{Adaptive compressed sensing - A new class of self-organizing coding models for neuroscience}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2010},
  doi = {10.1109/icassp.2010.5495209},
}
Figure