Variance predicts salience in central sensory processing

Ann M. Hermundstad, John J. Briguglio, Mary M. Conte, Jonathan D. Victor, Vijay Balasubramanian, Gašper TkačikeLife 2014 · DOI

TL;DR

This paper proposes that the brain prioritizes processing of more variable sensory features in higher cortical areas, a departure from the periphery's "whitening" strategy. They demonstrate that human visual sensitivity to complex image features can be accurately predicted by the statistical variability of those features in natural images, suggesting a unified efficient coding principle across sensory processing.

What Problem Does This Paper Address?

The brain is constantly bombarded with sensory information, yet it must efficiently process the most relevant parts. While the "efficient coding" principle has successfully explained how sensory systems in the periphery (like the retina) compress redundant information, it's unclear how this principle applies to the more complex feature extraction performed by higher cortical areas. Specifically, how does the brain decide which complex features to prioritize for processing when faced with limited resources and sampling constraints? This paper investigates this by examining how human visual sensitivity to complex image statistics relates to their prevalence in natural scenes.

Methods

The authors employed a multi-pronged approach:

  1. Image Statistics Analysis: They analyzed natural images (from the UPenn Natural Image Database and van Hateren Database) by breaking them down into small patches. These patches were preprocessed (block-averaged, whitened, and binarized) and then characterized by a set of 10 "image statistic coordinates" that describe local correlations between pixels, ranging from simple pairwise relationships to more complex multi-point interactions. They then calculated the "degree of variation" (standard deviation) for each of these coordinates across a large ensemble of natural images.
  2. Psychophysical Experiments: They created synthetic visual textures with controlled image statistic coordinates. Human subjects performed a figure/ground segmentation task where they had to distinguish a target texture from noise. By measuring the threshold at which subjects could perform this task, the researchers quantified human "perceptual sensitivity" to these image statistics.
  3. Comparison and Prediction: They compared the statistical variation of image features in natural scenes with human perceptual sensitivity to those same features. The core hypothesis was that features with higher variability in natural images would correspond to higher perceptual sensitivity. They used statistical methods, including principal component analysis and permutation tests, to rigorously assess the correspondence between image statistics and human sensitivity.

Datasets

  • UPenn Natural Image Database: A collection of 924 panoramic, eye-level images from a savannah habitat in Botswana.
  • van Hateren Natural Image Dataset: A dataset of approximately 2300 images used for validation.

The paper does not introduce new benchmark datasets but rather uses existing ones to test their hypotheses.

Key Insights and Ideas

  • Two Regimes of Efficient Coding: The paper highlights two distinct efficient coding regimes:
  • Transmission-limited (Periphery): Optimal coding involves "whitening" or decorrelating signals, where more variable signals are compressed. This is relevant when output capacity is the bottleneck (e.g., optic nerve).
  • Sampling-limited (Cortex): Optimal coding involves allocating more resources to more variable features, as they are less predictable and more informative when input is noisy (e.g., due to sampling limitations). This is proposed to be relevant for cortical processing.
  • Variance is Salience: The central idea is that in the sampling-limited regime, perceptual salience (how easily a feature is detected) increases with its variability in natural scenes.
  • Beyond Pairwise Correlations: The study extends efficient coding principles beyond simple pairwise pixel correlations to higher-order, multi-point interactions, which are crucial for defining complex visual features.
  • Neural Hardware Constraints: While natural image statistics are asymmetric for some features, human sensitivity is often symmetric. This suggests that neural mechanisms might employ simpler "opponent" processing to approximate the statistics, reflecting limitations in neural hardware.

Key Results

  • Quantitative Prediction: The degree of variation in natural image statistics for dozens of independently measured parameters (local multi-point spatial correlations) was quantitatively predicted by human visual sensitivity to those same parameters.
  • Rank-Ordering Match: The rank-ordering of perceptual sensitivities along single coordinate axes closely matched the rank-ordering of variation in natural image statistics ({β, β_} > {β, β} > α > {θ, θ, θ, θ}).
  • Isodiscrimination Contours: The shape and orientation of human isodiscrimination contours (which define perceptual sensitivity in pairwise coordinate planes) were accurately predicted by the covariance structure of natural image statistics.
  • High Correspondence: The overlap between predictions from natural image statistics and psychophysical measurements was consistently high (0.938-0.999) and statistically significant across various analyses and subjects.

Limitations and Future Work

  • Cortical Processing vs. Perception: The study measured perceptual sensitivity, not direct cortical responses. While they argue for cortical involvement based on stimulus properties and receptive field sizes, direct neural evidence is lacking.
  • Optimal Sampling Region: The paper acknowledges that the optimal size and scale of the sampling region for image statistics are unknown, though their predictions were robust across a range of patch sizes.
  • Neural Mechanisms: While the study predicts what should be encoded, the specific neural mechanisms (e.g., opponent processing) are speculative and require further investigation.
  • Higher-Level Features: The authors suggest extending this principle to even more complex features like shapes and contour fragments, and to higher brain areas like IT cortex.
  • Top-Down Influences: The role of task-specific goals and top-down influences on visual computation remains an open question.

Summary updated

Abstract

Information processing in the sensory periphery is shaped by natural stimulus statistics. In the periphery, a transmission bottleneck constrains performance; thus efficient coding implies that natural signal components with a predictably wider range should be compressed. In a different regime--when sampling limitations constrain performance--efficient coding implies that more resources should be allocated to informative features that are more variable. We propose that this regime is relevant for sensory cortex when it extracts complex features from limited numbers of sensory samples. To test this prediction, we use central visual processing as a model: we show that visual sensitivity for local multi-point spatial correlations, described by dozens of independently-measured parameters, can be quantitatively predicted from the structure of natural images. This suggests that efficient coding applies centrally, where it extends to higher-order sensory features and operates in a regime in which sensitivity increases with feature variability.
BibTeX
@inproceedings{hermundstad2014variance,
  author = {Ann M. Hermundstad and John J. Briguglio and Mary M. Conte and Jonathan D. Victor and Vijay Balasubramanian and Gašper Tkačik},
  title = {{Variance predicts salience in central sensory processing}},
  booktitle = {eLife},
  year = {2014},
  doi = {10.7554/elife.03722},
}
Figure