Neural dynamics at successive stages of the ventral visual stream are consistent with hierarchical error signals

Elias B. Issa, Charles F. Cadieu, James J. DiCarlobioRxiv 2016 · DOI

Abstract

Neural responses in the ventral visual stream are inherently dynamic, yet most computational models focus on static, time-averaged outputs. By recording from three successive stages of the macaque inferior temporal (IT) cortex during face detection, we observed a striking phenomenon: approximately 30 milliseconds after the initial response, neurons in intermediate areas (pIT) significantly decreased their face preference, often becoming "anti-face" preferring. Meanwhile, higher-level areas (aIT) maintained a consistent face preference. This hierarchical dynamic is poorly explained by standard feedforward or local recurrent models but is accurately captured by models that explicitly compute prediction errors between stages. These findings suggest that the ventral stream employs error-coding as a fundamental spiking primitive, providing a biological basis for predictive processing in vision.

Introduction

Deep Convolutional Neural Networks (DCNNs) have revolutionized our understanding of the ventral visual stream, serving as the current gold standard for modeling object recognition. However, these models are predominantly feedforward and static. In contrast, biological neural activity is highly temporal; even when presented with a static stimulus, firing rates fluctuate significantly over time. Understanding whether these dynamics are merely biological "noise" (like adaptation or normalization) or represent a specific computation is critical for bridging the gap between artificial and biological intelligence.

Experimental Framework

To investigate these dynamics, researchers recorded multi-unit activity from three hierarchical "face patches" in the macaque IT cortex: the posterior (pIT), central (cIT), and anterior (aIT) regions.

The study utilized a "Challenge Set" of images designed to create ambiguity. These images contained face parts (eyes, nose, mouth) arranged in scrambled or "anti-face" configurations (e.g., a "cyclops" eye in the center of a face). Initially, these scrambled images drive pIT neurons as strongly as normal faces because pIT responds to local features. However, as the hierarchy processes the global structure, the ambiguity must be resolved.

The Paradoxical Preference Flip

The most significant discovery was a temporal "flip" in neural selectivity within the pIT. Initially, pIT neurons responded strongly to both faces and scrambled faces. However, within 30 milliseconds of the initial response, 84% of these neurons showed a marked decrease in face preference. The population median selectivity ($d'$) shifted from 0.11 (early phase) to -0.12 (late phase), meaning the neurons actually began to prefer the scrambled images over the real faces.

In contrast, the anterior area (aIT) maintained a steady preference for faces throughout the stimulus presentation. This divergence suggests that the pIT is not simply passing a "face" signal forward, but is reacting to the discrepancy between the local features it detects and the global "face" identity determined by higher areas.

Modeling Hierarchical Error Signals

The researchers compared the neural data against three classes of dynamical models:

  1. Pure Feedforward: Simple signal propagation without feedback.
  2. Lateral Inhibition/Normalization: Within-stage recurrence where neurons suppress neighbors.
  3. Error-Coding Feedback: A normative model where stages minimize a reconstruction cost by passing error signals—the difference between top-down predictions and bottom-up input—back and forth.

Standard models of adaptation and normalization failed to replicate the preference flip, typically predicting that a neuron’s relative preference remains stable even as its total activity declines ($r^2 \approx 0.17$). The error-coding model, however, fit the data with remarkable accuracy ($r^2$ between 0.88 and 0.96). In this framework, pIT neurons act as "error detectors," signaling the mismatch between the "face" predicted by aIT and the "scrambled parts" physically present on the screen.

Broader Implications and Generalization

Beyond explaining the preference flip, the error-coding model generalized to other well-known IT phenomena without additional parameter tuning:

  • Sublinear Integration: The model explains why the response to a "whole face" is often less than the sum of its parts; the "whole" is predicted by the hierarchy, reducing the error signal.
  • Familiarity Sharpening: The model accounts for why familiar images produce sharper, more efficient neural transients compared to novel ones.

Furthermore, MRI-aligned recordings revealed a laminar component: superficial layers exhibited slower, "state-like" integration, while deeper layers showed faster, "error-like" transients. This suggests a specific physical architecture for error-propagation within the cortical columns.

Conclusion

These results suggest that the ventral stream is not a simple feedforward pipeline but a sophisticated system for hierarchical inference. By explicitly encoding errors, the brain can resolve visual ambiguities and refine its internal representations. While the current models are largely linear, integrating these dynamical error signals into non-linear DCNNs could lead to more robust and biologically plausible computer vision systems, potentially revealing how the brain implements learning algorithms similar to backpropagation.

Summary updated

Abstract

ABSTRACT Ventral visual stream neural responses are dynamic, even for static image presentations. However, dynamical neural models of visual cortex are lacking as most progress has been made modeling static, time-averaged responses. Here, we studied population neural dynamics during face detection across three cortical processing stages. Remarkably, ~30 milliseconds after the initially evoked response, we found that neurons in intermediate level areas decreased their preference for faces, becoming anti-face preferring on average even while neurons in higher level areas achieved and maintained a face preference. This pattern of hierarchical neural dynamics was inconsistent with extensions of standard feedforward circuits that implemented recurrence within a cortical stage. Rather, recurrent models computing errors between stages captured the observed temporal signatures. Without additional parameter fitting, this model of neural dynamics, which simply augments the standard feedforward model of online vision to encode errors, also explained seemingly disparate dynamical phenomena in the ventral stream.
BibTeX
@misc{issa2016neural,
  author = {Elias B. Issa and Charles F. Cadieu and James J. DiCarlo},
  title = {{Neural dynamics at successive stages of the ventral visual stream are consistent with hierarchical error signals}},
  year = {2016},
  doi = {10.1101/092551},
  howpublished = {bioRxiv},
}
Figure