Moving a Shape behind a Slit: Partial Shape Representations in Inferior Temporal Cortex

Anna Bognár, Rufin VogelsJournal of Neuroscience 2021 · DOI

Abstract

Current models of object recognition are based on spatial representations built from object features that are simultaneously present in the retinal image. However, one can recognize an object when it moves behind a static occluder, and only a small fragment of its shape is visible through a slit at a given moment in time. Such anorthoscopic perception requires spatiotemporal integration of the successively presented shape parts during slit-viewing. Human fMRI studies suggested that ventral visual stream areas represent whole shapes formed through temporal integration during anorthoscopic perception. To examine the time course of shape-selective responses during slit-viewing, we recorded the responses of single inferior temporal (IT) neurons of rhesus monkeys to moving shapes that were only partially visible through a static narrow slit. The IT neurons signaled shape identity by their response when that was cumulated across the duration of the shape presentation. Their shape preference during slit-viewing equaled that for static, whole-shape presentations. However, when analyzing their responses at a finer time scale, we showed that the IT neurons responded to particular shape fragments that were revealed by the slit. We found no evidence for temporal integration of slit-views that result in a whole-shape representation, even when the monkey was matching slit-views of a shape to static whole-shape presentations. These data suggest that, although the temporally integrated response of macaque IT neurons can signal shape identity in slit-viewing conditions, the spatiotemporal integration needed for the formation of a whole-shape percept occurs in other areas, perhaps downstream to IT.

Significance Statement

One recognizes an object when it moves behind a static occluder and only a small fragment of its shape is visible through a static slit at a given moment in time. Such anorthoscopic perception requires spatiotemporal integration of the successively presented partial shape parts. Human fMRI studies suggested that ventral visual stream areas represent shapes formed through temporal integration. We recorded the responses of inferior temporal (IT) cortical neurons of macaques during slit-viewing conditions. Although the temporally summated response of macaque IT neurons could signal shape identity under slit-viewing conditions, we found no evidence for a whole-shape representation using analyses at a finer time scale. Thus, the spatiotemporal integration needed for anorthoscopic perception does not occur within IT.

Introduction

A fundamental challenge in visual neuroscience is understanding how the brain recognizes objects under suboptimal conditions. A classic example is anorthoscopic perception (AP): the ability to recognize a whole object when it is viewed through a narrow slit as it moves. In this scenario, the retina never receives a complete image of the object at any single point in time. Instead, the brain must perform spatiotemporal integration, stitching together a sequence of partial fragments into a coherent whole.

While human fMRI studies have implicated the ventral visual stream—specifically the Lateral Occipital Complex (LOC)—in this process, the underlying neural mechanisms remain unclear. Does the "what" pathway actually reconstruct the whole shape, or does it merely process the fragments? This study investigates these questions by recording single-neuron activity in the macaque inferior temporal (IT) cortex, the homologue of the human LOC.

Methods

The researchers recorded from 196 single neurons in the ASB body patch of three rhesus monkeys. This region was identified using fMRI localizers for its robust response to animal silhouettes.

Experimental Paradigm

The primary task involved monkeys viewing animal silhouettes moving behind a static, narrow slit (0.48° width). At a movement speed of 6.2°/s, only a small fraction (~10%) of the shape was visible in any given frame. The study compared neural responses across several conditions:

  1. Static Whole Shapes: Full silhouettes presented for comparison.
  2. Ordered Slit-Viewing: Fragments presented in their natural temporal sequence as the shape moves.
  3. Random Slit-Viewing: Fragments presented in a scrambled temporal order.
Slit-viewing experimental setup showing horizontal and vertical slits, temporal frames of fragments, and the random temporal order condition.

Behavioral Task

To ensure the monkeys were actually perceiving the whole shapes, one monkey was trained on a Delayed Matching to Sample (DMS) task. In this task, the monkey viewed a slit-view "sample" and, after a brief delay, had to choose the matching whole shape from two alternatives.

Key Results

Fragment-Based Neural Responses

The most significant finding was that IT neurons did not exhibit a sustained response representing the "reconstructed" whole shape. Instead, they showed transient, burst-like activity corresponding to specific fragments as they passed through the slit. For example, a neuron might fire only when the "head" or "legs" of an animal silhouette were visible.

Decoding and Integration

While the population of IT neurons could accurately signal shape identity (often >90% accuracy) when activity was integrated over the entire presentation window, this was a result of temporal summation rather than true integration. Cross-temporal decoding revealed that the neural code was highly dynamic; a classifier trained on the response at one time point could not generalize to another, confirming that the neurons were tracking changing fragments rather than a stable whole-shape representation.

Behavioral Evidence for Perception

The monkey successfully performed the DMS task with approximately 70% accuracy, well above chance. Crucially, performance dropped to chance levels in the "random" condition, where the temporal sequence was disrupted. This confirms that the temporal order is essential for the perceptual construction of the shape, even if IT neurons themselves do not show evidence of this construction.

Timeline of the Delayed Matching to Sample (DMS) task, where monkeys matched a slit-view sample to static whole-shape test stimuli.

Discussion

The results suggest a functional "gap" in the IT cortex. While IT neurons are selective for the fragments that constitute an object, they do not appear to be the site where these fragments are integrated into a persistent whole-shape representation. The fact that neural behavior remained fragment-selective even during the DMS task—where the monkey was actively performing integration—suggests that the "merging" of these pieces into a conscious percept occurs elsewhere.

Potential candidates for this integration include the Prefrontal Cortex (PFC), which handles working memory and task-related synthesis, or the Dorsal Stream (e.g., area hMT/V5), which might provide a motion-based spatial framework for the incoming fragments. For computational modeling, these findings highlight the need for architectures that go beyond simple spatial templates to incorporate complex temporal dynamics.

Summary updated

Abstract

Current models of object recognition are based on spatial representations build from object features that are simultaneously present in the retinal image. However, one can recognize an object when it moves behind a static occlude, and only a small fragment of its shape is visible through a slit at a given moment in time. Such anorthoscopic perception requires spatiotemporal integration of the successively presented shape parts during slit-viewing. Human fMRI studies suggested that ventral visual stream areas represent whole shapes formed through temporal integration during anorthoscopic perception. To examine the time course of shape-selective responses during slit-viewing, we recorded the responses of single inferior temporal (IT) neurons of rhesus monkeys to moving shapes that were only partially visible through a static narrow slit. The IT neurons signaled shape identity by their response when that was cumulated across the duration of the shape presentation. Their shape preference during slit-viewing equaled that for static, whole-shape presentations. However, when analyzing their responses at a finer time scale, we showed that the IT neurons responded to particular shape fragments that were revealed by the slit. We found no evidence for temporal integration of slit-views that result in a whole-shape representation, even when the monkey was matching slit-views of a shape to static whole-shape presentations. These data suggest that, although the temporally integrated response of macaque IT neurons can signal shape identity in slit-viewing conditions, the spatiotemporal integration needed for the formation of a whole-shape percept occurs in other areas, perhaps downstream to IT.<b>SIGNIFICANCE STATEMENT</b> One recognizes an object when it moves behind a static occluder and only a small fragment of its shape is visible through a static slit at a given moment in time. Such anorthoscopic perception requires spatiotemporal integration of the successively presented partial shape parts. Human fMRI studies suggested that ventral visual stream areas represent shapes formed through temporal integration. We recorded the responses of inferior temporal (IT) cortical neurons of macaques during slit-viewing conditions. Although the temporally summated response of macaque IT neurons could signal shape identity under slit-viewing conditions, we found no evidence for a whole-shape representation using analyses at a finer time scale. Thus, the spatiotemporal integration needed for anorthoscopic perception does not occur within IT.
BibTeX
@article{bognar2021moving,
  author = {Anna Bognár and Rufin Vogels},
  title = {{Moving a Shape behind a Slit: Partial Shape Representations in Inferior Temporal Cortex}},
  journal = {Journal of Neuroscience},
  year = {2021},
  doi = {10.1523/jneurosci.0348-21.2021},
}
Figure