Distinct Mechanisms for Coding of Visual Actions in Macaque Temporal Cortex

Joris Vangeneugden, Patrick De Mazière, Marc M. Van Hulle, Tobias Jaeggli, Luc Van Gool, Rufin VogelsJournal of Neuroscience 2011 · DOI

Abstract

Temporal cortical neurons are known to respond to visual dynamic-action displays. Many human psychophysical and functional imaging studies examining biological motion perception have used treadmill walking, in contrast to previous macaque single-cell studies. We assessed the coding of locomotion in rhesus monkey (Macaca mulatta) temporal cortex using movies of stationary walkers, varying both form and motion (i.e., different facing directions) or varying only the frame sequence (i.e., forward vs backward walking). The majority of superior temporal sulcus and inferior temporal neurons were selective for facing direction, whereas a minority distinguished forward from backward walking. Support vector machines using the temporal cortical population responses as input classified facing direction well, but forward and backward walking less so. Classification performance for the latter improved markedly when the within-action response modulation was considered, reflecting differences in momentary body poses within the locomotion sequences. Responses to static pose presentations predicted the responses during the course of the action. Analyses of the responses to walking sequences wherein the start frame was varied across trials showed that some neurons also carried a snapshot sequence signal. Such sequence information was present in neurons that responded to static snapshot presentations and in neurons that required motion. Our data suggest that actions are analyzed by temporal cortical neurons using distinct mechanisms. Most neurons predominantly signal momentary pose. In addition, temporal cortical neurons, including those responding to static pose, are sensitive to pose sequence, which can contribute to the signaling of learned action sequences.

Introduction

Recognizing the actions of others is a cornerstone of social behavior and survival. While the Superior Temporal Sulcus (STS) and Inferior Temporal (IT) cortex are known to be involved in "biological motion" perception, the underlying computational logic remains debated. The core challenge lies in distinguishing between form (the static appearance of the body) and motion (the temporal change in that appearance). For instance, when viewing a person walking on a treadmill, the individual "snapshots" of the body are identical whether they walk forward or backward; only the order of those snapshots changes. This study investigates whether neurons in the macaque temporal cortex simply respond to specific poses or if they are sensitive to the temporal sequence of an action.

Methods and Experimental Design

Researchers recorded single-neuron activity from two rhesus monkeys while they viewed "humanoid-like" 3D animations. These stimuli were derived from high-resolution human motion-capture (MoCap) data and rendered as cylindrical primitives to balance realism with experimental control.

Experimental stimuli depicting eight facing directions and their respective walk cycles.

To isolate action from simple translation, the agents walked "on a treadmill" in the center of the screen. The experimental parameters varied in two primary ways: facing direction (8 angles) and walking direction (forward vs. backward). To ensure neurons were not merely "timing" their responses from the start of a movie, the researchers randomized the starting frame of the walk cycle across trials. Recordings were focused on the STS and IT regions, known hubs for visual object and action processing.

Anatomical recording site within the macaque temporal cortex.

Key Insights

Facing Direction vs. Sequence Sensitivity

The study found that the temporal cortex is primarily a "form" processor. Approximately 65% of recorded neurons were highly selective for the walker's facing direction. In contrast, only a minority (18%) could distinguish between forward and backward walking. This suggests that while the brain is excellent at identifying the orientation of a body, distinguishing the direction of movement through sequence alone is a more specialized task.

The Importance of Temporal Binning

Using Support Vector Machines (SVMs) to decode neural signals revealed a critical nuance: classification accuracy for forward/backward walking was mediocre when looking at the average response over a full second. However, when the researchers analyzed the data in 50ms "bins," accuracy improved significantly. This indicates that the information distinguishing forward from backward motion is encoded in the rapid fluctuations of neural firing as the walk cycle progresses, reflecting the momentary changes in body pose.

Snapshot vs. Action Neurons

The researchers categorized neurons into "Snapshot" (responding to static frames) and "Action" (requiring motion) types. Interestingly, even when the start frame was randomized, a subset of neurons maintained their selectivity for forward vs. backward walking. This proves these neurons are not just responding to a single "keyframe" but are integrating a pose sequence signal.

Conclusion

The data supports a dual-mechanism model of action recognition in the macaque temporal cortex. One pathway recognizes momentary body shapes (form), while a more specialized pathway tracks how those shapes transition over time (sequence). While the use of stationary treadmill walkers is an abstraction of natural movement—which usually involves translatory motion across space—this study clarifies how the ventral visual stream contributes to the complex task of decoding biological motion. Future work may explore how much of this sequence sensitivity is innate versus learned through repeated exposure to common social actions.

Summary updated

Abstract

Temporal cortical neurons are known to respond to visual dynamic-action displays. Many human psychophysical and functional imaging studies examining biological motion perception have used treadmill walking, in contrast to previous macaque single-cell studies. We assessed the coding of locomotion in rhesus monkey (Macaca mulatta) temporal cortex using movies of stationary walkers, varying both form and motion (i.e., different facing directions) or varying only the frame sequence (i.e., forward vs backward walking). The majority of superior temporal sulcus and inferior temporal neurons were selective for facing direction, whereas a minority distinguished forward from backward walking. Support vector machines using the temporal cortical population responses as input classified facing direction well, but forward and backward walking less so. Classification performance for the latter improved markedly when the within-action response modulation was considered, reflecting differences in momentary body poses within the locomotion sequences. Responses to static pose presentations predicted the responses during the course of the action. Analyses of the responses to walking sequences wherein the start frame was varied across trials showed that some neurons also carried a snapshot sequence signal. Such sequence information was present in neurons that responded to static snapshot presentations and in neurons that required motion. Our data suggest that actions are analyzed by temporal cortical neurons using distinct mechanisms. Most neurons predominantly signal momentary pose. In addition, temporal cortical neurons, including those responding to static pose, are sensitive to pose sequence, which can contribute to the signaling of learned action sequences.
BibTeX
@article{vangeneugden2011distinct,
  author = {Joris Vangeneugden and Patrick De Mazière and Marc M. Van Hulle and Tobias Jaeggli and Luc Van Gool and Rufin Vogels},
  title = {{Distinct Mechanisms for Coding of Visual Actions in Macaque Temporal Cortex}},
  journal = {Journal of Neuroscience},
  year = {2011},
  doi = {10.1523/jneurosci.2703-10.2011},
}
Figure