A Model for Encoding Multiple Object Motions and Self-Motion in Area MST of Primate Visual Cortex

Richard S. Zemel, T. J. SejnowskiJournal of Neuroscience 1998 · DOI

Abstract

Many cells in the dorsal part of the medial superior temporal (MST) region of visual cortex respond selectively to specific combinations of expansion/contraction, translation, and rotation motions. Previous investigators have suggested that these cells may respond selectively to the flow fields generated by self-motion of an observer. These patterns can also be generated by the relative motion between an observer and a particular object. We explored a neurally constrained model based on the hypothesis that neurons in MST partially segment the motion fields generated by several independently moving objects. Inputs to the model were generated from sequences of ray-traced images that simulated realistic motion situations, combining observer motion, eye movements, and independent object motions. The input representation was based on the response properties of neurons in the middle temporal area (MT), which provides the primary input to area MST. After applying an unsupervised optimization technique, the units became tuned to patterns signaling coherent motion, matching many of the known properties of MST cells. The results of this model are consistent with recent studies indicating that MST cells primarily encode information concerning the relative three-dimensional motion between objects and the observer.

Introduction

When navigating a complex environment, the primate visual system must disentangle a chaotic stream of "optic flow"—the pattern of apparent motion of objects, surfaces, and edges in a visual scene. Traditionally, Area MST (Medial Superior Temporal) has been viewed primarily as a "heading detector" for self-motion. However, this 1998 study by Richard Zemel and Terrence Sejnowski proposes a more sophisticated role: MST acts as a segmentation system that parses the visual field into independent 3D motion components.

The Challenge of Motion Segmentation

The primary difficulty in interpreting optic flow is that it is often composed of multiple overlapping sources: the observer's own movement, eye rotations, and the independent trajectories of various objects.

A complex optic flow field containing multiple motion sources that the model must segment.

The researchers argue that MST neurons are optimized to represent these multiple "causes" simultaneously. Instead of just identifying a single direction of travel, the system learns to decompose the scene into coherent patches, each corresponding to a distinct 3D motion cause.

Modeling the Visual Hierarchy

To test this hypothesis, the authors developed a three-layer unsupervised neural network designed to mimic the primate visual pathway.

The three-layer neural network architecture modeling the transition from local motion vectors in MT to global motion patterns in MST.
  1. Input Layer (Area MT): This layer consists of units with small, overlapping receptive fields tuned to local velocities.
  2. Hidden Layer (Area MST): A bottleneck layer with large receptive fields. Through unsupervised learning, these units must find efficient ways to represent the complex input.
  3. Output Layer: Used during training to reconstruct the input, ensuring the hidden layer captures essential features.

The model utilizes a "Multiple-Cause" optimization technique. Unlike simpler models that might pick a single "winner," this approach assumes that while each point in space has one cause (due to occlusion), the entire scene can have several. A sparsity penalty ensures that the network explains the scene using as few active MST units as possible, favoring coherent global explanations over fragmented local ones.

Realistic Visual Inputs

Rather than using abstract dot patterns, the study employed the POV-Ray ray-tracing engine to generate realistic 15-frame movies. These scenes featured diverse backgrounds and 3D objects like teapots and animals. The simulations included observer walking, camera rotations (simulating eye movements), and up to four independently moving objects. This high-fidelity data forced the model to handle the noise and complexity inherent in real-world vision.

Key Insights and Biological Realism

The most significant finding is that classic MST tuning properties—such as selectivity for expansion, rotation, and spiral motions—emerge naturally from the requirement to segment multiple objects. The model did not need to be "pre-programmed" with these detectors; they were the most efficient way to parse the 3D world.

Furthermore, the model successfully replicated "position invariance." Real MST neurons maintain their preference for a motion type (e.g., clockwise rotation) even if the center of that motion shifts within their receptive field. The model's units showed an average shift in preferred direction of only 14.3°, closely matching the 10.7° observed in macaque physiological studies.

Results and Discussion

The unsupervised training resulted in 71% of model units becoming highly selective for specific 3D motions, mirroring the distribution of cell types in the macaque MSTd. The model proved robust, accurately extracting translational components even in "transparent" flow fields where two motions overlap in the same region—a task that standard competitive learning models fail to achieve.

While the model currently lacks extraretinal signals (like vestibular input) and explicit depth cues like binocular disparity, it provides a powerful framework for understanding how the brain transforms local motion signals into a coherent representation of a dynamic 3D environment.

Summary updated

Abstract

Many cells in the dorsal part of the medial superior temporal (MST) region of visual cortex respond selectively to specific combinations of expansion/contraction, translation, and rotation motions. Previous investigators have suggested that these cells may respond selectively to the flow fields generated by self-motion of an observer. These patterns can also be generated by the relative motion between an observer and a particular object. We explored a neurally constrained model based on the hypothesis that neurons in MST partially segment the motion fields generated by several independently moving objects. Inputs to the model were generated from sequences of ray-traced images that simulated realistic motion situations, combining observer motion, eye movements, and independent object motions. The input representation was based on the response properties of neurons in the middle temporal area (MT), which provides the primary input to area MST. After applying an unsupervised optimization technique, the units became tuned to patterns signaling coherent motion, matching many of the known properties of MST cells. The results of this model are consistent with recent studies indicating that MST cells primarily encode information concerning the relative three-dimensional motion between objects and the observer.
BibTeX
@article{zemel1998model,
  author = {Richard S. Zemel and T. J. Sejnowski},
  title = {{A Model for Encoding Multiple Object Motions and Self-Motion in Area MST of Primate Visual Cortex}},
  journal = {Journal of Neuroscience},
  year = {1998},
  doi = {10.1523/jneurosci.18-01-00531.1998},
}
Figure