Grouping Components of Three-Dimensional Moving Objects in Area MST of Visual Cortex

Richard S. Zemel, Terrence J. SejnowskiNeurIPS 1994 · OpenReview

When navigating a crowded environment, the visual system is inundated with a complex "flow field" of moving vectors. This motion arises from a combination of self-motion (ego-motion), eye movements, and the independent trajectories of objects like pedestrians or vehicles. Historically, neuroscientists have believed that neurons in the dorsal medial superior temporal (MSTd) area of the visual cortex—which respond to large-scale patterns like expansion and rotation—primarily serve to determine our heading.

However, in their research, Richard S. Zemel and Terrence J. Sejnowski propose a compelling alternative: MST's primary function is motion segmentation. By grouping local motion signals that share a common underlying cause, MST helps the brain separate independent moving objects from the background.

The Computational Model

To test whether MST-like tuning could emerge naturally from the demands of scene segmentation, the authors developed a computational neural network model designed to mimic the hierarchy of the visual cortex.

  • Synthetic Stimuli: The researchers generated movies using a ray-tracing program to simulate realistic visual scenarios. These 15-frame sequences combined random observer motion, eye tracking, and up to four independent 3D objects (such as a teapot, fish, or snail) moving across various backgrounds.
  • MT-like Input Layer: The model extracted optical flow from these movies and fed it into an input layer modeled after area MT, the cortical precursor to MST. This layer utilized population encoding, where sets of neurons at each location were tuned to specific directions of motion.
  • Unsupervised Learning: Crucially, the network was not explicitly trained to recognize "spirals" or specific motion patterns. Instead, it utilized an unsupervised learning algorithm akin to an autoencoder, forcing the network to reconstruct the input pattern at the output layer. To achieve this efficiently, the hidden "MST" layer had to identify the most statistically significant patterns—the "common causes"—within the motion data.

Key Findings and Insights

By training the network to reconstruct the scene, the model learned to ignore noise and focus on coherent structures, mirroring how the brain parses a messy visual world into distinct objects.

Emergence of Complex Tuning

Following training, the hidden layer units spontaneously developed selective tuning to expansion, contraction, and rotation. This perfectly matches the known physiological properties of biological MSTd neurons in monkeys. The spontaneous emergence of this "spiral" tuning suggests that these patterns are the mathematically optimal way to represent 3D motion components within a 2D visual flow field.

MST as a Hypothesis Generator

The authors posit that each MST cell acts as a "motion hypothesis." Rather than simply detecting expansion, a cell effectively signals that a specific portion of the flow field is consistent with a single 3D object moving relative to the observer. This supports the idea that MST is deeply involved in processing 3D object motion, rather than acting solely as a heading sensor.

Limitations and Future Directions

While the model successfully demonstrates that the statistics of realistic flow fields are sufficient to drive the development of complex motion-sensitive neurons without explicit supervision, it does have limitations. The study relies on a synthetic, ray-traced environment that, while sophisticated for its time, lacks the full textural and lighting complexity of the real world.

Furthermore, the model focuses heavily on the "what" of motion grouping, leaving the "where" (precise spatial localization) and the integration of other sensory inputs (like vestibular balance) for future exploration. Future work could scale this approach to handle more complex scenes and compare the model's segmentation errors to human psychophysical performance in motion-grouping tasks.

Summary updated

Abstract

Many cells in the dorsal part of the medial superior temporal (MST) area of visual cortex respond selectively to spiral flow patterns-specific combinations of expansion/ contraction and ro(cid:173) tation motions. Previous investigators have suggested that these cells may represent self-motion. Spiral patterns can also be gener(cid:173) ated by the relative motion of the observer and a particular object. An MST cell may then account for some portion of the complex flow field, and the set of active cells could encode the entire flow; in this manner, MST effectively segments moving objects. Such a grouping operation is essential in interpreting scenes containing several independent moving objects and observer motion. We de(cid:173) scribe a model based on the hypothesis that the selective tuning of MST cells reflects the grouping of object components undergo(cid:173) ing coherent motion. Inputs to the model were generated from sequences of ray-traced images that simulated realistic motion sit(cid:173) uations, combining observer motion, eye movements, and indepen(cid:173) dent object motion. The input representation was modeled after response properties of neurons in area MT, which provides the pri(cid:173) mary input to area MST. After applying an unsupervised learning algorithm, the units became tuned to patterns signaling coherent motion. The results match many of the known properties of MST cells and are consistent with recent studies indicating that these cells process 3-D object motion information. 166 Richard S. Zemel, Terrence J. Sejnowski
BibTeX
@inproceedings{zemel1994grouping,
  author = {Richard S. Zemel and Terrence J. Sejnowski},
  title = {{Grouping Components of Three-Dimensional Moving Objects in Area MST of Visual Cortex}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {1994},
  url = {https://openreview.net/forum?id=SyEgbub_Wr},
}
Figure