Good continuation in 3D: the neurogeometry of stereo vision

Maria Virginia Bolelli, Giovanna Citti, Alessandro Sarti, Steven W. ZuckerFrontiers of Computer Science 2023 · DOI

TL;DR

This paper proposes a novel neurogeometric model for stereo vision that extends the concept of "good continuation" from 2D curves to 3D space. By formalizing stereo vision within a sub-Riemannian geometry framework, the authors provide a biologically plausible explanation for how the visual system might solve the correspondence problem and perceive depth.

What Problem Does This Paper Address?

The fundamental challenge in stereo vision is the "correspondence problem": determining which features in the left eye's image correspond to features in the right eye's image to reconstruct a 3D scene. While 2D contour integration has been well-modeled using principles like "good continuation" (the tendency to perceive smooth, continuous contours), extending this to 3D depth perception has been less developed. This paper aims to bridge this gap by developing a computational model that generalizes good continuation to 3D spatial curves, offering insights into the underlying neurobiology and computational mechanisms of stereo vision.

Methods

The authors introduce a sub-Riemannian geometry framework to model stereo vision. This approach builds upon the idea of "cortical space" where neurons are organized by position and orientation.

  • Cortical Fiber Bundle: They model the visual cortex as a fiber bundle where each point in the 2D retinal plane is associated with a fiber representing different orientations. For stereo vision, this is extended to a "stereo fiber bundle" that incorporates information from both eyes.
  • Sub-Riemannian Geometry: Instead of standard Euclidean geometry, they use sub-Riemannian geometry, which imposes constraints on movement within this cortical space. This reflects the idea that neural processing is not arbitrary but follows specific pathways.
  • Integral Curves: The core of their model involves "integral curves" within this sub-Riemannian space. These curves represent the flow of information and are constrained by the geometry, naturally encoding concepts like curvature and torsion. These curves are proposed to be the neural correlate of "association fields" that link corresponding points.
  • Binocular Energy Model: They utilize a simplified binocular energy model as a starting point for modeling binocular neuron responses, which is then integrated into their geometric framework.

Datasets

Not discussed in the paper. The paper focuses on theoretical modeling and comparison with existing psychophysical and biological data rather than introducing new datasets for training or evaluation.

Key Insights and Ideas

  • 3D Good Continuation is a Generalization: The paper demonstrates that good continuation in 3D depth is a natural extension of good continuation in 2D planes, requiring a richer geometric framework.
  • Sub-Riemannian Geometry as a Neural Correlate: The choice of sub-Riemannian geometry is motivated by biological constraints (e.g., orientation selectivity, long-range horizontal connections) and provides a mathematically rigorous way to model these interactions.
  • Integral Curves as Association Fields: The integral curves within the proposed geometric space directly correspond to the concept of association fields, which are crucial for perceptual organization and linking visual elements.
  • Implicit Cortical Architecture: The model suggests that the neural architecture supporting stereo vision might be implicitly organized through these connections, rather than requiring explicit "stereo columns."
  • Curvature and Torsion as Key Features: The geometry naturally incorporates curvature and torsion, which are shown to be important for distinguishing true correspondences from false ones.

Key Results

  • The proposed neurogeometric model provides a formal framework for extending 2D good continuation to 3D stereo vision.
  • The model offers a computational explanation for how the visual system might establish correspondences by leveraging geometric constraints and the concept of integral curves.
  • The integral curves derived from the sub-Riemannian structure are shown to locally connect 3D relatable edges, aligning with psychophysical findings on perceptual organization.
  • The model suggests that high curvature and torsion values can be used to identify and discard false matches in stereo correspondence.

Limitations and Future Work

  • Biological Data: While motivated by biological data, the paper acknowledges that direct evidence for "stereo columns" analogous to orientation columns is weaker, suggesting an implicit organization.
  • Computational Complexity: The paper focuses on the theoretical framework. Future work will involve developing the model further to solve the stereo correspondence problem explicitly.
  • Probability of Co-occurrence: The authors suggest that defining the probability of co-occurrence between elements and developing a similar harmonic analysis for 3D stimuli would be a valuable future direction.
  • Torsional Eye Movements: The formal theory has direct implications for understanding torsional eye movements, which is explored in a companion paper.

Summary updated

BibTeX
@article{bolelli2023good,
  author = {Maria Virginia Bolelli and Giovanna Citti and Alessandro Sarti and Steven W. Zucker},
  title = {{Good continuation in 3D: the neurogeometry of stereo vision}},
  journal = {Frontiers of Computer Science},
  year = {2023},
  doi = {10.3389/fcomp.2023.1142621},
}
Figure