Abstract
Visual segmentation is a key perceptual function that partitions visual space and allows for detection, recognition and discrimination of objects in complex environments. The processes underlying human segmentation of natural images are still poorly understood. In part, this is because we lack segmentation models consistent with experimental and theoretical knowledge in visual neuroscience. Biological sensory systems have been shown to approximate probabilistic inference to interpret their inputs. This requires a generative model that captures both the statistics of the sensory inputs and expectations about the causes of those inputs. Following this hypothesis, we propose a probabilistic generative model of visual segmentation that combines knowledge about 1) the sensitivity of neurons in the visual cortex to statistical regularities in natural images; and 2) the preference of humans to form contiguous partitions of visual space. We develop an efficient algorithm for training and inference based on expectation-maximization and validate it on synthetic data. Importantly, with the appropriate choice of the prior, we derive an intuitive closed-form update rule for assigning pixels to segments: at each iteration, the pixel assignment probabilities to segments is the sum of the evidence (i.e. local pixel statistics) and prior (i.e. the assignments of neighboring pixels) weighted by their relative uncertainty. The model performs competitively on natural images from the Berkeley Segmentation Dataset (BSD), and we illustrate how the likelihood and prior components improve segmentation relative to traditional mixture models. Furthermore, our model explains some variability across human subjects as reflecting local uncertainty about the number of segments. Our model thus provides a viable approach to probe human visual segmentation.
Introduction
Image segmentation remains a fundamental challenge in computer vision, particularly in unsupervised settings. While deep learning has advanced semantic segmentation, these models often lack generalizability and do not reflect the biological mechanisms of human vision. A significant gap exists: most models rely on Gaussian statistical assumptions that fail to capture the heavy-tailed nature of natural scene statistics. Moreover, human segmentation is inherently ambiguous, yet existing models rarely account for this perceptual uncertainty.
Methods
We propose a generative model that frames segmentation as Bayesian inference, built upon three pillars:
- V1-Inspired Features: The model utilizes features mimicking primary visual cortex (V1) neurons, specifically color combined with wavelet coefficients representing various orientations and scales.
- Student-t Mixture Model (mGSM): To account for the heavy-tailed statistics of natural images, we employ a Mixture of Gaussian Scale Mixtures—specifically the multivariate Student-t distribution. This approach is more robust to the statistical noise of natural textures than standard Gaussian distributions.
- Dirichlet Prior: To promote spatial contiguity, we introduce a spatial prior based on the Dirichlet distribution. We derive a closed-form update rule where pixel assignments are determined by a weighted balance between local evidence and neighboring assignments, scaled by their relative uncertainty.
The model is optimized using an Expectation-Maximization (EM) algorithm to iteratively refine segment boundaries and statistical parameters.
Results
The "Student-t + Dirichlet" model significantly outperforms standard Gaussian mixtures. On the BSD 500 dataset, it achieved an Adjusted Rand Index (aRI) of 0.461, compared to 0.171 for the Gaussian version.
Qualitatively, the Dirichlet prior effectively removes isolated pixel noise, creating perceptually meaningful segments that align with real-world objects. Furthermore, the model's uncertainty maps serve as accurate contour detectors, bridging region-based and contour-based segmentation.
Our analysis also addresses human variability. We found that human disagreements in segmentation follow the Weber-Fechner Law, where variation in segment count scales with the total number of segments. Our model reflects this behavior, suggesting that human variability is a reflection of local statistical uncertainty.
Conclusion
This model provides a biologically plausible framework for understanding visual segmentation. While it does not yet match specialized state-of-the-art contour detectors due to a lack of high-level semantic knowledge, it offers a robust probabilistic approach. Future work will explore hierarchical extensions to capture higher-level cortical features and multi-scale processing to better handle fine details and large background regions.
Summary updated