Abstract
Visual segmentation is a key perceptual function that partitions visual space and allows for detection, recognition and discrimination of objects in complex environments. The processes underlying human segmentation of natural images are still poorly understood. In part, this is because we lack segmentation models consistent with experimental and theoretical knowledge in visual neuroscience. Biological sensory systems have been shown to approximate probabilistic inference to interpret their inputs. This requires a generative model that captures both the statistics of the sensory inputs and expectations about the causes of those inputs. Following this hypothesis, we propose a probabilistic generative model of visual segmentation that combines knowledge about 1) the sensitivity of neurons in the visual cortex to statistical regularities in natural images; and 2) the preference of humans to form contiguous partitions of visual space. We develop an efficient algorithm for training and inference based on expectation-maximization and validate it on synthetic data. Importantly, with the appropriate choice of the prior, we derive an intuitive closed--form update rule for assigning pixels to segments: at each iteration, the pixel assignment probabilities to segments is the sum of the evidence (i.e. local pixel statistics) and prior (i.e. the assignments of neighboring pixels) weighted by their relative uncertainty. The model performs competitively on natural images from the Berkeley Segmentation Dataset (BSD), and we illustrate how the likelihood and prior components improve segmentation relative to traditional mixture models. Furthermore, our model explains some variability across human subjects as reflecting local uncertainty about the number of segments. Our model thus provides a viable approach to probe human visual segmentation.
Introduction
General unsupervised segmentation—partitioning an image into meaningful objects without pre-defined classes—remains a major challenge in computer vision. While deep learning excels at semantic segmentation, it often lacks a grounding in biological reality. Human vision is remarkably adept at grouping features into contiguous objects, even in noisy or ambiguous scenes. This ability is thought to stem from the brain performing probabilistic inference, using internal models of natural image statistics. This paper proposes a generative model consistent with neuroscientific principles, specifically targeting the heavy-tailed statistics of natural images and the spatial contiguity of objects.
Methods and Mathematical Framework
The core of the approach is a generative model trained via Expectation-Maximization (EM). It consists of two primary components:
- Likelihood (The "Evidence"): Instead of standard Gaussian distributions, the model utilizes multivariate Student-t distributions. Natural image features, such as orientations and colors, exhibit "heavy-tailed" statistics, meaning they contain more outliers than a normal distribution. Student-t distributions capture these statistics accurately and align with the responses of neurons in the primary visual cortex (V1).
- Prior (The "Context"): A Dirichlet prior is employed to encourage spatial contiguity. This ensures that a pixel's assignment is influenced by its neighbors, favoring the formation of continuous regions.
The authors derive a closed-form update rule for pixel assignments where the probability of a pixel belonging to a segment is a weighted sum of local evidence and spatial prior. Crucially, these components are weighted by their relative uncertainty. If local data is ambiguous, the model relies more heavily on the surrounding context.
Results on Natural Images
The model was evaluated on the Berkeley Segmentation Dataset (BSD 500), a benchmark containing natural images segmented by multiple human subjects.
As shown in Figure 4, the "Student-t + Dirichlet" model outperformed standard Gaussian Mixture Models (GMMs) and plain Student-t models. The Student-t likelihood provides robustness to outliers that often "stretch" GMM covariances, while the spatial prior effectively removes "salt-and-pepper" noise, leading to much cleaner segments that better match human perception.
Predicting Human Variability
A key contribution of this work is its ability to model human uncertainty. The authors observed that human subjects follow a Weber-Fechner law for segmentation: the variation in the number of segments chosen by different people increases linearly with the average number of segments in the image.
The model’s internal uncertainty (entropy) correlates with this human behavior (Figure 6). This suggests that "disagreements" between human annotators on segment boundaries are not random errors but reflections of local image ambiguity that the probabilistic model successfully captures.
Conclusion
This research bridges the gap between computer vision and computational neuroscience by providing a probabilistic framework for visual segmentation. By incorporating V1-like statistics and human-like spatial priors, the model achieves competitive unsupervised results and offers a mathematical link to human perceptual uncertainty. Future work involves extending the model to multi-scale architectures to better handle complex textures and high-level object features.
Summary updated