Introduction
In binocular stereo vision, the primary goal is to calculate "disparity"—the horizontal shift of objects between two different camera views—to infer depth. However, accurately estimating this depth faces two major hurdles: discontinuities and occlusions.
While depth is generally smooth across a continuous surface, it jumps abruptly at the edges of objects (discontinuities). Standard smoothing models often erroneously "blur" these sharp edges. Furthermore, some areas are visible to one eye but blocked by a foreground object for the other (occlusions). In these occluded regions, disparity is technically undefined.
Traditional high-performing methods, such as those based on Graph Cuts, are often deterministic and computationally heavy. To address this, this paper introduces a probabilistic framework based on a "switched" Gaussian Process (GP). This model simultaneously estimates depth and identifies foreground/background layers, effectively handling sharp depth discontinuities and occlusions while remaining flexible enough to incorporate new data streams.
The Switched Gaussian Process
The core innovation of the paper is the Switched Gaussian Process, which allows the model's behavior to adapt based on a pixel's segmentation label.
The author defines a GP prior for the disparity map where the mean and covariance functions depend on a latent segmentation label: Foreground (F), Background (B), or Occluded (O).
- Conditional Smoothness: If two points share a label (both F or both B), the model expects their depths to be spatially correlated, enforcing smoothness.
- Sharp Jumps: If the points have different labels, their correlation is set to zero. This allows the model to be locally smooth but globally sharp, preserving crisp object boundaries.
- Handling Occlusions: Occluded points are treated mathematically as independent noise, preventing them from corrupting the depth estimates of visible surfaces.
Instead of solving the entire image simultaneously, the algorithm employs Incremental Greedy Labeling. It adds observations one by one, calculating the marginal likelihood (evidence) for all three labels and greedily selecting the one that fits best.
Active Learning and Sparse Sampling
A well-known limitation of Gaussian Processes is their $O(n^3)$ computational complexity, which scales poorly with large numbers of pixels. To circumvent this, the author introduces an Active Learning observation schedule.
Rather than processing every pixel or relying on independent horizontal scanlines (which creates "streaky" artifacts by ignoring vertical context), the model uses an entropy-based criterion to select only the most "informative" pixels. It prioritizes areas where the model is most uncertain, which naturally cluster around the tricky boundaries between objects.
Remarkably, by using active learning, the model achieves high-quality results by observing only 1.3% of the pixels (roughly 1,000 out of 76,800 pixels in a standard test image). Because the GP provides a measure of uncertainty (variance), it also offers a principled way to fuse stereo data with other sensors. If the stereo match is "blurry" or lacks texture, the variance increases, prompting the model to rely more heavily on fused color histogram data via a Viterbi algorithm.
Results and Performance
The model was evaluated using 320 x 240 pixel stereo image pairs from the Microsoft Research "i2i" project, comparing the results against hand-labeled ground truth segmentations.
The active learning approach successfully removed the streaky artifacts common in simpler scanline methods. For small numbers of points ($n=200$), the system proved extremely fast, processing a pair of images in about 30 milliseconds.
When fused with color information, the model achieved a highly competitive labeling error rate of 0.58%. This performance is remarkably close to the 0.25% error rate of the then-state-of-the-art Graph Cut method, but it provides the significant added benefit of a fully probabilistic output.
Limitations and Future Work
Despite the success of sparse sampling, the inherent $O(n^3)$ scaling of Gaussian Processes remains a bottleneck for higher-resolution images. The author suggests that approximate GP techniques will be necessary to scale the framework further.
Additionally, the current method approximates the stereo matching cost as a simple parabola. More sophisticated techniques, such as Expectation Propagation (EP), could provide a more accurate probabilistic fit. Finally, the color fusion relied on a basic histogram model; upgrading to a more sophisticated color model could likely close the remaining accuracy gap with state-of-the-art deterministic methods.
Summary updated