Gaussian Mixture based Evidential Learning for Stereo Matching

Weide Liu, Xingxing Wang, Lu Wang, Jun Cheng, Fayao Liu, Xulei YangarXiv 2024 · DOI · arXiv

Abstract

Stereo matching is a fundamental task in computer vision, yet traditional deep learning models often struggle with reliable uncertainty estimation. This paper presents a novel evidential learning framework that replaces the standard single-Gaussian assumption with a Mixture-of-Gaussian (MoG) distribution. By modeling pixels through multiple potential distributions and utilizing an Expectation-Maximization (EM) approach, the proposed method achieves state-of-the-art depth estimation and robust uncertainty quantification, particularly in complex cross-domain scenarios.

Introduction

Stereo matching—the process of calculating depth by comparing two images—is critical for autonomous systems. However, existing models frequently fail to "know what they don't know." Traditional Evidential Deep Learning (EDL) attempts to address this by placing priors on model predictions, but typically assumes a single Gaussian distribution for every pixel. In reality, image data is often multimodal; for instance, a pixel near an object boundary may represent foreground, background, or noise. Forcing these complex realities into a single "bell curve" results in imprecise depth maps and unreliable uncertainty estimates, which can be hazardous in safety-critical applications.

Methodology: Mixture-of-Gaussian Evidential Learning (MOG EL)

To overcome the limitations of single-Gaussian models, the authors propose the Mixture-of-Gaussian based Evidential Learning (MOG EL) framework. Instead of a single prediction, the network estimates parameters for multiple Gaussian components that could explain the observed data.

Overview of the MOG EL framework, showing the neural network pipeline and parameter estimation for uncertainty quantification.

Key Technical Components

  • Probabilistic Hierarchy: The framework employs a Normal-Inverse-Gamma (NIG) distribution as a higher-order prior for each component in the mixture. This enables the separate estimation of aleatoric uncertainty (inherent data noise) and epistemic uncertainty (model-related uncertainty).
  • Expectation-Maximization (EM) Training: To handle the non-convex optimization challenges inherent in MoG models, an EM-style approach is integrated directly into the neural network's training loop.
  • Architecture: Built upon the STTR (Stereo Transformer) backbone, the model is augmented with five specialized heads to predict the necessary hyperparameters: $\gamma, \nu, \alpha, \beta, \pi$.
  • Evidence Penalty: An "incorrect evidence penalty" is applied during training to mitigate overconfidence in erroneous predictions.

Experimental Setup and Datasets

The model's performance was rigorously evaluated across several industry-standard benchmarks:

  • Scene Flow (FlyingThings 3D): A large-scale synthetic dataset for primary training and in-domain validation.
  • KITTI 2015: Real-world driving sequences used to assess generalization in high-stakes outdoor environments.
  • Middlebury 2014: High-resolution indoor stereo pairs for cross-domain evaluation.

Key Insights: The Power of Multimodality

The transition to a multimodal approach provides several distinct advantages. By allowing for $K$ Gaussian components (with $K=20$ found to be optimal), the model can effectively segment objects in depth space, leading to significantly sharper object edges.

Illustration of the Mixture-of-Gaussian component selection process for handling complex, multimodal pixel data.

Furthermore, the model demonstrates exceptional zero-shot generalization. Because it is "aware" of its own uncertainty, it avoids overfitting on synthetic training data, resulting in superior performance on unseen real-world datasets. The framework also exhibits enhanced adversarial robustness; under attacks like FGSM, the model flags perturbed areas with high uncertainty rather than providing confident, incorrect answers.

Results and Performance

The MOG EL framework established new state-of-the-art (SOTA) benchmarks:

  • Accuracy: On Scene Flow, it achieved an End-Point Error (EPE) of 0.31, a 25% improvement over the previous SOTA (ELFNet).
  • Cross-Domain Performance: When transferred from synthetic data to the real-world Middlebury dataset, EPE was reduced from 2.33 to 1.30, a 44.2% improvement.
  • Efficiency: The computational overhead introduced by the MoG heads is marginal, making it a practical enhancement for existing architectures.

Conclusion and Future Directions

While MOG EL represents a significant leap forward, several avenues for future research remain. Currently, the number of Gaussian components ($K$) is a fixed hyperparameter; future iterations could utilize Dirichlet Process mixtures for dynamic component determination. Additionally, while cross-domain results are strong, the synthetic-to-real gap persists under extreme lighting or weather conditions. Finally, further optimization of the EM-based loss calculation could enhance performance in ultra-low-latency environments.

Summary updated

Abstract

In this paper, we introduce a novel Gaussian mixture based evidential learning solution for robust stereo matching. Diverging from previous evidential deep learning approaches that rely on a single Gaussian distribution, our framework posits that individual image data adheres to a mixture-of-Gaussian distribution in stereo matching. This assumption yields more precise pixel-level predictions and more accurately mirrors the real-world image distribution. By further employing the inverse-Gamma distribution as an intermediary prior for each mixture component, our probabilistic model achieves improved depth estimation compared to its counterpart with the single Gaussian and effectively captures the model uncertainty, which enables a strong cross-domain generation ability. We evaluated our method for stereo matching by training the model using the Scene Flow dataset and testing it on KITTI 2015 and Middlebury 2014. The experiment results consistently show that our method brings improvements over the baseline methods in a trustworthy manner. Notably, our approach achieved new state-of-the-art results on both the in-domain validated data and the cross-domain datasets, demonstrating its effectiveness and robustness in stereo matching tasks.
BibTeX
@misc{liu2024gaussian,
  author = {Weide Liu and Xingxing Wang and Lu Wang and Jun Cheng and Fayao Liu and Xulei Yang},
  title = {{Gaussian Mixture based Evidential Learning for Stereo Matching}},
  year = {2024},
  eprint = {2408.02796},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2408.02796},
}
Figure