Triple-SGM: Stereo Processing using Semi-Global Matching with Cost Fusion

Jan Kallwies, Torsten Engler, Bianca Forkel, Hans-Joachim WuenscheWACV 2020

TL;DR

This paper introduces Triple-SGM, a novel stereo vision algorithm that fuses cost information from three cameras (two horizontal pairs and one vertical pair) to improve disparity estimation. By fusing costs early in the pipeline, Triple-SGM overcomes limitations of traditional stereo, particularly in handling challenging structures and occlusions, leading to more accurate and robust 3D reconstructions.

What Problem Does This Paper Address?

Accurate 3D reconstruction is crucial for many technical systems, and stereo vision is a popular method due to its high spatial resolution and readily available color information. However, traditional stereo vision systems suffer from two main drawbacks: accuracy degrades with distance, and the "aperture problem" (difficulty in perceiving depth from repetitive or textureless structures) leads to incorrect or missing measurements. While using more cameras can help, simply fusing disparity maps from multiple stereo pairs late in the process can still be susceptible to local optima and increases computation time. This paper addresses the need for a more robust and efficient stereo vision system that can overcome these limitations by leveraging information from multiple camera viewpoints.

Methods

The core of the proposed method, Triple-SGM, extends the popular Semi-Global Matching (SGM) algorithm. The key innovation lies in fusing the matching costs from two stereo pairs: a standard horizontal pair and a newly introduced vertical pair.

Here's a breakdown of the novelties:

  • Trinocular Rectification: The paper adapts existing methods to rectify three cameras with arbitrary baselines, ensuring that epipolar lines are aligned with image axes. This is crucial for combining information from different camera pairs.
  • Cost-Level Fusion: Instead of fusing disparity maps late in the process, Triple-SGM fuses the matching costs from the horizontal and vertical stereo pairs at an early stage. This is achieved using cubic spline interpolation to smoothly combine cost values, especially near image boundaries.
  • Advanced Weighting Strategy: A specific weighting strategy is proposed for cost values near the left/bottom image boundaries to handle occlusions and improve robustness.
  • Efficient Cost Aggregation: The fused costs are then fed into the SGM algorithm for cost aggregation and disparity estimation. This approach avoids redundant computations and allows for a single cost aggregation step, saving significant processing time compared to running SGM separately for each pair and then fusing disparities.
  • Census Transform: For calculating the initial matching costs, the paper utilizes the census transform, which is known for its robustness to illumination changes and its efficiency.

Datasets

  • Synthetic Dataset: The authors created a synthetic dataset using the CARLA autonomous driving simulator. This dataset consists of 2000 scenes with three camera views and corresponding ground truth disparity images. This dataset is made publicly available to encourage further research in triple stereo algorithms.
  • Real-World Data: The algorithm was also evaluated on real-world scenes using data from a LiDAR sensor (Velodyne HDL-32E) for comparison.

Key Insights and Ideas

  • Early Fusion is Key: Fusing matching costs before the computationally intensive cost aggregation step in SGM is more efficient and leads to better results than late disparity fusion. This is because it allows for a single, unified cost aggregation process.
  • Vertical Stereo Complements Horizontal: A vertical stereo pair, with a baseline perpendicular to the horizontal one, effectively captures structures that are problematic for horizontal stereo (e.g., horizontal edges, periodic structures).
  • Cost Fusion Handles Occlusions Better: By combining information from multiple viewpoints at the cost level, Triple-SGM can recover depth information in regions that would be occluded in a single stereo pair.
  • Cubic Splines for Smoothness: Using cubic Hermite splines for cost interpolation is crucial for avoiding "pixel-locking" artifacts that can arise from simpler interpolation methods when scaling disparities between different baselines.

Key Results

  • Improved Accuracy: Triple-SGM demonstrates a significant increase in correct disparity pixels. For instance, on their synthetic dataset, a median of 88% of pixels have a disparity error below 2 px, compared to 82% for a traditional horizontal stereo pair.
  • Superior to Disparity Fusion: The proposed cost fusion method outperforms disparity fusion [17] across various thresholds and metrics, achieving higher percentages of correct pixels.
  • Robustness to Challenging Scenes: Qualitative results show significant improvements in handling low-texture objects, periodic structures, and object outlines. The boom barrier example highlights the algorithm's ability to detect objects that are missed or poorly reconstructed by traditional stereo methods.
  • Efficiency: Despite processing three cameras, the matching cost fusion step adds only 7.2 ms to the total runtime, resulting in a total runtime of 25 ms for Triple-SGM, compared to 17 ms for standard dual-camera SGM. This is a significant advantage over disparity fusion methods that require running SGM twice.

Limitations and Future Work

  • Vertical Baseline Limitation: The paper notes that the accuracy of the vertical stereo rig is worse than the horizontal one due to its smaller baseline. A higher vertical baseline would mitigate this issue.
  • Real-World Localization Accuracy: The worse results on real-world data compared to synthetic data are attributed to potential inaccuracies in camera localization within the accumulated point cloud.
  • Future Work: The authors suggest that a higher vertical baseline could further improve results. The availability of the synthetic dataset is intended to facilitate further comparisons and development of triple stereo algorithms.

Summary updated

BibTeX
@inproceedings{kallwies2020triple,
  author = {Jan Kallwies and Torsten Engler and Bianca Forkel and Hans-Joachim Wuensche},
  title = {{Triple-SGM: Stereo Processing using Semi-Global Matching with Cost Fusion}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2020},
}
Figure