Leveraging Confident Points for Accurate Depth Refinement on Embedded Systems

Fabio Tosi, Matteo Poggi, Stefano Mattoccia

Abstract

Stereo vision is a cornerstone of autonomous navigation, yet the high computational demands of state-of-the-art deep learning models often preclude their use on resource-constrained embedded systems. This paper presents Non-Local Anchoring (NLA), a lightweight, hardware-agnostic framework designed to refine noisy depth maps. By utilizing a Random Forest classifier to automatically identify reliable pixels and a non-local weighted median filter for reconstruction, NLA significantly improves stereo accuracy without the power overhead of convolutional neural networks (CNNs). Evaluations on the Middlebury and KITTI datasets demonstrate that NLA effectively bridges the gap between traditional block-matching algorithms and complex deep-learning refinement strategies.

Introduction

Calculating depth through stereo vision—comparing two offset images—is essential for robotics and drones. While modern CNNs achieve high accuracy, they are often impractical for mobile processors due to their massive power consumption. Furthermore, many commercial-off-the-shelf (COTS) stereo cameras provide "black box" disparity maps, hiding the internal raw matching scores (cost volumes) required by standard refinement techniques. There is a critical need for a refinement method that is fast, automated, and compatible with any stereo camera output.

The Non-Local Anchoring (NLA) Framework

The NLA pipeline operates entirely in the disparity domain, requiring only the initial noisy depth map and the original reference image. It consists of two primary stages:

  1. Threshold-free Reliable Point (RP) Selection: Instead of relying on hand-tuned sensitivity thresholds, NLA uses a Random Forest classifier. This model analyzes local and global features of a confidence map to automatically categorize pixels as either Reliable Points (RP) or Unreliable Points (UP).
  2. Non-Local Inference and Refinement: For every unreliable pixel, the algorithm searches in up to 16 directions to find the nearest reliable anchors. The final depth value is calculated using a weighted median of these anchors, where weights are determined by spatial distance and color similarity to ensure depth edges align with object boundaries.
Visual comparison of the NLA refinement process on indoor scenes from the Middlebury dataset. Columns show (a) original image, (b) initial noisy disparity, (c) identified reliable points, and (d) final refined output.

Experimental Evaluation

The framework was evaluated on the Middlebury v3 (indoor) and KITTI 2015 (outdoor) benchmarks. The Random Forest classifier was trained on small subsets of these datasets to ensure generalization across diverse environments.

Key Results

  • Accuracy Improvement: On the Middlebury dataset, NLA reduced the "bad pixel" rate (errors > 1px) of a standard Block-Matching algorithm from 35.13% to 22.90%.
  • Outdoor Robustness: In the challenging KITTI dataset, NLA successfully reconstructed depth in areas affected by complex lighting and motion, outperforming traditional filters like the Bilateral or Weighted Median filters.
  • Embedded Performance: Tested on an NVIDIA Jetson TX2, the core NLA algorithm executed in 1.82 seconds on the CPU. While not yet real-time for high-speed flight, it is significantly more efficient than running full-scale deep-learning stereo networks on the same hardware.
Application of NLA to the KITTI 2015 outdoor dataset, demonstrating the framework's ability to handle real-world road scenes and motion noise.

Discussion and Future Work

NLA's primary advantage is its "agnostic" nature; it can be integrated as a post-processing step for any stereo system. By replacing heuristic thresholds with machine learning, the system becomes self-tuning, adapting to the specific noise characteristics of different sensors automatically.

However, the performance remains tied to the quality of the initial confidence measure. Future research will focus on optimizing the local aggregation steps—potentially through GPU acceleration—to achieve real-time frame rates suitable for high-speed autonomous robotics.

Summary updated

Abstract

Despite the notable progress in stereo disparity estimation, algorithms are still prone to errors in challenging conditions. Thus, heuristic disparity refinement techniques are usually deployed to improve accuracy. Moreover, state-of-the-art methods rely on complex CNNs requiring power hungry GPUs not suited for many practical applications constrained by limited computing resources. In this paper, we propose a novel technique for disparity refinement leveraging on confidence measures and a novel, automatic learning-based selection method to discard outliers. Then, a non-local strategy infers missing disparities by analyzing the closest reliable points. This framework is very fast and does not require any hand-tuned thresholding. We assess the performance of our Non-Local Anchoring (NLA), standalone refinement techniques and methods leveraging on confidence measures inside the stereo algorithm. Our evaluation with two popular stereo algorithms shows that our proposal significantly ameliorates their accuracy on Middlebury v3 and KITTI 2015 datasets. Moreover, since our method relies only on cues computed in the disparity domain, it is suited even for COTS stereo cameras coupled with embedded systems, e.g. nVidia Jetson TX2.
BibTeX
@inproceedings{tosi2019leveraging,
  author = {Fabio Tosi and Matteo Poggi and Stefano Mattoccia},
  title = {{Leveraging Confident Points for Accurate Depth Refinement on Embedded Systems}},
  booktitle = {Workshops @ CVPR},
  year = {2019},
  doi = {10.1109/cvprw.2019.00025},
}
Figure