Abstract
This paper introduces a novel variational framework for fusing multiple noisy data sources, such as depth maps, by jointly estimating the fused image and a spatially varying confidence map. By dynamically adapting to local noise and outliers, the model preserves sharp edges and fine details that are often lost in traditional global-parameter models. Evaluation on synthetic datasets and the KITTI stereo benchmark demonstrates that this approach achieves state-of-the-art performance in 3D reconstruction by effectively letting the data judge its own accuracy.
Introduction
In computer vision, the fusion of multiple noisy observations into a single, coherent representation is a fundamental challenge. A common application is the integration of several depth maps—captured from different camera angles—into a single accurate 3D surface. Traditionally, this is addressed using variational regularization techniques, such as Total Variation (TV) or Total Generalized Variation (TGV). These models balance a fidelity term, which ensures the result matches the raw data, and a regularization term, which enforces smoothness.
However, standard models typically rely on a global trade-off parameter. This assumes uniform noise across the entire dataset, which is rarely the case in real-world scenarios. A depth sensor may be highly precise on flat, matte surfaces but fail significantly on sharp edges or reflective windows. A global parameter forces a compromise: it either over-smooths high-quality data or fails to suppress noise in low-quality regions. This paper addresses this limitation by proposing a model that automatically determines where to trust the data and where to rely on smoothing, pixel by pixel.
The Confidence-Driven TGV Framework
The authors propose a "confidence-driven" TGV fusion model. The core innovation is the introduction of a spatially varying confidence matrix, $\Lambda$, which weights the fidelity term. Unlike traditional methods where weights are pre-calculated or fixed, this model employs joint estimation: it simultaneously optimizes the fused depth map and the confidence map based on the spatial coherence of the input data.
Robust Fidelity and Scale-Awareness
The model utilizes an $L_1$ norm for the fidelity term. In the context of depth map fusion, the $L_1$ norm is significantly more robust to outliers—such as the sudden, erroneous depth jumps often found at object boundaries—than the standard $L_2$ norm. This choice prevents the "contrast distortion" typically seen in quadratic models.
Furthermore, this spatially varying approach enables scale-aware smoothing. In uniform regularization models, small objects often disappear before the noise on larger structures is fully addressed. By allowing the confidence map to adapt, small, high-confidence structures are preserved while large, noisy areas receive the heavy smoothing they require.
Optimization Strategy
The joint estimation of the image and the confidence map results in a biconvex optimization problem. While the problem is convex if either the image or the confidence map is held constant, it is not jointly convex. To solve this, the authors adapted and proved the convergence for three specific optimization algorithms:
- Alternating Convex Search (ACS)
- Alternate Minimization (AMA)
- Primal-Dual Hybrid Gradient (PDHG)
To improve the initialization of the confidence map, the authors also introduced geometric priors. For instance, a surface facing the camera head-on is assigned a higher initial confidence than a surface viewed at a steep, glancing angle. This "hint" from the scene geometry significantly accelerates convergence and improves final accuracy.
Experimental Evaluation
Synthetic Benchmarks
The model was evaluated using several synthetic datasets, including the Stanford 3D scanning repository (Bunny, Dragon) and Urban Landscapes. The adaptive model utilizing the geometric prior consistently achieved the lowest Root Mean Square Error (RMSE) and Mean Absolute Error (MAE). Visually, the method outperformed standard TGV fusion by maintaining sharp discontinuities while effectively smoothing planar surfaces.
Real-World Performance: KITTI Dataset
The framework was tested on the KITTI stereo benchmark, a standard for autonomous driving vision. The fusion model improved the accuracy of disparity maps over baseline algorithms like SGM and ELAS. Most notably, when paired with disparity maps generated by a top-tier Convolutional Neural Network (CNN), the fusion process further reduced errors by 3.14% in non-occluded regions and by 5.76% in highly noisy reflective regions. At the time of publication, this approach established a new state-of-the-art on the KITTI benchmark.
Conclusion and Future Work
Confidence-driven TGV fusion represents a significant step forward in robust data integration. By allowing the model to locally adapt to noise levels, it overcomes the fundamental limitations of global regularization. While the biconvex nature of the problem means a global optimum is not mathematically guaranteed, the results on both synthetic and real-world data demonstrate superior performance over existing methods.
Future research will focus on deepening the theoretical convergence analysis of the PDHG algorithm for biconvex problems, particularly in high signal-to-noise ratio scenarios. Additionally, the authors suggest extending this work to Total Variation regularization directly on manifolds to further improve the consistency of 3D surfaces generated from multi-view imagery.
Summary updated