Confidence driven TGV fusion

Valsamis Ntouskos, Fiora PirriarXiv 2016 · arXiv

Abstract

This paper introduces a novel variational framework for fusing multiple noisy data sources, such as depth maps, by jointly estimating the fused image and a spatially varying confidence map. By dynamically adapting to local noise and outliers, the model preserves sharp edges and fine details that are often lost in traditional global-parameter models. Evaluation on synthetic datasets and the KITTI stereo benchmark demonstrates that this approach achieves state-of-the-art performance in 3D reconstruction by effectively letting the data judge its own accuracy.

Introduction

In computer vision, the fusion of multiple noisy observations into a single, coherent representation is a fundamental challenge. A common application is the integration of several depth maps—captured from different camera angles—into a single accurate 3D surface. Traditionally, this is addressed using variational regularization techniques, such as Total Variation (TV) or Total Generalized Variation (TGV). These models balance a fidelity term, which ensures the result matches the raw data, and a regularization term, which enforces smoothness.

However, standard models typically rely on a global trade-off parameter. This assumes uniform noise across the entire dataset, which is rarely the case in real-world scenarios. A depth sensor may be highly precise on flat, matte surfaces but fail significantly on sharp edges or reflective windows. A global parameter forces a compromise: it either over-smooths high-quality data or fails to suppress noise in low-quality regions. This paper addresses this limitation by proposing a model that automatically determines where to trust the data and where to rely on smoothing, pixel by pixel.

The Confidence-Driven TGV Framework

The authors propose a "confidence-driven" TGV fusion model. The core innovation is the introduction of a spatially varying confidence matrix, $\Lambda$, which weights the fidelity term. Unlike traditional methods where weights are pre-calculated or fixed, this model employs joint estimation: it simultaneously optimizes the fused depth map and the confidence map based on the spatial coherence of the input data.

Robust Fidelity and Scale-Awareness

The model utilizes an $L_1$ norm for the fidelity term. In the context of depth map fusion, the $L_1$ norm is significantly more robust to outliers—such as the sudden, erroneous depth jumps often found at object boundaries—than the standard $L_2$ norm. This choice prevents the "contrast distortion" typically seen in quadratic models.

Furthermore, this spatially varying approach enables scale-aware smoothing. In uniform regularization models, small objects often disappear before the noise on larger structures is fully addressed. By allowing the confidence map to adapt, small, high-confidence structures are preserved while large, noisy areas receive the heavy smoothing they require.

Optimization Strategy

The joint estimation of the image and the confidence map results in a biconvex optimization problem. While the problem is convex if either the image or the confidence map is held constant, it is not jointly convex. To solve this, the authors adapted and proved the convergence for three specific optimization algorithms:

  1. Alternating Convex Search (ACS)
  2. Alternate Minimization (AMA)
  3. Primal-Dual Hybrid Gradient (PDHG)

To improve the initialization of the confidence map, the authors also introduced geometric priors. For instance, a surface facing the camera head-on is assigned a higher initial confidence than a surface viewed at a steep, glancing angle. This "hint" from the scene geometry significantly accelerates convergence and improves final accuracy.

Experimental Evaluation

Synthetic Benchmarks

The model was evaluated using several synthetic datasets, including the Stanford 3D scanning repository (Bunny, Dragon) and Urban Landscapes. The adaptive model utilizing the geometric prior consistently achieved the lowest Root Mean Square Error (RMSE) and Mean Absolute Error (MAE). Visually, the method outperformed standard TGV fusion by maintaining sharp discontinuities while effectively smoothing planar surfaces.

Real-World Performance: KITTI Dataset

The framework was tested on the KITTI stereo benchmark, a standard for autonomous driving vision. The fusion model improved the accuracy of disparity maps over baseline algorithms like SGM and ELAS. Most notably, when paired with disparity maps generated by a top-tier Convolutional Neural Network (CNN), the fusion process further reduced errors by 3.14% in non-occluded regions and by 5.76% in highly noisy reflective regions. At the time of publication, this approach established a new state-of-the-art on the KITTI benchmark.

Conclusion and Future Work

Confidence-driven TGV fusion represents a significant step forward in robust data integration. By allowing the model to locally adapt to noise levels, it overcomes the fundamental limitations of global regularization. While the biconvex nature of the problem means a global optimum is not mathematically guaranteed, the results on both synthetic and real-world data demonstrate superior performance over existing methods.

Future research will focus on deepening the theoretical convergence analysis of the PDHG algorithm for biconvex problems, particularly in high signal-to-noise ratio scenarios. Additionally, the authors suggest extending this work to Total Variation regularization directly on manifolds to further improve the consistency of 3D surfaces generated from multi-view imagery.

Summary updated

Abstract

We introduce a novel model for spatially varying variational data fusion, driven by point-wise confidence values. The proposed model allows for the joint estimation of the data and the confidence values based on the spatial coherence of the data. We discuss the main properties of the introduced model as well as suitable algorithms for estimating the solution of the corresponding biconvex minimization problem and their convergence. The performance of the proposed model is evaluated considering the problem of depth image fusion by using both synthetic and real data from publicly available datasets.
BibTeX
@misc{ntouskos2016confidence,
  author = {Valsamis Ntouskos and Fiora Pirri},
  title = {{Confidence driven TGV fusion}},
  year = {2016},
  eprint = {1603.09302},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1603.09302},
}
Figure