LAF-Net: Locally Adaptive Fusion Networks for Stereo Confidence Estimation

Sunok Kim, Seungryong Kim, Dongbo Min, Kwanghoon SohnCVPR 2019 · OpenReview

Abstract

We present a novel method that estimates confidence map of an initial disparity by making full use of tri-modal input, including matching cost, disparity, and color image through deep networks. The proposed network, termed as Locally Adaptive Fusion Networks (LAF-Net), learns locally-varying attention and scale maps to fuse the tri-modal confidence features. The attention inference networks encode the importance of tri-modal confidence features and then concatenate them using the attention maps in an adaptive and dynamic fashion. This enables us to make an optimal fusion of the heterogeneous features, compared to a simple concatenation technique that is commonly used in conventional approaches. In addition, to encode the confidence features with locally-varying receptive fields, the scale inference networks learn the scale map and warp the fused confidence features through convolutional spatial transformer networks. Finally, the confidence map is progressively estimated in the recursive refinement networks to enforce a spatial context and local consistency. Experimental results show that this model outperforms the state-of-the-art methods on various benchmarks.

1. Introduction

Stereo matching—estimating depth by comparing two images—is a cornerstone of computer vision, but it is notoriously brittle. It often fails in challenging regions like reflective surfaces, textureless walls, or areas hidden behind occlusions. To make stereo depth maps useful for downstream tasks such as autonomous driving, a reliable confidence map is required: a per-pixel probability that the estimated depth is correct.

While previous methods relied on handcrafted rules or simple neural networks using limited data modalities, we propose LAF-Net. This architecture intelligently fuses "tri-modal" inputs—matching cost, disparity, and color—to detect depth errors with high precision.

Overview of the LAF-Net framework showing tri-modal inputs and confidence map output.

2. Proposed Method

The LAF-Net architecture consists of four specialized sub-networks designed to handle heterogeneous data and adapt to local image context.

2.1 Feature Extraction Networks

These networks process the tri-modal input. To handle the raw matching cost, which can vary wildly in scale, the network converts it into a "top-K matching probability" volume. This transformation makes the feature extraction invariant to the search range.

2.2 Attention Inference Networks

Instead of simple concatenation, this module learns pixel-wise attention maps. It calculates which modality (cost, disparity, or color) is most informative for a specific pixel. For example, color information might be weighted more heavily near object boundaries to improve edge localization.

2.3 Scale Inference Networks

Standard CNNs use a fixed kernel size, but LAF-Net learns a scale map that allows the network to adaptively adjust its receptive field for each pixel. Implemented using a bilinear sampler, this allows the model to capture the optimal amount of local context—looking at wide areas for flat regions while focusing narrowly on sharp details.

2.4 Recursive Refinement Networks

The confidence map is progressively estimated over several iterations (typically $t=3$). By using the previous estimate as a guide, the network enforces spatial context and local consistency, resulting in smoother and more accurate maps.

Detailed architecture of LAF-Net, comprising Feature Extraction, Attention Inference, Scale Inference, and Recursive Refinement Networks.

3. Experiments and Results

LAF-Net was trained on the MPI Sintel and KITTI 2012 datasets and evaluated on Middlebury 2006/2014 and KITTI 2015 benchmarks. The primary metric used was the Area Under the Curve (AUC) of the sparsification curve, which measures the effectiveness of the confidence ranking.

3.1 Key Insights

  • Tri-modal Superiority: Ablation studies demonstrate that incorporating color information alongside cost and disparity significantly boosts performance by providing vital cues about edges and textures.
  • Adaptive Fusion: The attention mechanism proves that locally-varying importance is key; certain features are better at detecting occlusion errors, while others excel in textureless regions.
  • Efficiency: Despite its adaptive layers, LAF-Net is relatively lightweight with only 1.3M parameters, compared to 9.2M in models like ConfNet.

3.2 Quantitative and Qualitative Performance

LAF-Net outperformed existing state-of-the-art methods across all tested datasets. On the KITTI 2015 dataset, it achieved an AUC of 0.0225, nearing the theoretical optimal limit of 0.0170. Qualitatively, the model excels at identifying mismatched pixels in complex areas, such as thin structures or distant objects, where previous models produced noisy results.

4. Conclusion

LAF-Net sets a new standard for stereo confidence estimation by leveraging tri-modal inputs and locally adaptive fusion mechanisms. While the model provides superior accuracy, future work will address the computational overhead of bilinear sampling and recursive refinement to enable real-time performance. Additionally, exploring unsupervised learning approaches remains a promising direction for improving scalability.

Summary updated

Abstract

We present a novel method that estimates confidence map of an initial disparity by making full use of tri-modal input, including matching cost, disparity, and color image through deep networks. The proposed network, termed as Locally Adaptive Fusion Networks (LAF-Net), learns locally-varying attention and scale maps to fuse the tri-modal confidence features. The attention inference networks encode the importance of tri-modal confidence features and then concatenate them using the attention maps in an adaptive and dynamic fashion. This enables us to make an optimal fusion of the heterogeneous features, compared to a simple concatenation technique that is commonly used in conventional approaches. In addition, to encode the confidence features with locally-varying receptive fields, the scale inference networks learn the scale map and warp the fused confidence features through convolutional spatial transformer networks. Finally, the confidence map is progressively estimated in the recursive refinement networks to enforce a spatial context and local consistency. Experimental results show that this model outperforms the state-of-the-art methods on various benchmarks.
BibTeX
@inproceedings{kim2019laf,
  author = {Sunok Kim and Seungryong Kim and Dongbo Min and Kwanghoon Sohn},
  title = {{LAF-Net: Locally Adaptive Fusion Networks for Stereo Confidence Estimation}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2019},
  url = {https://openreview.net/forum?id=N_nKPjg0yzpI},
}
Figure