Towards Adversarially Robust and Domain Generalizable Stereo Matching by Rethinking DNN Feature Backbones

Kelvin Cheng, Christopher G. Healey, Tianfu WuarXiv 2021 · arXiv

Introduction

Stereo matching—the process of calculating depth by comparing two slightly offset images—is a fundamental component of autonomous driving systems. While Deep Neural Networks (DNNs) have significantly improved accuracy on standard benchmarks, they have introduced two critical vulnerabilities: adversarial fragility and poor domain generalization. Tiny, imperceptible pixel perturbations can lead to catastrophic depth estimation failures, and models trained on synthetic data often fail to translate to real-world environments (the "Sim2Real" gap).

Researchers from NC State argue that these issues stem from the learnable feature backbones used in modern architectures. These backbones often overfit to specific textures and are highly sensitive to the gradient-based noise used in adversarial attacks.

Rethinking the Feature Backbone

To address these vulnerabilities, the authors propose a hybrid approach that replaces complex, learnable feature extractors with a classic, non-parametric computer vision technique: the Census Transform.

Proposed Methodology

Instead of training a network to learn pixel descriptors, the model uses a Multi-Scale Census Transform to represent pixels by comparing their intensity to their neighbors, creating a Local Binary Pattern. This method is inherently robust to lighting changes and is non-differentiable, making it significantly harder for mathematical attacks to exploit.

The proposed architecture combines hand-crafted Census Transform features with a learnable 3D Hourglass optimizer to generate robust disparity maps.

While the matching features are hand-crafted, the model still utilizes a deep 3D "Hourglass" sub-network. This component acts as a learnable optimizer that searches the cost volume to find the best disparities. Additionally, a small DNN branch can be used to extract high-level semantic "context" from the reference image without compromising the core robustness of the matching process.

Experimental Results

The model was evaluated on industry-standard benchmarks including SceneFlow (synthetic), KITTI 2012/2015 (real-world automotive), and Middlebury (indoor).

Superior Generalization

When trained on synthetic data and tested on real-world KITTI 2015 data without any fine-tuning, the proposed method achieved an End-Point Error (EPE) of 1.25. This vastly outperformed state-of-the-art models like PSMNet (6.89) and GANet (1.66), demonstrating that the Census Transform's lack of texture-dependency allows it to bridge the domain gap effectively.

Adversarial Resilience

To test security, the authors developed a "Stereo-Constrained PGD Attack" that ensures perturbations are physically consistent across both images. Under this realistic threat model ($\epsilon=0.06$), the error rate for the popular GANet-Deep model spiked to 89.48%, whereas the proposed method remained stable at 36.42%.

Qualitative comparison of adversarial robustness. While GANet-Deep suffers massive errors (red in the error map) under attack, the proposed method maintains a much cleaner disparity estimation.

Conclusion and Future Work

By returning to non-parametric features for matching, this research demonstrates that we can achieve significant gains in robustness and generalization with minimal impact on peak performance. While there is a slight accuracy trade-off compared to the absolute top of the KITTI leaderboard, the gain in reliability for safety-critical applications like autonomous driving is substantial.

Future work will focus on optimizing the 3D Hourglass architecture to improve real-time performance and exploring further hyperparameter tuning to bridge the remaining accuracy gap on clean datasets.

Summary updated

Abstract

Stereo matching has recently witnessed remarkable progress using Deep Neural Networks (DNNs). But, how robust are they? Although it has been well-known that DNNs often suffer from adversarial vulnerability with a catastrophic drop in performance, the situation is even worse in stereo matching. This paper first shows that a type of weak white-box attacks can overwhelm state-of-the-art methods. The attack is learned by a proposed stereo-constrained projected gradient descent (PGD) method in stereo matching. This observation raises serious concerns for the deployment of DNN-based stereo matching. Parallel to the adversarial vulnerability, DNN-based stereo matching is typically trained under the so-called simulation to reality pipeline, and thus domain generalizability is an important problem. This paper proposes to rethink the learnable DNN-based feature backbone towards adversarially-robust and domain generalizable stereo matching by completely removing it for matching. In experiments, the proposed method is tested in the SceneFlow dataset and the KITTI2015 benchmark, with promising results. We compute the matching cost volume using the classic multi-scale census transform (i.e., local binary pattern) of the raw input stereo images, followed by a stacked Hourglass head sub-network solving the matching problem. It significantly improves the adversarial robustness, while retaining accuracy performance comparable to state-of-the-art methods. It also shows better generalizability from simulation (SceneFlow) to real (KITTI) datasets when no fine-tuning is used.
BibTeX
@misc{cheng2021towards,
  author = {Kelvin Cheng and Christopher G. Healey and Tianfu Wu},
  title = {{Towards Adversarially Robust and Domain Generalizable Stereo Matching by Rethinking DNN Feature Backbones}},
  year = {2021},
  eprint = {2108.00335},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2108.00335},
}
Figure