Towards Understanding Adversarial Robustness of Optical Flow Networks

Simon Schrodi, Tonmoy Saikia, Thomas Brox

Introduction

Optical flow estimation—the process of determining pixel-level motion between consecutive frames—is foundational for autonomous navigation and robotics. However, recent studies have exposed a critical vulnerability: small adversarial patches can induce catastrophic, global errors in flow predictions. This paper investigates the root cause of this fragility, identifying the "aperture problem" and limited receptive fields as the primary culprits.

The Aperture Problem and Architectural Vulnerability

The authors hypothesize that adversarial patches exploit the classic "aperture problem" in computer vision. When a motion-estimating network views a scene through a small "aperture" (its receptive field), self-similar or ambiguous patterns can lead to incorrect motion vectors. If the network's receptive field is smaller than the adversarial patch, it lacks the context needed to distinguish the patch's artificial motion from the true scene motion.

Heatmaps showing the spatial distribution of End-Point Error (EPE) across different architectures under attack. Note how errors propagate far beyond the patch location in vulnerable models like FlowNetC.

This vulnerability is most pronounced at the correlation layer. If the encoder fails to provide sufficient spatial context, the correlation layer generates ambiguous signals that the decoder subsequently amplifies, resulting in the widespread flow distortions seen in the heatmaps above.

Methodology: Building Robustness

To validate their theory, the researchers developed Robust FlowNetC. While the original FlowNetC utilizes a shallow encoder with large 5x5 kernels, the robust variant employs a deeper encoder with multiple 3x3 kernels. This architectural shift significantly expands the network's receptive field before the correlation layer, allowing it to "see" the stable environment surrounding an adversarial patch.

The team also utilized t-SNE visualizations and Maximum Mean Discrepancy (MMD) to track how feature distributions change under attack, confirming that increased context prevents the "poisoning" of features entering the correlation bottleneck.

Key Results

The results demonstrate that architectural context is a powerful defense. Under a standard adversarial patch attack, the original FlowNetC’s End-Point Error (EPE) increased from 11.50 to 52.66. In contrast, Robust FlowNetC maintained an EPE of 9.60, effectively ignoring the patch's influence on the rest of the scene.

Comparison of optical flow predictions under attack. Robust FlowNetC successfully recovers the scene flow, whereas the standard FlowNetC produces significant artifacts.

Further experiments confirmed this finding: when the robust RAFT architecture was modified to use the shallow FlowNetC encoder, it immediately became vulnerable to the same patch attacks. This confirms that the receptive field, rather than the specific decoder type, is the primary line of defense.

Conclusion

This research shifts the focus from general "encoder-decoder" weaknesses to specific architectural bottlenecks. By addressing the aperture problem through expanded receptive fields, optical flow networks can be made significantly more resilient to physical-world adversarial attacks, a crucial step for the safety of autonomous systems. Future work will explore the remaining challenges of rotational errors and the theoretical limits of universal adversarial perturbations in motion estimation.

Summary updated

Abstract

Recent work demonstrated the lack of robustness of optical flow networks to physical patch-based adversarial attacks. The possibility to physically attack a basic component of automotive systems is a reason for serious concerns. In this paper, we analyze the cause of the problem and show that the lack of robustness is rooted in the classical aperture problem of optical flow estimation in combination with bad choices in the details of the network architecture. We show how these mistakes can be rectified in order to make optical flow networks robust to physical patch-based attacks. Additionally, we take a look at global white-box attacks in the scope of optical flow. We find that targeted white-box attacks can be crafted to bias flow estimation models towards any desired output, but this requires access to the input images and model weights. However, in the case of universal attacks, we find that optical flow networks are robust. Code is available at https://github.com/lmb-freiburg/understanding_flow_robustness.
BibTeX
@inproceedings{schrodi2021towards,
  author = {Simon Schrodi and Tonmoy Saikia and Thomas Brox},
  title = {{Towards Understanding Adversarial Robustness of Optical Flow Networks}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2021},
}
Figure