Abstract
Recent work demonstrated the lack of robustness of optical flow networks to physical patch-based adversarial attacks. The possibility to physically attack a basic component of automotive systems is a reason for serious concerns. In this paper, we analyze the cause of the problem and show that the lack of robustness is rooted in the classical aperture problem of optical flow estimation in combination with bad choices in the details of the network architecture. We show how these mistakes can be rectified in order to make optical flow networks robust to physical patch-based attacks. Additionally, we take a look at global white-box attacks in the scope of optical flow. We find that targeted white-box attacks can be crafted to bias flow estimation models towards any desired output, but this requires access to the input images and model weights. However, in the case of universal attacks, we find that optical flow networks are robust.
Introduction: The Vulnerability of Motion Estimation
Optical flow—the task of estimating the motion of pixels between two consecutive camera frames—is a cornerstone of autonomous driving and robotics. However, recent studies showed that these networks are remarkably easy to "blind" using physical adversarial patches. A simple sticker placed in a scene can cause a network to perceive massive, non-existent motion, potentially causing a self-driving car to swerve or brake unnecessarily.
While the existence of these vulnerabilities was known, the underlying cause remained elusive. Previous theories often blamed the general encoder-decoder structure, but this research identifies specific architectural flaws—namely, small receptive fields—that allow a localized patch to derail an entire motion field.
Methodology: Diagnosing the Network
To understand why architectures like FlowNetC are so vulnerable, the researchers employed several diagnostic techniques:
- Spatial Heat Maps: By moving an adversarial patch across an image, they mapped the resulting End-Point Error (EPE) to identify regions of high sensitivity.
- Feature Analysis: Using t-SNE visualizations, they compared the feature distributions of clean versus attacked images to see how the network's internal representations were being manipulated.
- Handcrafted Attacks: They designed a "zero-query" patch consisting of high-frequency vertical stripes. This patch exploits the physics of the aperture problem without requiring any gradient-based optimization or model access.
Key Insights: The Aperture Problem and Receptive Fields
The core finding is that the vulnerability is rooted in the classical aperture problem. In computer vision, the aperture problem occurs when a moving pattern is viewed through a small opening (aperture), making it impossible to determine the true direction of motion if the pattern is repetitive or lacks sufficient context.
Adversarial patches exploit this by creating self-similar patterns. In FlowNetC, the receptive field (the "aperture") before the correlation layer is only 31x31 pixels. This small window provides insufficient context to distinguish the patch's artificial pattern from real motion. In contrast, more robust networks like PWC-Net or RAFT utilize much larger receptive fields (up to 631x631), allowing them to see the surrounding "real" world and ignore the local ambiguity of the patch.
Results: Engineering Robustness
By identifying the receptive field size as the primary culprit, the authors developed Robust FlowNetC. By simply adding convolutional layers to increase the receptive field before the correlation layer, they significantly mitigated the attack's impact.
Under a patch attack, the original FlowNetC’s EPE skyrocketed from 11.50 to 52.66. The Robust FlowNetC, however, maintained an EPE of 13.08—nearly matching its performance on clean data. Beyond patch attacks, the study also explored white-box attacks. With access to model weights, the authors could craft imperceptible noise to force the network into "hallucinating" arbitrary patterns across the entire flow field.
Conclusion
This research shifts the focus from general model vulnerability to specific architectural limitations. By addressing the aperture problem through increased receptive fields, the authors provide a practical engineering roadmap for building safer vision systems. While challenges remain regarding rotational errors and natural corruptions like fog or snow, these findings represent a significant step toward reliable motion estimation in the real world.
Summary updated