Physically Real-time Infrared Attack against Optical Flow Estimation Networks

Shen You, Wei Jiang, Jiarui Liu, Yijian Ye, Qiuzhen Lin, Xiangtao Li, Ka-Chun WongarXiv 2026 · arXiv

TL;DR

This paper introduces a real-time, physical-world adversarial attack against Optical Flow Estimation Networks (OFENs) using invisible infrared lights. By combining a genetic algorithm for offline data generation with a lightweight generative network for real-time deployment, the attack successfully blinds motion-tracking systems without alerting human bystanders.

What Problem Does This Paper Address?

Optical Flow Estimation Networks (OFENs) are critical upstream components in computer vision pipelines, widely used for motion detection, object tracking, and obstacle avoidance in autonomous driving. Because downstream decision-making models rely heavily on their outputs, the robustness of OFENs is paramount to safety.

While digital adversarial attacks are well-studied, translating them into the physical world presents major hurdles:

  1. Domain Shift & Noise: Printed adversarial patches often fail because camera sensors introduce unexpected noise and lighting variations.
  2. Dynamic Environments: Existing physical attacks struggle to adapt when a target object changes its position, angle, or speed.
  3. Stealth: Most physical attacks rely on visible patches or projected light, which are easily noticed by humans, defeating the purpose of a stealthy adversarial attack.

The authors address these gaps by proposing a physically real-time, dynamic, and completely invisible attack using infrared (IR) light. Since many surveillance and automotive cameras lack infrared filters to perform better in low-light conditions, they are highly sensitive to IR spectrum modulations that humans cannot see.

Figure 1. The application infrared attack.

Methods

The authors develop RILA (Real-time Infrared Light Attack), a two-stage framework designed to bypass the computational latency of traditional optimization techniques:

  1. Stage 1: Non-Real-Time Training (Genetic Algorithm): Because the physical world transformation function is non-derivative, gradient-based optimization cannot be used directly. Instead, the authors use a Genetic Algorithm (GA) to find the optimal intensities for an array of infrared lights attached to a target. The GA iteratively evolves the perturbation patterns to maximize the disruption of the victim OFEN. This stage is used to collect a robust training dataset of physical adversarial examples.

  2. Stage 2: Real-Time Attacking (Adversarial Generative Network): To match the real-time constraints of video tracking (typically $<30\text{ms}$ per frame), the authors train a lightweight Adversarial Generative Network (AGN) consisting of a simple fully connected layer. The AGN takes the previous frame's light intensity state and instantly predicts the next optimal adversarial perturbation.

Figure 2. (a) Timeline and (b) Sequence Chart of the OFEN attacking process with AEs in the physical world.

Bespoke Loss Functions

To guide the optimization, the authors introduce three specialized sub-fitness terms:

  • Invisibility Loss ($\mathcal{L}_{Vis}$): Forces the network to output a blank optical flow field, effectively making the moving object disappear from the tracker's perspective.
  • Imperceptibility Loss ($\mathcal{L}_{Per}$): Maximizes the chaos and variance within the optical flow, making it impossible for downstream models to discern true motion.
  • Flash Loss ($\mathcal{L}_{Flash}$): Maximizes the brightness difference between consecutive frames, directly sabotaging the core "brightness constancy" assumption that optical flow algorithms depend on.

Datasets

Rather than relying on standard digital benchmarks, the authors evaluate their method using a custom-built Physical Test Environment. The setup consists of a motorized track to control target movement, adjustable ambient LED lighting, a camera capturing at 30 frames per second, and a target plate embedded with nine $840\text{nm}$ infrared lights controlled by an ESP32 microcontroller.

The training data for the real-time network (AGN) is generated directly within this physical environment during the Genetic Algorithm phase.

Key Insights and Ideas

  • Exploiting Core Algorithmic Assumptions: Optical flow algorithms rely on brightness constancy (the assumption that an object's pixels don't drastically change brightness between consecutive frames). By rapidly flashing IR lights at calculated intensities, RILA systematically shatters this assumption.
  • Bridging the Speed Gap: Genetic algorithms are powerful for black-box physical optimization but far too slow for real-time deployment. Using the GA strictly as a data collector to train a ultra-lightweight neural network (AGN) elegantly solves the real-time constraint.
  • Ablation Revelations: Disabling individual loss functions highlights their unique roles. Without the Flash Loss, the general outline of the object remains visible because the IR lights fail to sufficiently alter surface lighting across frames, proving it to be the most critical component of the attack.
Visualizing the impact of removing individual loss components during an attack on RAFT.

Key Results

The attack was evaluated against two state-of-the-art optical flow models: RAFT and PWC-Net, using two primary metrics: Average Invisibility (AIV) and Average Imperceptibility (AIP) (where values closer to 1 indicate a more successful attack).

  • High Effectiveness: The full RILA method achieved an AIV of 0.900 and an AIP of 0.979 against RAFT, and an AIV of 0.907 and an AIP of 0.980 against PWC-Net.
  • Robustness to Speed: The attack is highly effective at lower speeds (e.g., $6\text{ cm/s}$ yielded an AIP of 0.989), with performance degrading only slightly as the object's speed increased up to $33\text{ cm/s}$.
  • Distance Invariance: Testing across eleven different distances (from $30\text{ cm}$ to $120\text{ cm}$) showed that camera distance had negligible impact on attack success, as the IR lights consistently covered the target's surface.
  • Ambient Light Resilience: The attack performs optimally under standard ambient lighting intensities ($0.3$ to $0.7$). Performance drops only under extreme brightness ($>0.7$), where natural light overpowers the infrared LEDs.

Limitations and Future Work

While highly effective, the authors acknowledge a few limitations:

  • Physical Constraints: Due to laboratory space limitations, the physical test environment was restricted to relatively close camera distances ($30\text{–}120\text{ cm}$) and linear track movements.
  • Single-Modality Focus: The current attack specifically targets optical flow. Modern autonomous systems often employ multi-modal safety nets, combining optical flow with object detection models (like YOLO) or LiDAR.

Future work will focus on expanding this infrared attack paradigm to simultaneously disrupt object detection models, creating a more comprehensive multi-modal physical threat.

Summary updated

Abstract

With the promising performance of deep neural networks on image-based tasks, different real-world applications such as autonomous driving and motion detection have become increasingly mature and relevant to human lives. In particular, Optical Flow Estimation Networks (OFENs), as upstream models, play a critical role in different domains. Its outputs are heavily assumed and adopted for different downstream tasks, and it is essential to test its robustness to prevent safety accidents. We present an approach for real-time attacks on OFENs in the physical world, leveraging infrared lights for their stealthiness. By generating a large number of Adversarial Examples in advance, our approach computes AEs in real time and dynamically displays them, which allows our method to facilitate precise and targeted attacks without modifying the victim system. Unlike previous digital-to-physical attack techniques, our method directly attacks victim models within the physical world, thereby overcoming the limitations associated with the ineffectiveness of AEs. Experimental results demonstrate the efficacy of our approach in compromising OFENs across diverse lighting conditions, varying object motion velocities, and different object placements, ultimately impairing the network's ability to accurately estimate optical flow.
BibTeX
@misc{you2026physically,
  author = {Shen You and Wei Jiang and Jiarui Liu and Yijian Ye and Qiuzhen Lin and Xiangtao Li and Ka-Chun Wong},
  title = {{Physically Real-time Infrared Attack against Optical Flow Estimation Networks}},
  year = {2026},
  eprint = {2607.26651},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2607.26651},
}
Figure