Spatial-aware Online Adversarial Perturbations Against Visual Object Tracking.

Qing Guo, Xiaofei Xie, Lei Ma, Zhongguo Li, Wei Feng, Yang Liu

Abstract

Visual Object Tracking (VOT) is a fundamental component in real-time autonomous systems, such as drones and self-driving vehicles. Unlike static image classification, VOT requires continuous, frame-by-frame predictions, making it susceptible to unique adversarial challenges. This paper presents SPARK (Spatial-aware online incremental attack), a novel framework designed to hijack the trajectory of real-time visual trackers. By leveraging incremental optimization and spatial-temporal sparsity, SPARK generates imperceptible perturbations that can force a tracker to follow a maliciously pre-defined path. Our extensive evaluation across four major benchmarks—OTB100, VOT2018, UAV123, and LaSOT—demonstrates that SPARK is both highly effective and computationally efficient, achieving real-time performance.

Introduction

Adversarial attacks have traditionally focused on static image classification, where subtle perturbations can mislead a model into misidentifying an object. However, real-world applications increasingly rely on Visual Object Tracking (VOT), which operates in an online, real-time manner. Attacking VOT systems is significantly more complex due to two primary hurdles:

  1. Temporal Transferability: Perturbations must remain effective as the object and background evolve across frames.
  2. Efficiency: Trackers often operate at 30+ FPS, requiring attacks to be generated with minimal latency.

To address these challenges, we introduce SPARK, an online incremental attack that exploits the temporal continuity of video to efficiently hijack tracker trajectories.

Methodology: The SPARK Framework

SPARK is built upon three core technical pillars designed for speed and stealth:

1. Incremental Optimization

Traditional adversarial attacks often restart the optimization process for every new frame, which is computationally prohibitive for real-time tracking. SPARK utilizes a "warm start" approach by calculating only the incremental difference needed from the previous frame's perturbation. Since consecutive video frames are highly similar, the perturbation required for frame $t$ is likely very close to that of frame $t-1$. This method allows the attack to converge in as few as two iterations, whereas standard iterative attacks might require ten or more, drastically reducing computational overhead.

2. Spatial-Temporal Sparsity

To ensure the attack remains imperceptible to human observers, SPARK employs an $L_{2,1}$ norm constraint. This mathematical regularizer encourages the noise to be spatially sparse—concentrated in small, critical regions—and temporally consistent. This minimizes visual flickering and makes the adversarial changes nearly invisible to the human eye.

3. Trajectory Hijacking

SPARK supports both Untargeted Attacks (UA), which aim to cause the tracker to lose the target, and Targeted Attacks (TA). In a TA scenario, the attacker can specify an arbitrary "Targeted Trajectory" (e.g., a spiral or a straight line away from the target), and SPARK will generate the necessary perturbations to "drive" the tracker along that path.

The SPARK framework: Incremental perturbations are added to clean frames to hijack the tracker's trajectory over time.

Experimental Evaluation

We evaluated SPARK against state-of-the-art Siamese-network-based trackers (e.g., SiamRPN, SiamRPN++, SiamDW) using four benchmark datasets: OTB100, VOT2018, UAV123, and LaSOT.

Key Results

  • High Success Rate: On the OTB100 benchmark, SPARK achieved a 78.9% success rate in targeted attacks against the SiamRPN-AlexNet tracker. This demonstrates that even sophisticated trackers can be reliably misled by carefully crafted incremental noise.
  • Real-Time Efficiency: The incremental optimization allowed SPARK to reach its goal in an average of 2.25 iterations per frame. This efficiency is crucial for real-world applications where the attack must keep pace with the high-speed processing of modern trackers.
Visual output of the SPARK attack on a vehicle tracking sequence.
  • Stealthiness: SPARK produced significantly lower Mean Absolute Perturbation compared to standard iterative attacks like BIM or C&W, maintaining high visual quality and making detection difficult for human observers.
  • Cross-Model Transferability: Perturbations generated for shallower models (e.g., AlexNet) were often effective against deeper architectures (e.g., ResNet-50), indicating a shared vulnerability in Siamese tracking paradigms.

Discussion and Limitations

While SPARK is highly effective, its performance varies with model depth. Deeper architectures like ResNet-50 exhibit greater robustness, leading to a decrease in attack success rates. Furthermore, while untargeted attacks transfer well across different models, targeted attacks—which require precise trajectory control—show lower transferability (approximately 6.5%). This suggests that the specific noise patterns required for path-following are highly optimized for the target model's internal representations.

Conclusion

SPARK demonstrates that the temporal continuity of video, while a strength for tracking, is a significant vulnerability that can be exploited for efficient adversarial attacks. By hijacking trajectories in real-time with imperceptible noise, SPARK highlights the need for more robust defensive mechanisms in safety-critical visual tracking applications.

Summary updated

Abstract

Adversarial attacks of deep neural networks have been intensively studied on image, audio, natural language, patch, and pixel classification tasks. Nevertheless, as a typical, while important real-world application, the adversarial attacks of online video object tracking that traces an object's moving trajectory instead of its category are rarely explored. In this paper, we identify a new task for the adversarial attack to visual tracking: online generating imperceptible perturbations that mislead trackers along an incorrect (Untargeted Attack, UA) or specified trajectory (Targeted Attack, TA). To this end, we first propose a \textit{spatial-aware} basic attack by adapting existing attack methods, i.e., FGSM, BIM, and C&W, and comprehensively analyze the attacking performance. We identify that online object tracking poses two new challenges: 1) it is difficult to generate imperceptible perturbations that can transfer across frames, and 2) real-time trackers require the attack to satisfy a certain level of efficiency. To address these challenges, we further propose the spatial-aware online incremental attack (a.k.a. SPARK) that performs spatial-temporal sparse incremental perturbations online and makes the adversarial attack less perceptible. In addition, as an optimization-based method, SPARK quickly converges to very small losses within several iterations by considering historical incremental perturbations, making it much more efficient than basic attacks. The in-depth evaluation on state-of-the-art trackers (i.e., SiamRPN++ with AlexNet, MobileNetv2, and ResNet-50, and SiamDW) on OTB100, VOT2018, UAV123, and LaSOT demonstrates the effectiveness and transferability of SPARK in misleading the trackers under both UA and TA with minor perturbations.
BibTeX
@misc{guo2019spatial,
  author = {Qing Guo and Xiaofei Xie and Lei Ma and Zhongguo Li and Wei Feng and Yang Liu},
  title = {{Spatial-aware Online Adversarial Perturbations Against Visual Object Tracking.}},
  year = {2019},
  eprint = {1910.08681},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1910.08681},
}

Loading related content...

Figure