F&F Attack: Adversarial Attack against Multiple Object Trackers by Inducing False Negatives and False Positives

Tao Zhou, Qi Ye, Wenhan Luo, Kaihao Zhang, Zhiguo Shi, Jiming Chen

Introduction

Multi-Object Tracking (MOT) is a critical component in autonomous systems, typically following a tracking-by-detection paradigm. In this framework, an object detector identifies targets in each frame, and an association module links these detections across time to form trajectories. While robust, these systems are vulnerable to adversarial attacks.

Existing attacks generally target either the detector or the association logic. Detector-based attacks attempt to make objects "disappear" (False Negatives), but trackers often employ a "reserved period" to maintain trajectories through brief detection gaps, requiring attackers to perturb many consecutive frames. Association-based attacks attempt to hijack the linking process by shifting bounding boxes, which is efficient but often requires white-box access to internal tracker logic and struggles in crowded environments. The F&F Attack bridges this gap by simultaneously inducing False Negatives (FN) and False Positives (FP) to efficiently disrupt tracking with minimal frame perturbations.

The F&F Attack Strategy

The F&F Attack perturbs input images to force the detector into a specific failure mode that confuses the association module. It employs a two-pronged approach:

  1. False Negative (FN) Component: The attack suppresses the correct detection of a target.
  2. False Positive (FP) Component: The attack injects multiple deceptive false alarms near the target's original location.

Target-Size-Aware Shift Strategy

A key innovation of the F&F Attack is the Target-Size-Aware Shift Strategy. Rather than placing random noise, the attack injects fake detections shifted by a factor of the target's width and height. This strategy serves two purposes:

  • NMS Evasion: By shifting boxes sufficiently, the attack ensures that the detector's Non-Maximum Suppression (NMS) does not consolidate the fake boxes into the original target location.
  • Implicit Association Disruption: By providing multiple "near-miss" boxes, the tracker is forced to associate with an incorrect detection. This induces an incorrect velocity calculation. When the attack ceases, the tracker's predicted location for the target is so far from the actual object that it fails to reconnect, resulting in a persistent Identity Switch (IDSW).

The attack is optimized using Projected Gradient Descent (PGD) to find the minimal pixel-level perturbation required to minimize the detection loss of models like YOLOX or CenterNet.

Experimental Evaluation

The researchers evaluated the F&F Attack on the MOT17 and MOT20 benchmarks, which represent diverse viewpoints and high-density crowd scenarios. The attack was tested against three state-of-the-art trackers: ByteTrack, SORT, and CenterTrack.

Visualization of the F&F attack on ByteTrack, SORT, and CenterTrack. By perturbing only 1-2 frames (red text), the attack induces persistent identity switches in subsequent clean frames.

Key Results

The F&F Attack demonstrated significant superiority over existing baselines in terms of both efficiency and impact:

  • Efficiency: Unlike previous detector-based attacks that required targeting 30+ frames, F&F achieved higher success rates by attacking only 1 to 3 frames.
  • Success Rate: On MOT17, the attack achieved an immediate identity switch rate (IDSWim) of 85% for ByteTrack and 78.29% for SORT.
  • Metric Degradation: The attack caused substantial drops in tracking quality metrics. On MOT20, ByteTrack's Association Accuracy (AssA) dropped by over 24 points, and significant decreases were noted in the Identity F1 Score (IDF1).

Discussion and Future Work

The F&F Attack highlights a fundamental vulnerability in motion-based association: by providing "wrong" data rather than "no" data, attackers can exploit the tracker's own robustness mechanisms.

However, the attack currently has limitations. It is a "digital" attack requiring direct pixel manipulation, and its effectiveness may be reduced against trackers that rely heavily on Appearance Embeddings (Re-ID), as visual features remain consistent even if boxes are shifted. Future research may explore physical-world implementations or defensive strategies, such as strengthening smoothness constraints in state estimation to mitigate the impact of adversarial jitter.

Summary updated

Abstract

Multi-object tracking (MOT) aims to build moving trajectories for number-agnostic objects. Modern multi-object trackers commonly follow the tracking-by-detection strategy. Therefore, fooling detectors can be an effective solution but it usually requires attacks in multiple successive frames, resulting in low efficiency. Attacking association processes improves efficiency but may require model-specific design, leading to poor generalization. In this paper, we propose a novel False negative and False positive attack (F&F attack) mechanism: it perturbs the input image to erase original detections and to inject deceptive false alarms around original ones while integrating the association attack implicitly. The mechanism can produce effective identity switches against multi-object trackers by only fooling detectors in a few frames. To demonstrate the flexibility of the mechanism, we deploy it to three multi-object trackers (ByteTrack, SORT, and CenterTrack) which are enabled by two representative detectors (YOLOX and CenterNet). Comprehensive experiments on MOT17 and MOT20 datasets show that our method significantly outperforms existing attackers, revealing the vulnerability of the tracking-by-detection paradigm to detection attacks.
BibTeX
@inproceedings{zhou2023f,
  author = {Tao Zhou and Qi Ye and Wenhan Luo and Kaihao Zhang and Zhiguo Shi and Jiming Chen},
  title = {{F\&F Attack: Adversarial Attack against Multiple Object Trackers by Inducing False Negatives and False Positives}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2023},
  doi = {10.1109/iccv51070.2023.00422},
}
Figure