Introduction
Multi-Object Tracking (MOT) is a critical component in autonomous systems, typically following a tracking-by-detection paradigm. In this framework, an object detector identifies targets in each frame, and an association module links these detections across time to form trajectories. While robust, these systems are vulnerable to adversarial attacks.
Existing attacks generally target either the detector or the association logic. Detector-based attacks attempt to make objects "disappear" (False Negatives), but trackers often employ a "reserved period" to maintain trajectories through brief detection gaps, requiring attackers to perturb many consecutive frames. Association-based attacks attempt to hijack the linking process by shifting bounding boxes, which is efficient but often requires white-box access to internal tracker logic and struggles in crowded environments. The F&F Attack bridges this gap by simultaneously inducing False Negatives (FN) and False Positives (FP) to efficiently disrupt tracking with minimal frame perturbations.
The F&F Attack Strategy
The F&F Attack perturbs input images to force the detector into a specific failure mode that confuses the association module. It employs a two-pronged approach:
- False Negative (FN) Component: The attack suppresses the correct detection of a target.
- False Positive (FP) Component: The attack injects multiple deceptive false alarms near the target's original location.
Target-Size-Aware Shift Strategy
A key innovation of the F&F Attack is the Target-Size-Aware Shift Strategy. Rather than placing random noise, the attack injects fake detections shifted by a factor of the target's width and height. This strategy serves two purposes:
- NMS Evasion: By shifting boxes sufficiently, the attack ensures that the detector's Non-Maximum Suppression (NMS) does not consolidate the fake boxes into the original target location.
- Implicit Association Disruption: By providing multiple "near-miss" boxes, the tracker is forced to associate with an incorrect detection. This induces an incorrect velocity calculation. When the attack ceases, the tracker's predicted location for the target is so far from the actual object that it fails to reconnect, resulting in a persistent Identity Switch (IDSW).
The attack is optimized using Projected Gradient Descent (PGD) to find the minimal pixel-level perturbation required to minimize the detection loss of models like YOLOX or CenterNet.
Experimental Evaluation
The researchers evaluated the F&F Attack on the MOT17 and MOT20 benchmarks, which represent diverse viewpoints and high-density crowd scenarios. The attack was tested against three state-of-the-art trackers: ByteTrack, SORT, and CenterTrack.
Key Results
The F&F Attack demonstrated significant superiority over existing baselines in terms of both efficiency and impact:
- Efficiency: Unlike previous detector-based attacks that required targeting 30+ frames, F&F achieved higher success rates by attacking only 1 to 3 frames.
- Success Rate: On MOT17, the attack achieved an immediate identity switch rate (IDSWim) of 85% for ByteTrack and 78.29% for SORT.
- Metric Degradation: The attack caused substantial drops in tracking quality metrics. On MOT20, ByteTrack's Association Accuracy (AssA) dropped by over 24 points, and significant decreases were noted in the Identity F1 Score (IDF1).
Discussion and Future Work
The F&F Attack highlights a fundamental vulnerability in motion-based association: by providing "wrong" data rather than "no" data, attackers can exploit the tracker's own robustness mechanisms.
However, the attack currently has limitations. It is a "digital" attack requiring direct pixel manipulation, and its effectiveness may be reduced against trackers that rely heavily on Appearance Embeddings (Re-ID), as visual features remain consistent even if boxes are shifted. Future research may explore physical-world implementations or defensive strategies, such as strengthening smoothness constraints in state estimation to mitigate the impact of adversarial jitter.
Summary updated