Abstract
Recent works have proposed several long term tracking benchmarks and highlight the importance of moving towards long-duration tracking to bridge the gap with application requirements. The current evaluation methodologies, however, do not focus on several aspects that are crucial in a long term perspective like Re-detection, Recovery, and Reliability. In this paper, we propose novel evaluation strategies for a more in-depth analysis of trackers from a long-term perspective. More specifically, (a) we test redetection capability of the trackers in the wild by simulating virtual cuts, (b) we investigate the role of chance in the recovery of tracker after failure and (c) we propose a novel metric allowing visual inference on the ability of a tracker to track contiguously (without any failure) at a given accuracy. We present several original insights derived from an extensive set of quantitative and qualitative experiments.
Introduction
In computer vision, tracking involves following a specific object across a video sequence. While deep learning has significantly advanced this field, most evaluations rely on short video clips averaging 20 seconds. Real-world applications—such as surveillance or long-duration drone monitoring—present much harsher conditions where objects frequently disappear, reappear, and interact with similar-looking "distractors."
Existing metrics like "Success" and "Precision" are often too coarse to capture these nuances. They fail to explain why a tracker failed or how it recovered. A tracker might appear to "recover" simply by chance—for instance, by remaining stationary until the target happens to re-enter its bounding box. This paper introduces the "3 R’s" framework—Re-detection, Recovery, and Reliability—to provide a more rigorous diagnostic evaluation of long-term tracking performance.
Methodology and Evaluation Strategies
To move beyond standard metrics, we propose three novel analytical methods:
1. Re-detection in the Wild (Virtual Cuts)
To evaluate a tracker's ability to re-acquire an object after total disappearance, we simulate "virtual cuts." This involves jumping from one frame to a distant future frame where the target is in a significantly different position. We utilize Generalized IoU (GIoU) minimization to ensure these jumps are as challenging as possible, forcing the tracker to perform a global search rather than relying on local spatial proximity.
2. Analyzing "Recovery by Chance"
We formally categorize recoveries that occur due to environmental factors rather than algorithmic robustness:
- Tracking Alternate Objects: The tracker follows a distractor (e.g., another person) that eventually crosses paths with the original target, leading to an accidental re-acquisition.
- Recovery with No Motion (Static Recovery): The tracker "freezes" on a background patch, and the target eventually moves back into that fixed bounding box.
3. 3D Longest Subsequence Measure (3D-LSM)
We extend the LSM metric into a 3D visualization that plots the longest contiguous segment tracked successfully. By varying the required accuracy (IoU threshold) and failure tolerance (slack), we create a reliability "heat map" that distinguishes between stable trackers and those that are jittery or prone to frequent failure.
Experimental Insights
Our study primarily utilizes the Track Long and Prosper (TLP) dataset, which contains 50 high-resolution sequences averaging 8 minutes in length. This duration is essential for observing multiple failure and recovery cycles.
- Offline vs. Online Failure Modes: Offline-trained trackers (e.g., SiamRPN, ATOM) excel at identifying "objectness" but are easily lured by distractors of the same class. Conversely, online-updated trackers (e.g., ECO, CMT) better distinguish specific targets but often "freeze" on the background due to model drift during updates.
- The Impact of Chance: A significant portion of long-term tracking success is accidental. For trackers like SiamRPN, the Success Metric dropped from 43.80 to 25.19 when recoveries caused by tracking distractors were discounted.
Conclusion
The "3R" framework reveals that many state-of-the-art trackers lack robust re-detection and often rely on lucky encounters to maintain high success scores. While this framework does not provide a single ranking number, it offers a multi-faceted diagnostic tool for developers. Future work should focus on trackers that can explicitly detect their own failure states and trigger global re-detection modules rather than relying on static waiting or local search windows.
Summary updated