TL;DR
This paper introduces THOR, a novel framework that enhances existing template-matching trackers by building a "holistic" representation of objects. By dynamically accumulating diverse templates over time, THOR significantly improves tracking robustness and accuracy without requiring retraining of the base tracker, even outperforming state-of-the-art methods while maintaining high speeds.
What Problem Does This Paper Address?
Visual tracking, the task of following an object across a video sequence, is a fundamental problem in computer vision. Many modern trackers rely on template matching, where a representation of the target object (the template) is used to find its location in subsequent frames. A key challenge for these trackers is handling variations in object appearance due to changes in illumination, pose, scale, occlusions, and even object deformation. Traditional template-matching methods often struggle with these variations, leading to tracking drift and failure. While some methods update templates, they can be slow or require extensive retraining. This paper addresses the need for a robust and efficient way to represent an object's appearance over time, capturing its diversity without compromising speed or requiring re-training of the underlying tracking mechanism.
Methods
THOR is a framework designed to be plugged into existing template-matching trackers. It consists of two main modules:
- Long-Term Module (LTM): This module aims to store a diverse set of templates that capture the object's appearance over extended periods and various conditions. To achieve diversity, it uses a novel measure based on the Gram determinant of feature vectors. A new template is added only if it significantly increases the diversity of the existing set, effectively maximizing the volume of the parallelotope spanned by the feature vectors. To prevent drift, a lower bound on the similarity between candidate templates and the base template is enforced, with options for dynamic or ensemble bounds. Template masking is also used to reduce the influence of background.
- Short-Term Module (STM): This module handles abrupt changes in appearance, such as sudden occlusions or rapid movements, which might be too dissimilar for the LTM. Templates are stored in a first-in, first-out manner up to a fixed capacity. The STM also contributes to a diversity measure that can dynamically adjust the LTM's lower bound.
During inference, THOR uses a Modulation step to combine information from both LTM and STM by computing a weighted spatial average of activation maps. An ST-LT Switch then decides which module's prediction to use, prioritizing the STM for short-term robustness but falling back to the LTM if significant drift is detected (measured by IoU with the LTM's prediction).
Datasets
The paper evaluates THOR on two standard visual tracking benchmarks:
- VOT2018: Used for evaluating tracking performance in terms of accuracy, robustness, and Expected Average Overlap (EAO).
- OTB2015: Used for evaluating performance with Area Under Curve (AUC) and precision metrics.
Key Insights and Ideas
- Holistic Representation is Key: The core idea is that a single template is insufficient for robust tracking. By accumulating a diverse set of templates, THOR builds a richer, "holistic" representation of the object that can adapt to changing conditions.
- Gram Determinant for Diversity: Using the Gram determinant of feature vectors is an elegant way to quantify the diversity of templates in a high-dimensional feature space, directly linking to the geometric concept of parallelotope volume.
- Decoupled Enhancement: THOR's strength lies in its ability to enhance existing trackers without requiring them to be retrained. This makes it a versatile and practical addition to many tracking systems.
- Balancing Short-Term and Long-Term: The separation into STM and LTM modules effectively addresses both immediate appearance changes and long-term variations, a crucial aspect for robust tracking.
- Analytical Approach to Learning: Unlike some methods that embed template management into a learning process, THOR uses an analytical approach, leading to greater interpretability and easier integration.
Key Results
- Performance Improvement: THOR significantly improves the performance of state-of-the-art Siamese trackers (SiamFC, SiamRPN, SiamMask) on both VOT2018 and OTB2015 benchmarks. For example, THOR-SiamRPN achieves an EAO of 0.416 on VOT2018, outperforming the original SiamRPN++ (0.414 EAO) while running at 112 FPS compared to 35 FPS.
- Robustness: THOR substantially mitigates tracking drift, as evidenced by improved robustness metrics on VOT2018 and a lower number of drifted templates in ablation studies.
- Speed: Despite adding computations, THOR maintains high speeds. For instance, THOR-SiamRPN runs at 244 FPS, which is respectable and comparable to other multi-template approaches.
- Ablation Study: The ablation study confirms the importance of each component: turning off modulation or the STM has the biggest negative impact on performance.
Limitations and Future Work
- Short Sequences: THOR's improvement is less pronounced on very short sequences, as there isn't enough time to accumulate a diverse set of templates.
- Hyperparameter Sensitivity: Like many Siamese trackers, THOR is sensitive to hyperparameter choices, which could be an area for further research.
- Long-Term Tracking: While tested on short-term benchmarks, THOR's potential for long-term tracking (e.g., on VOT-LT) is highlighted as a promising future direction, with a need for careful lower bound selection to manage drift.
- Joint Optimization: A future direction is to train the base tracker to also optimize the THOR objective, potentially leading to even greater synergy.
- Beyond Tracking: The authors suggest exploring the holistic object representation and diversity measure for applications beyond visual tracking, such as active vision and robotic manipulation tasks.
Summary updated