TL;DR
This paper introduces a novel "depth-adaptive" convolutional neural network for visual object tracking that dynamically adjusts its computational depth based on the difficulty of each video frame. This allows the model to achieve state-of-the-art accuracy while significantly reducing computational cost on easier frames, enabling a flexible trade-off between speed and performance.
What Problem Does This Paper Address?
Current visual tracking algorithms, despite their impressive accuracy, often employ a fixed computational budget for every frame in a video. This is inefficient because the difficulty of tracking an object can vary greatly from frame to frame due to factors like background clutter, object motion, and scene complexity. Applying the same amount of computation to an easy frame as to a difficult one leads to wasted resources. The paper addresses this by proposing a system that intelligently allocates computational effort, spending more on challenging frames and less on simpler ones, thereby optimizing efficiency without sacrificing accuracy.
Methods
The core innovation is a depth-adaptive convolutional Siamese network. Instead of processing a frame through the entire network, this model uses "gating functions" to decide how many layers (or "blocks") of the convolutional feature extractor to activate. These gates are trained to predict the difficulty of tracking based on intermediate feature maps.
Key methodological aspects include:
- Siamese Network Architecture: The model builds upon existing fully-convolutional Siamese networks, which are effective for tracking by comparing a "key frame" (containing the target object) with a "search frame."
- Depth-Adaptive Computation: The convolutional layers are divided into blocks, and parametric gating functions are introduced between these blocks. These gates output a confidence score for halting computation at that point.
- Budgeted Gating: To prevent "polarized" policies (always shallow or always deep), a "budgeted" confidence score is used, where the scores across all gates sum to 1. This encourages a more balanced decision-making process.
- End-to-End Differentiable Training: The gating functions are designed to be differentiable, allowing for end-to-end training using back-propagation. This avoids the high variance issues associated with reinforcement learning methods for training discrete gates.
- Two-Phase Training: The VGG weights are first fine-tuned using a sum of tracking losses across all intermediate depths (intermediate supervision). Then, the gate parameters are trained while keeping the VGG weights fixed, minimizing a joint loss of tracking accuracy and computational cost.
Datasets
- VOT2016: Used for both training and evaluation. The authors note that this dataset is smaller and more challenging than typical large-scale datasets like ImageNet VID. They randomly split the videos for testing.
Key Insights and Ideas
- Frame Difficulty is Key: The paper highlights that not all frames are created equal in terms of tracking difficulty, and exploiting this variability is crucial for efficiency.
- Intermediate Supervision Matters: Training with losses at intermediate layers, not just the final output, significantly improves the performance of adaptive models. This provides more informative gradients for the gating mechanisms.
- Budgeted Gating for Balanced Policies: Directly using confidence scores from gates can lead to extreme decisions. The "budgeted gating" approach ensures a more nuanced and adaptive policy.
- Trading Speed for Accuracy: The framework explicitly allows for a tunable trade-off between computational cost and tracking accuracy by adjusting a hyper-parameter ($\lambda$).
Key Results
- State-of-the-Art Accuracy: The proposed depth-adaptive model achieves accuracy comparable to state-of-the-art methods on the VOT2016 benchmark.
- Significant Computational Savings: The adaptive policies (hard-gating) achieve comparable or better accuracy than fixed-depth baselines while using significantly less computational cost. For example, Figure 7 shows hard-gating achieving high accuracy at a fraction of the theoretical cost of deeper fixed-depth models.
- Real-time Performance: The system can achieve real-time speeds, with shallowest layer computations running at over 54 FPS and deepest at over 37 FPS on their hardware. The authors suggest further optimization could push FPS much higher.
Limitations and Future Work
- VOT2016 Dataset Limitations: The model is finetuned solely on VOT2016, which is smaller and potentially less diverse than other tracking datasets.
- No Online Model Updating: Fully-convolutional Siamese networks, as used here, do not inherently support updating the tracking model based on the key frame's object appearance during tracking.
- Coarse Cross-Correlation Resolution: The 16x16 resolution of the final cross-correlation map is considered coarse for fine-grained tracking. Experiments with higher resolutions showed promise but were incompatible with hard-gating due to the need for cross-correlation at multiple depths.
- Future Work Suggestions: The authors suggest exploring multi-scale tracking, more complex gating features (e.g., using history), adaptive spatial computation (e.g., focusing attention on moving parts), and frame skipping. They also note the potential to extend this conditional computation approach to other tasks beyond tracking.
Summary updated