Depth-Adaptive Computational Policies for Efficient Visual Tracking

Chris Ying, Katerina FragkiadakiEMMCVPR 2017 · OpenReview

TL;DR

This paper introduces a novel "depth-adaptive" convolutional neural network for visual object tracking that dynamically adjusts its computational depth based on the difficulty of each video frame. This allows the model to achieve state-of-the-art accuracy while significantly reducing computational cost on easier frames, enabling a flexible trade-off between speed and performance.

What Problem Does This Paper Address?

Current visual tracking algorithms, despite their impressive accuracy, often employ a fixed computational budget for every frame in a video. This is inefficient because the difficulty of tracking an object can vary greatly from frame to frame due to factors like background clutter, object motion, and scene complexity. Applying the same amount of computation to an easy frame as to a difficult one leads to wasted resources. The paper addresses this by proposing a system that intelligently allocates computational effort, spending more on challenging frames and less on simpler ones, thereby optimizing efficiency without sacrificing accuracy.

Methods

The core innovation is a depth-adaptive convolutional Siamese network. Instead of processing a frame through the entire network, this model uses "gating functions" to decide how many layers (or "blocks") of the convolutional feature extractor to activate. These gates are trained to predict the difficulty of tracking based on intermediate feature maps.

Key methodological aspects include:

  • Siamese Network Architecture: The model builds upon existing fully-convolutional Siamese networks, which are effective for tracking by comparing a "key frame" (containing the target object) with a "search frame."
  • Depth-Adaptive Computation: The convolutional layers are divided into blocks, and parametric gating functions are introduced between these blocks. These gates output a confidence score for halting computation at that point.
  • Budgeted Gating: To prevent "polarized" policies (always shallow or always deep), a "budgeted" confidence score is used, where the scores across all gates sum to 1. This encourages a more balanced decision-making process.
  • End-to-End Differentiable Training: The gating functions are designed to be differentiable, allowing for end-to-end training using back-propagation. This avoids the high variance issues associated with reinforcement learning methods for training discrete gates.
  • Two-Phase Training: The VGG weights are first fine-tuned using a sum of tracking losses across all intermediate depths (intermediate supervision). Then, the gate parameters are trained while keeping the VGG weights fixed, minimizing a joint loss of tracking accuracy and computational cost.

Datasets

  • VOT2016: Used for both training and evaluation. The authors note that this dataset is smaller and more challenging than typical large-scale datasets like ImageNet VID. They randomly split the videos for testing.

Key Insights and Ideas

  • Frame Difficulty is Key: The paper highlights that not all frames are created equal in terms of tracking difficulty, and exploiting this variability is crucial for efficiency.
  • Intermediate Supervision Matters: Training with losses at intermediate layers, not just the final output, significantly improves the performance of adaptive models. This provides more informative gradients for the gating mechanisms.
  • Budgeted Gating for Balanced Policies: Directly using confidence scores from gates can lead to extreme decisions. The "budgeted gating" approach ensures a more nuanced and adaptive policy.
  • Trading Speed for Accuracy: The framework explicitly allows for a tunable trade-off between computational cost and tracking accuracy by adjusting a hyper-parameter ($\lambda$).

Key Results

  • State-of-the-Art Accuracy: The proposed depth-adaptive model achieves accuracy comparable to state-of-the-art methods on the VOT2016 benchmark.
  • Significant Computational Savings: The adaptive policies (hard-gating) achieve comparable or better accuracy than fixed-depth baselines while using significantly less computational cost. For example, Figure 7 shows hard-gating achieving high accuracy at a fraction of the theoretical cost of deeper fixed-depth models.
  • Real-time Performance: The system can achieve real-time speeds, with shallowest layer computations running at over 54 FPS and deepest at over 37 FPS on their hardware. The authors suggest further optimization could push FPS much higher.

Limitations and Future Work

  • VOT2016 Dataset Limitations: The model is finetuned solely on VOT2016, which is smaller and potentially less diverse than other tracking datasets.
  • No Online Model Updating: Fully-convolutional Siamese networks, as used here, do not inherently support updating the tracking model based on the key frame's object appearance during tracking.
  • Coarse Cross-Correlation Resolution: The 16x16 resolution of the final cross-correlation map is considered coarse for fine-grained tracking. Experiments with higher resolutions showed promise but were incompatible with hard-gating due to the need for cross-correlation at multiple depths.
  • Future Work Suggestions: The authors suggest exploring multi-scale tracking, more complex gating features (e.g., using history), adaptive spatial computation (e.g., focusing attention on moving parts), and frame skipping. They also note the potential to extend this conditional computation approach to other tasks beyond tracking.

Summary updated

Abstract

Current convolutional neural networks algorithms for video object tracking spend the same amount of computation for each object and video frame. However, it is harder to track an object in some frames than others, due to the varying amount of clutter, scene complexity, amount of motion, and object's distinctiveness against its background. We propose a depth-adaptive convolutional Siamese network that performs video tracking adaptively at multiple neural network depths. Parametric gating functions are trained to control the depth of the convolutional feature extractor by minimizing a joint loss of computational cost and tracking error. Our network achieves accuracy comparable to the state-of-the-art on the VOT2016 benchmark. Furthermore, our adaptive depth computation achieves higher accuracy for a given computational cost than traditional fixed-structure neural networks. The presented framework extends to other tasks that use convolutional neural networks and enables trading speed for accuracy at runtime.
BibTeX
@inproceedings{ying2017depth,
  author = {Chris Ying and Katerina Fragkiadaki},
  title = {{Depth-Adaptive Computational Policies for Efficient Visual Tracking}},
  booktitle = {EMMCVPR},
  year = {2017},
  url = {https://openreview.net/forum?id=N0QAm8aHIVzE},
}
Figure