Uncertainty-Guided Inference-Time Depth Adaptation for Transformer-Based Visual Tracking

Patrick Poggi, Divake Kumar, Theja Tulabandhula, Amit Ranjan TrivediarXiv 2026 · DOI

TL;DR

This paper introduces UncL-STARK, a novel approach to make transformer-based visual trackers more efficient by dynamically adjusting their computational depth during inference. By leveraging a lightweight uncertainty estimation derived from the tracker's own output, UncL-STARK intelligently skips computations for easy frames, significantly reducing computational cost without sacrificing accuracy.

What Problem Does This Paper Address?

Transformer-based visual trackers have achieved state-of-the-art performance by employing deep encoder-decoder architectures. However, these models are computationally expensive, as they process every frame with the full network depth, regardless of the frame's complexity or the tracker's confidence. This is particularly wasteful in long video sequences where many frames are temporally coherent and easy to track. Existing methods for making trackers more efficient often involve architectural modifications or auxiliary components, which can increase complexity and hinder deployment. The core problem is how to achieve significant computational savings in transformer trackers without compromising accuracy or introducing substantial overhead.

Methods

UncL-STARK tackles this problem through an "architecture-preserving" approach, meaning it doesn't alter the fundamental structure of existing transformer trackers like STARK. The key innovations are:

  • Random-Depth Training with Knowledge Distillation: The tracker is fine-tuned using a novel training strategy. During training, the model is exposed to various "depths" (i.e., using only a subset of encoder/decoder layers). Knowledge distillation is used to transfer the performance of the full-depth model to these shallower configurations, ensuring they remain robust.
  • Heatmap-Derived Uncertainty Estimation: Instead of complex uncertainty estimation methods, UncL-STARK leverages the corner localization heatmaps already produced by the tracker. The spatial concentration and mass of these heatmaps serve as a proxy for prediction confidence. A simple scalar score is computed from the top-k heatmap mass.
  • Feedback-Driven Depth Selection Policy: At runtime, the computed uncertainty score from frame t is used to decide the computational depth for frame t+1. A simple threshold-based policy assigns shallower depths to high-confidence frames and deeper depths to low-confidence (uncertain) frames, effectively exploiting temporal coherence in video.

Datasets

The paper uses the following datasets for training and evaluation:

  • GOT-10k: Used for training and validation.
  • LaSOT: Used for training and testing.

No new benchmarks were introduced.

Key Insights and Ideas

  • Leveraging Existing Outputs for Efficiency: The most striking insight is that valuable information for adaptive computation (uncertainty) is already present in the standard outputs (corner heatmaps) of many modern trackers. This avoids the need for extra prediction heads or complex modules.
  • Counter-intuitive Benefit of Shallower Depths Under Occlusion: Surprisingly, the adaptive tracker performed better under occlusion when using shallower depths. The paper hypothesizes that coarser predictions from shallower depths are less prone to drifting, aiding in recovery when the object reappears.
  • The Power of "Good Enough" Predictions: The research demonstrates that for many frames, a "good enough" prediction from a shallower network is sufficient, and the computational savings are substantial. The system intelligently reserves full computation for genuinely challenging frames.

Key Results

UncL-STARK achieves significant efficiency gains while maintaining high accuracy:

  • Computational Savings: Up to 12% GFLOPs reduction.
  • Latency Improvement: Up to 8.9% latency reduction.
  • Energy Savings: Up to 10.8% energy savings.
  • Accuracy Preservation: Tracking accuracy remains within 0.2% of the full-depth baseline across datasets.
  • Robustness: The adaptive policy generalizes well across sequences of varying lengths and difficulty, including challenging occluded scenarios.

Limitations and Future Work

  • Limitations:
    • The paper acknowledges that the shallowest configuration (1,1) still incurs a noticeable accuracy loss, indicating a trade-off exists at extreme depths.
  • The effectiveness relies on the quality of the corner heatmaps produced by the base tracker.
  • Future Work:
    • The authors suggest that this uncertainty-guided adaptation mechanism could be extended to other transformer-based computer vision tasks beyond tracking.
  • Further exploration of more sophisticated feedback policies or uncertainty estimation methods could potentially yield even greater efficiency gains.

Summary updated

BibTeX
@misc{poggi2026uncertainty,
  author = {Patrick Poggi and Divake Kumar and Theja Tulabandhula and Amit Ranjan Trivedi},
  title = {{Uncertainty-Guided Inference-Time Depth Adaptation for Transformer-Based Visual Tracking}},
  year = {2026},
  eprint = {2602.16160},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2602.16160},
}

Loading related content...

Figure