Learn to Match: Automatic Matching Network Design for Visual Tracking

Zhipeng Zhang, Yihao Liu, Xiao Wang, Bing Li, Weiming Hu

Abstract

Siamese tracking has achieved groundbreaking performance in recent years, where the essence is the efficient matching operator cross-correlation and its variants. Besides the remarkable success, it is important to note that the heuristic matching network design relies heavily on expert experience. Moreover, we experimentally find that one sole matching operator is difficult to guarantee stable tracking in all challenging environments. Thus, in this work, we introduce six novel matching operators from the perspective of feature fusion instead of explicit similarity learning, namely Concatenation, Pointwise-Addition, Pairwise-Relation, FiLM, Simple-Transformer and Transductive-Guidance, to explore more feasibility on matching operator selection. The analyses reveal these operators' selective adaptability on different environment degradation types, which inspires us to combine them to explore complementary features. To this end, we propose binary channel manipulation (BCM) to search for the optimal combination of these operators. BCM determines to retrain or discard one operator by learning its contribution to other tracking steps. By inserting the learned matching networks to a strong baseline tracker Ocean, our model achieves favorable gains by $67.2 \rightarrow 71.4$, $52.6 \rightarrow 58.3$, $70.3 \rightarrow 76.0$ success on OTB100, LaSOT, and TrackingNet, respectively. Notably, Our tracker, dubbed AutoMatch, uses less than half of training data/time than the baseline tracker, and runs at 50 FPS using PyTorch.

Introduction

In the realm of visual object tracking, Siamese networks have become the dominant paradigm. These architectures typically bifurcate the tracking task into two distinct components: relation learning, which involves matching a target template to a search region, and state estimation, which refines the object's bounding box. While state estimation has benefited from advancements in object detection, the relation learning component has remained largely stagnant, relying on hand-designed, heuristic operators like cross-correlation.

The authors of "AutoMatch" argue that a single operator is insufficient for the diverse challenges of real-world tracking, such as occlusions, background clutter, and rapid scale changes. Manual design of these networks is not only tedious but often fails to capture the optimal data-driven relationships required for robust tracking.

Methods: The AutoMatch Framework

AutoMatch redefines matching network design as an architecture search problem. The framework introduces a "toolbox" of six diverse matching operators:

  1. Concatenation: Simple stacking of features.
  2. Pointwise-Addition: Element-wise fusion.
  3. Pairwise-Relation: A non-local attention variant measuring affinity between cells.
  4. FiLM: Feature-wise Linear Modulation for affine transformations.
  5. Simple-Transformer: Multi-head attention-based matching.
  6. Transductive-Guidance: Utilizing a pseudo-mask from the initial frame for spatial guidance.
Architectural diagrams of the proposed matching operators: (a) Concatenation, (b) Pointwise-addition, (c) Pairwise-Relation, and (d) FiLM.

To navigate this expanded search space, the authors developed Binary Channel Manipulation (BCM). This differentiable search algorithm assigns a learnable weight (manipulator) to every channel of an operator's output. Using the Gumbel-Softmax trick, the system can automatically decide which operators to retain or discard during training via bilevel optimization. A key finding of this research is that the classification and regression branches of a tracker benefit from different matching strategies, leading AutoMatch to implement decoupled, task-specific matching networks.

Key Insights and Efficiency

The study challenges the supremacy of depthwise cross-correlation, showing it is often outperformed by simpler or more complex operators depending on the environmental context. For instance, operators like Pairwise-Relation focus intensely on the target, while others like Concatenation exploit surrounding context to prevent overfitting to "easy" backgrounds.

Remarkably, this automated approach is highly efficient. AutoMatch requires less than half the training data and time compared to its baseline, Ocean, while achieving superior results. This suggests that a well-searched architecture can significantly reduce the computational burden of training high-performance trackers.

Experimental Results

AutoMatch was integrated into the Ocean tracker and evaluated against major benchmarks. The results demonstrate significant performance leaps:

  • OTB100: Success score increased from 67.2 to 71.4.
  • LaSOT: Success score improved from 52.6 to 58.3.
  • TrackingNet: Success score rose from 70.3 to 76.0.
Qualitative comparison of AutoMatch (Ours) against SiamRPN++ and Ocean on challenging tracking sequences involving occlusion and background clutter.

Despite the complexity of multiple operators, the final optimized model maintains a real-time speed of 50 FPS on standard hardware. It outperforms high-end trackers like DiMP and KYS without requiring expensive online updates.

Conclusion

AutoMatch demonstrates that moving away from "one-size-fits-all" matching operators toward an automated, task-specific design significantly enhances tracking robustness and efficiency. By decoupling the matching requirements for classification and regression, and providing a diverse operator toolbox, AutoMatch sets a new standard for Siamese tracking architectures. Future work may involve expanding the operator vocabulary and exploring the transferability of these searched architectures to different backbone networks.

Summary updated

Abstract

Siamese tracking has achieved groundbreaking performance in recent years, where the essence is the efficient matching operator cross-correlation and its variants. Besides the remarkable success, it is important to note that the heuristic matching network design relies heavily on expert experience. Moreover, we experimentally find that one sole matching operator is difficult to guarantee stable tracking in all challenging environments. Thus, in this work, we introduce six novel matching operators from the perspective of feature fusion instead of explicit similarity learning, namely Concatenation, Pointwise-Addition, Pairwise-Relation, FiLM, Simple-Transformer and Transductive-Guidance, to explore more feasibility on matching operator selection. The analyses reveal these operators' selective adaptability on different environment degradation types, which inspires us to combine them to explore complementary features. To this end, we propose binary channel manipulation (BCM) to search for the optimal combination of these operators. BCM determines to retrain or discard one operator by learning its contribution to other tracking steps. By inserting the learned matching networks to a strong baseline tracker Ocean, our model achieves favorable gains by $67.2 \rightarrow 71.4$, $52.6 \rightarrow 58.3$, $70.3 \rightarrow 76.0$ success on OTB100, LaSOT, and TrackingNet, respectively. Notably, Our tracker, dubbed AutoMatch, uses less than half of training data/time than the baseline tracker, and runs at 50 FPS using PyTorch. Code and model will be released at https://github.com/JudasDie/SOTS.
BibTeX
@inproceedings{zhang2021learn,
  author = {Zhipeng Zhang and Yihao Liu and Xiao Wang and Bing Li and Weiming Hu},
  title = {{Learn to Match: Automatic Matching Network Design for Visual Tracking}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2021},
  doi = {10.1109/iccv48922.2021.01309},
}
Figure