For years, Siamese networks were the gold standard for real-time visual tracking due to their elegant framing of the problem: tracking is simply a similarity match between a "template" (the object) and a "search region" (the current frame). However, while the rest of computer vision transitioned to massive architectures like ResNet, Siamese trackers remained stuck using shallow backbones like AlexNet.
The transition to deeper networks was blocked by two primary issues: a massive explosion in parameters during the feature fusion stage and a subtle "center bias" caused by zero-padding. The introduction of Depth-wise Cross-Correlation (DW-XCorr) solved both, enabling the first generation of truly deep Siamese trackers.
The Bottleneck of Standard Correlation
In a traditional Siamese tracker, the template and search features are fused using a standard cross-correlation layer. This layer treats the template as a convolution kernel and slides it across the search region.
If you use a deep backbone like ResNet-50, the number of feature channels increases significantly (e.g., to 256 or 512). In a standard correlation, every channel in the template interacts with every channel in the search region. This creates a "heavy" tracking head with an enormous number of parameters. As Bo Li and Qiang Wang observed in SiamRPN++, this parameter imbalance makes training unstable and prevents the model from benefiting from the representational power of deep networks.
Depth-wise Correlation and Semantic Orthogonality
The breakthrough was to replace dense correlation with depth-wise separable correlation. Instead of mixing all channels, DW-XCorr performs correlation channel-by-channel. The $i$-th channel of the template is only compared against the $i$-th channel of the search region.
This works because of a phenomenon called semantic orthogonality. In deep networks, different feature channels naturally learn to specialize. One channel might activate for "round shapes," another for "red textures," and another for "high-contrast edges."
As Ngai-Man Cheung and Yiluan Guo argue in their work on Multi-Level Similarity, comparing these channels directly preserves their semantic integrity. If the template has a strong "red texture" signal in channel 42, we only care if the search region also has a "red texture" signal in channel 42. Mixing channel 42 with channel 10 (e.g., "vertical lines") just adds noise. By restricting the comparison to direct counterparts, the network achieves:
- Lower Parameter Count: The tracking head becomes lightweight, allowing the backbone to do the heavy lifting.
- Higher Precision: The model ignores irrelevant cross-channel interference.
- Stable Training: The gradients are more focused, allowing for the successful training of deep architectures.
Breaking the Center Bias
Even with efficient correlation, deep networks initially failed at tracking because of "center bias." Deep architectures like ResNet require extensive padding to maintain spatial resolution. Bo Li identified that this padding inadvertently teaches the network that the target is always near the center of the search crop.
In SiamRPN++, the researchers solved this by using a spatial shift training strategy. By randomly shifting the target's location during training, they broke the network's reliance on the center-bias "cheat," forcing it to actually learn the similarity matching provided by the depth-wise correlation.
Evolution to Anchor-Free Tracking
The efficiency of DW-XCorr opened the door for more sophisticated tracking heads. While earlier models relied on predefined "anchor boxes" to guess object sizes, newer models like SiamCPN by Fan Tang and Weiming Dong use the fused features to predict the target's center point and size directly.
Because the depth-wise correlation provides such a clean, channel-separated map of similarities, these "anchor-free" models can treat tracking as a simple keypoint detection problem. This eliminates the need for complex hyperparameter tuning (like anchor ratios) and results in trackers that are both more accurate and significantly faster.