Reusing a Pretrained Correspondence Decoder as the Matcher in Siamese Trackers

Siamese trackers follow an object through a video by comparing a small picture of the target (the "template") against a larger search region in each new frame. Two twin branches with shared weights encode both images, and a matching step produces a similarity map. Separately, self-supervised video pretraining has produced encoders that are very good at this kind of frame-to-frame correspondence. Siamese Masked Autoencoders train a weight-sharing Vision Transformer, plus a cross-attention decoder that rebuilds a heavily masked frame by looking up an unmasked earlier frame. The standard move is to keep the encoder, throw away the decoder, and attach a tracker on top. We propose the opposite. The decoder is itself a learned matcher: it already answers "where in the reference frame does this patch come from?" It should become the tracker's template–search comparison module. It would be brought into service through a staged curriculum that first adapts the tracking heads, then gradually shifts the decoder from masked to unmasked queries, and only then unfreezes everything.

The opportunity matters because of how the main tracking benchmark is built. GOT-10k tests on object classes that never appear in training, so generalizing correspondence to new object types matters more than memorizing category appearance. The core claim fits in two sentences. A matcher pretrained for generic correspondence should transfer to unseen classes better than a matching module learned from scratch on tracking labels. It must be introduced in stages, because its pretraining inputs (95%-masked query tokens) differ sharply from tracking inputs (fully visible search regions), and naive joint training would overwrite the pretrained matching behavior before the new heads are useful.

Where this sits in prior work

The mechanism comes from Siamese Masked Autoencoders (SiamMAE). It feeds one frame unmasked and a second frame 95% masked to a shared ViT, and uses a cross-attention decoder to reconstruct the masked frame. Using k-NN matching on frozen features, it reaches 71.4% J&F on DAVIS-2017, against 69.9% for DINO. SiamMAE never trains a downstream box-regression head and never fine-tunes. We take its encoder and its discarded decoder into a supervised tracker.

The tracker template is SiamCAR. It uses a Siamese ResNet-50, depth-wise cross-correlation, and three pixel-level heads (foreground classification, distances to box edges, and center-ness), all trained jointly from the start. It reports 0.581 Average Overlap on GOT-10k, against 0.517 for SiamRPN++ and 0.556 for ATOM. SiamBAN is a similar anchor-free design, reporting 0.452 Expected Average Overlap on VOT2018. We keep their head design and replace depth-wise cross-correlation with the pretrained cross-attention decoder.

The closest conceptual precedent is Siamese DETR. It showed that pretraining the task-specific Transformer modules of a detector, not just the backbone, improves strict-threshold localization and data efficiency: it matches full-data PASCAL VOC performance with 30% of the labels. It pretrains on augmented views of static images, and it pretrains the backbone separately. Our proposal differs in three ways. The task is tracking. The pretraining signal is real temporal correspondence in video. The encoder and the matching module are pretrained together in a single stage by SiamMAE, which addresses the unified-pretraining gap that Siamese DETR itself names.

The staging design is constrained by the staged-optimization literature. Here, modular pretraining followed by frozen assembly and then joint fine-tuning repeatedly beats both extremes:

  • Separate vs. staged-and-joint training. Deep LSTM adaptive beamforming networks reports 24.91% vs 32.88% word error rate on CHiME-3 real. Joint fine-tuning helps most after separate pretraining.
  • Monolithic end-to-end vs. progressive training. Progressive Joint Modeling for overlapped speech reports over 30% relative error reduction over a jointly trained system.
  • Generalization to new environments. StereoNavNet freezes perception while training a navigation policy, then fine-tunes end-to-end. It reaches 35% vs 21% success in unseen scenes.

These papers motivate staging, but none of them involve a pretrained module whose input statistics change between stages. That mismatch is the specific problem our Stage 2 is built to handle.

Core Claim

A SiamMAE cross-attention decoder, reused as the template–search matcher, will outperform a randomly initialized matcher of identical architecture when both are brought into the tracker by a three-stage curriculum. The key step in that curriculum is annealing the search-frame mask ratio from 95% to 0% while the encoder and decoder are frozen. We expect the advantage to be largest in two places: on GOT-10k's unseen classes, and when tracking labels are scarce. The staged recipe should also keep more of the encoder's generic correspondence ability than direct end-to-end fine-tuning does, which we measure by re-running SiamMAE's DAVIS-2017 k-NN evaluation after tracking training.

Research Program

Stage 1: pretraining. Stage 1 is SiamMAE pretraining on Kinetics-400 with a ViT-S/8, as in the original paper. Alternatively, we reuse released weights if available.

Stage 2: assembly with the pretrained modules frozen. - Attaching the heads. The template crop and the search region pass through the shared encoder. Search tokens act as queries in the decoder's cross-attention over template tokens. The decoder's output tokens, reshaped into a map, feed SiamCAR-style classification, box-distance and center-ness heads. - Mask annealing. The decoder was only ever trained to answer queries from mostly masked frames. We therefore begin Stage 2 with 95% of search tokens replaced by the mask token and linearly anneal toward 0%. Masked positions receive only the classification loss, since their box-distance targets are still defined. - What trains, and when. The encoder stays frozen throughout Stage 2. The decoder is frozen for the first half of Stage 2 and gets a small learning rate in the second half.

Stage 3: joint fine-tuning. Everything is unfrozen and trained on $$L = L_{\text{track}} + \lambda L_{\text{rec}},$$ where $L_{\text{track}}$ is the SiamCAR loss (classification, box-distance and center-ness terms), $L_{\text{rec}}$ is SiamMAE's masked-frame reconstruction loss computed on frame pairs from the tracking training videos, and $\lambda$ sets how strongly the original correspondence objective is preserved. We sweep $\lambda \in \{0, 0.1, 1\}$. The case $\lambda=0$ is plain staged fine-tuning.

Analysis of what the matcher learned. We compare the decoder's cross-attention maps before and after tracking training on GOT-10k test sequences. We measure the fraction of attention mass that falls inside the ground-truth target box. This tests whether pretrained matching behavior survives or is rewritten.

Scope. We deliberately leave out online template updating and multi-frame memory. Static template decay is a known Siamese weakness, but it is orthogonal to this question.

Baselines

Naive reference points. - Frozen SiamMAE features with no trained head. Localization is done by k-NN label propagation of the first-frame box mask, which is SiamMAE's own evaluation mode applied to boxes. - Randomly initialized ViT-S/8 trained end-to-end. This measures how much any pretraining contributes.

Strongest published competitor. SiamCAR reports 0.581 AO on GOT-10k SiamCAR. SiamBAN reports 0.452 EAO on VOT2018 SiamBAN. For LaSOT and TrackingNet, the file lists SiamCAR and SiamBAN as evaluated but gives no numbers, so there is no reported result found on Lacuna for those comparisons. We will retrain SiamCAR ourselves under our protocol.

Comparing to published numbers alone would confound backbone architecture and pretraining data with our contribution. We therefore also run SiamCAR's depth-wise cross-correlation matcher on top of three encoders, all ViT-S/8 with the same search and template resolution, the same GOT-10k-train-only data (per GOT-10k's one-shot protocol), the same number of optimizer steps, and the same hyperparameter search budget of eight configurations each:

  • the SiamMAE encoder;
  • a DINO-initialized encoder (DINO trails SiamMAE by only 1.5 J&F on DAVIS);
  • an ImageNet-supervised encoder.

The ablation that isolates the key ingredient keeps our full architecture and three-stage schedule but replaces the pretrained decoder with a randomly initialized decoder of identical shape. Two further ablations remove mask annealing (Stage 2 starting at 0% mask) and remove staging entirely (all modules trained jointly from step zero, with a lower encoder learning rate).

What counts as beating a baseline. We claim success only if both conditions hold:

  1. The pretrained-decoder tracker exceeds the random-decoder tracker and the SiamMAE-plus-cross-correlation tracker by at least 1.5 AO points on GOT-10k test, averaged over three seeds with non-overlapping seed ranges.
  2. It at least matches our retrained SiamCAR.

Why not just…?

Why not just fine-tune the SiamMAE encoder end-to-end with a lower backbone learning rate? This is the most serious threat, because layer-wise learning-rate reduction often recovers most of what staging buys. It has a specific weakness here. The heads start random while the decoder receives fully visible queries it was never trained on, so early gradients through the decoder are both large and pointed in arbitrary directions. The staged-optimization results above show that such early interference costs accuracy. The no-staging ablation, run under an identical step budget with its own tuned backbone learning rate, settles the question directly. If it matches the staged version, we will report that staging is unnecessary and that the gain comes from the decoder reuse alone.

Why not just keep the pretrained modules frozen and train only the heads? SiamMAE's strength is established only under frozen k-NN evaluation, so a frozen encoder with a trained head is a natural cheap option. The beamforming result shows frozen separately-trained modules leaving large error on the table (32.88% vs 24.91% word error rate), and SiamMAE was never trained to produce box geometry. Our schedule contains this baseline for free: it is the Stage 2 endpoint. The Stage 2 to Stage 3 difference measures exactly what unfreezing adds.

Why not just use the standard depth-wise cross-correlation on SiamMAE features? This is what SiamCAR and SiamBAN do. It is parameter-free and fast, in the spirit of the inner-product matching in Efficient Deep Learning for Stereo Matching. If generic encoder features already carry all the correspondence information, a learned matcher adds latency for nothing. The cross-correlation baseline on the same SiamMAE encoder tests this. We also report frames per second for both variants, since a gain bought with a large speed loss would weaken the claim.

Why not just use a stronger image-pretrained backbone? Much of the gain could simply come from a better encoder rather than from reusing the decoder. The DINO and ImageNet encoder controls hold the architecture fixed. Even so, only the random-decoder ablation cleanly separates "better features" from "pretrained matcher."

Proposed Experiments

Main comparison. All variants are trained on GOT-10k train and evaluated on:

  • GOT-10k test (AO and success rates at 0.5 and 0.75 overlap);
  • LaSOT (success and precision) for long sequences with heavy occlusion;
  • TrackingNet;
  • VOT2018 (EAO), for comparison with SiamBAN's reported 0.452.

Label-scarcity experiment. Following Siamese DETR's data-efficiency framing, we repeat the key comparison (pretrained vs random decoder, staged vs unstaged) using 10%, 30% and 100% of GOT-10k training videos. We predict the pretrained-decoder margin grows as labels shrink.

Forgetting check. After tracking training, we re-run DAVIS-2017 and JHMDB k-NN evaluation on each variant's encoder. The references are SiamMAE's reported 71.4% J&F and 61.9 [email protected]. We predict that staged training with $\lambda>0$ keeps DAVIS J&F within a few points of 71.4, while unstaged fine-tuning loses more.

Decisive checks. The proposal stands or falls on three results:

  1. Pretrained vs random decoder under the same staged schedule. If this gap is absent, the central claim fails.
  2. Mask annealing vs no annealing. If annealing does not matter, the train/test mismatch argument is wrong, and Stage 2 reduces to ordinary head warm-up.
  3. Unseen-class gain vs gain on classes resembling training. Using GOT-10k's disjoint-class test set, compared against the pooled LaSOT result, this tells us whether the benefit really comes from transferable correspondence rather than extra capacity.