Visual object trackers follow one object through a video after being shown it once in the first frame. Modern trackers replaced small convolution filters, which only see a few neighboring pixels at a time, with attention layers that let every image location read from every other location. That wider view helps match a deforming target. We argue it also has an unexamined cost under partial occlusion. When a tree branch covers half of a tracked car, global attention mixes branch features into the features of the car's visible half, and then into the stored template. Local filters would have kept that damage confined to the covered region. Our proposal is to give each image patch a learned visibility score, let attention read only from patches judged visible, and update the stored template patch by patch rather than all at once. The visible parts of the target then keep global context without absorbing the occluder.
This matters because occlusion remains a main way trackers fail. The GOT-10k benchmark labels every frame with the fraction of the target that is visible, and reports that performance "drops precipitously once an object is more than 50% occluded." Current occlusion defenses act on whole frames. STARK updates its template only when a confidence head approves the frame, and Jin et al. freeze the template when a verification module detects heavy occlusion. Jin et al. report that this leaves the tracker with a stale template after long occlusions. We think the right unit for both reading and writing is the patch, not the frame.
Core Claim
The claim has two testable parts. The first is diagnostic. In attention-based trackers, how much an occluder corrupts the features of the target's visible region should grow with receptive-field size and with attention depth. We will measure this directly. The second part is constructive. Gating attention by predicted per-patch visibility, and writing to the template only at visible patches, should remove most of that spread. It should also move the performance cliff below 50% visibility further down. It should do so at nearly the cost of an ordinary transformer tracker, and it should beat the simpler fix of training the same tracker on occlusion-augmented data.
Concretely, let the tracker's features be split into patch tokens, and let $v_j \in [0,1]$ be the predicted probability that token $j$ shows the unoccluded target (or clean background, for search-region tokens). Every attention layer in the template–search encoder then uses
$$a_{ij} = \mathrm{softmax}_j\!\left(\frac{q_i^\top k_j}{\sqrt{d}} + \log(v_j + \epsilon)\right),$$
where $q_i$ and $k_j$ are the query and key of tokens $i$ and $j$, $d$ is the key dimension, and $\epsilon$ is a small constant for numerical stability. Occluded tokens can still be queried and updated, but they barely broadcast. The dynamic template token $t_j$ is updated from the matched search token $s_j$ as
$$t_j \leftarrow (1-\alpha v_j)\, t_j + \alpha v_j\, s_j,$$
where $\alpha$ is a fixed update rate. The visible parts of the template keep tracking appearance change while the target is partly hidden, and hidden parts stay as they were last seen. This targets the stale-template failure that frame-level freezing causes.
Relation to Prior Work
Two older lines of work motivate the part-level view. PMT treats occlusion as a sparse error on top of low-rank part appearance across frames, solved by iterative optimization on hand-crafted features. Voting with salient local features lets each visible local feature vote for the object center. Part matching in crowds uses unoccluded body parts as identity anchors. All three show that treating visible and hidden parts separately is what buys robustness. None of them has learned global context. We keep their separation idea, in soft learned form, inside a global-attention network.
Three trackers already combine global receptive fields with some occlusion handling, and they are the closest work. SiamGAT matches every template location to every search location with graph attention, turning holistic matching into part-to-part matching. It has no occlusion estimate, so occluded template nodes still send attention, and its template never updates. STMTrack reads pixel-level features from a memory of six past frames, so a visible head can match an earlier unoccluded head. It stores whole frames and applies no visibility weighting. STARK has global attention and a dynamic template, but its gate decides for the whole frame. Jin et al. are closest in spirit. They train self-attention over template sub-blocks with masking so that the network fills in damaged blocks, and they add a separate gate that blocks updates. Their attention stays inside the target template, their template–search interaction is correlation, and their update decision is again all-or-nothing. We differ from all four in the same way: visibility is a per-token quantity that shapes every attention read and every template write.
The receptive-field literature supplies the building blocks and one constraint. Non-local blocks and their compact generalization, criss-cross attention, and feature broadcasting all widen what each location can see. None of them asks what happens when part of the input is corrupted. The research file notes that global attention has no structural prior, so it can mistake distant clutter for related foreground. Our diagnostic study measures that risk directly, and our visibility term is one structural prior against it. We use criss-cross attention only as a cheaper encoder variant, not as the contribution.
Research Program
The first step is the contamination study. We take trained trackers with growing receptive fields: a convolutional Siamese baseline, SiamGAT, STMTrack, and STARK. On sequences where the target is fully visible, we paste real occluder crops (other objects from the training set) over a controlled fraction of the target box. For each tracker we measure how far the features of the still-visible target tokens move, compared with the clean frame, as the covered fraction changes. We report this per layer and in the template after one update. If global attention spreads occlusion damage, visible-token drift should grow with depth and be larger for the attention trackers than for the convolutional one. If the drift does not grow, the premise of this proposal is wrong, and we would report that.
The second step builds the tracker. We start from a STARK-style encoder–decoder with a ResNet-50 backbone, the S50 setting. We add a small visibility head: a per-token classifier over template and search features. We use the gated attention and the patch-wise template update defined above. Visibility supervision comes free from the same pasting procedure. Every token covered by a pasted occluder is labeled occluded, which follows Jin et al.'s use of synthetic occlusion for training their verifier. The whole tracker trains end to end, with a binary cross-entropy loss on visibility added to the usual box losses. A second variant replaces dense encoder attention with two passes of criss-cross attention. This tests whether the gating survives the cheaper attention that edge trackers such as Jin et al.'s need.
The third step studies failure modes. We examine whether predicted visibility collapses to near zero everywhere during full occlusion, and whether the tracker then re-detects the target from the initial template. We also check whether visibility is miscalibrated on classes never seen in training, which is the setting GOT-10k tests.
Baselines
The naive reference point is the unmodified STARK-S50 retrained in our codebase with the same data, schedule, and seeds. Every other variant is judged against it.
The strongest existing methods are STARK, STMTrack, and SiamGAT, each run with its public training recipe and backbone. STARK is the most direct competitor because it already has global attention and an occlusion-aware update, differing from us only in frame-level versus patch-level gating. No reported result was found on Lacuna for STARK's GOT-10k or LaSOT scores, so we will reproduce it. For the published numbers we must beat, the file reports a GOT-10k average overlap of 0.642 for STMTrack and 0.627 for SiamGAT. It reports a LaSOT success (AUC) of 0.617 for CapsuleRRT, a TrackingNet AUC of 80.3% for STMTrack, and a UAV20L success rate of 70.6% for Jin et al.. Beating a baseline means a gain in average overlap on GOT-10k frames with less than 50% visibility that exceeds the spread across three seeds, without losing overall AO or LaSOT AUC.
The most important baseline is STARK trained with exactly the same occluder-pasting augmentation but no visibility head. It sees the same extra data and the same occlusion signal. Only the architectural use of visibility is missing. Two ablations isolate the key ingredient. One keeps gated attention but restores STARK's frame-level template update. The other keeps the patch-wise update but uses ungated attention. A third variant feeds ground-truth visibility masks on synthetic occlusions. It gives an upper bound and separates errors in the visibility head from the value of gating itself. All variants share backbone, training data (the GOT-10k train split for GOT-10k evaluation, the standard training mix otherwise), epochs, and a matched hyperparameter search of equal size.
Why not just…?
Why not just train an existing transformer tracker with occlusion augmentation? This is the real threat. The attention weights might learn to ignore occluder tokens on their own once they see enough pasted occluders. The augmentation-only STARK baseline tests exactly this. The contamination measure makes it sharper. If augmentation alone brings visible-token drift down to the level of the gated model, the architectural change is not needed, and we will say so. We expect a gap mainly at low visibility and after long occlusions. Augmentation cannot change the fact that STARK's template update is all-or-nothing.
Why not just use a frame-level gate, as STARK and Jin et al. already do? Frame-level gating protects the template, but it freezes the whole template, including the parts that are still visible. Jin et al. report that their tracker loses targets that re-emerge after extended occlusion with changed appearance. The ablation that swaps our patch-wise update for STARK's frame-level rule measures this cost directly on LaSOT and UAV20L, whose long sequences contain prolonged occlusion.
Why not just shrink the receptive field, using local or windowed attention, so occluders cannot spread? If our diagnostic is right, limiting attention to local windows would reduce contamination. It would also give up the long-range matching that let SiamGAT and STARK handle deformation. The STARK ablation cited in the file reports that removing its global encoder costs 5.3% LaSOT success. We will include a windowed-attention variant of the same size. If it matches the gated model on low-visibility frames while keeping overall accuracy, the simpler answer wins.
Why not just add more memory frames, as in STMTrack? More memory gives more chances that some stored frame shows each part unoccluded. Retrieval is still purely by appearance, though, and memory writes are unweighted, so occluded frames enter the memory too. We will run STMTrack with a larger memory at matched compute. We will also test adding our visibility weighting to its memory readout, to see whether the idea transfers beyond STARK.
Proposed Experiments
Main evaluation is on GOT-10k under its one-shot protocol, where test classes never appear in training, and on LaSOT, TrackingNet, UAV123, and UAV20L with their standard success, precision, and AUC metrics. We also report speed in frames per second on one GPU type. The key analysis stratifies average overlap by labeled visible ratio in 10% bins on the GOT-10k validation set. It also stratifies by synthetic occlusion fraction on clean sequences, which gives controlled curves down to 10% visibility.
Three results would decide the question. First, the contamination study must show that visible-token drift grows with receptive field in existing trackers; otherwise the motivation fails. Second, the gated model must shift the visibility cliff: at 30–50% visibility it should beat augmentation-only STARK by more than the seed spread. The ground-truth-mask variant should show how much headroom remains. Third, on LaSOT and UAV20L, the patch-wise update must reduce failures after re-emergence compared with the frame-level update. We count these failures directly as frames within 50 frames after an occlusion ends where overlap stays below 0.1. If the gains appear only with oracle masks, the bottleneck is estimating visibility, not routing, and the next step is a better visibility predictor rather than a new attention design.
References
- GOT-10k: A Large High-Diversity Benchmark for Generic Object Tracking in the Wild
- Learning Spatio-Temporal Transformer for Visual Tracking
- An Occlusion-Aware Tracker With Local-Global Features Modeling in UAV Videos
- Partial Occlusion Handling for Visual Tracking via Robust Part Matching
- Initialization-Insensitive Visual Tracking through Voting with Salient Local Features
- Tracking People in Crowds by a Part Matching Approach
- Graph Attention Tracking
- STMTrack: Template-Free Visual Tracking With Space-Time Memory Networks
- Compact Generalized Non-local Network
- CCNet: Criss-Cross Attention for Semantic Segmentation
- Broadcasting Convolutional Network for Visual Relational Reasoning
- CapsuleRRT: Relationships-Aware Regression Tracking via Capsules