Language-Gated Memory for Tracking, Calibrated in Simulation and Checked on Real Video

Visual object trackers fail in a predictable way. After each frame, the tracker updates its picture of the target using its own guess about where the target is. When that guess is slightly wrong, for example because an occluder or a look-alike object crept into the box, the error enters the model and compounds until the tracker locks onto the wrong thing. This failure is called tracker drift. We propose to stop it by giving the tracker a referee that cannot be corrupted by its own mistakes. The referee is a fixed text description of the target, such as "a white sedan," together with short contrastive phrases such as "not the silver van beside it." These are turned into vectors by a frozen vision-language encoder. A new entry is added to the tracker's memory only when it still matches this language anchor. The tracker's own predictions never touch the anchor, so the self-reinforcing error loop is cut.

The second half of the proposal concerns how such claims get tested. Drift depends on how long an occlusion lasts and how similar the distractors are, and real video benchmarks cannot vary either of these on purpose. Simulators can. Results obtained only in simulation, however, are known to overstate real-world performance. We therefore use simulation as a controlled lab for drift, where we set gate thresholds and map failure curves, and then test explicitly whether the conclusions carry over to real benchmarks. Two questions get answered together: does language gating reduce drift, and does a simulator rank anti-drift methods the same way real video does?

Core Claim

Our hypothesis is that most catastrophic drift in memory-based trackers comes from wrong memory commits, meaning frames stored as "target" that actually show background, an occluder, or a distractor. A gate that compares each candidate commit against a frozen language concept should cut these wrong commits most in the regimes where motion-based gates fail: occlusions longer than a motion filter can bridge, and scenes with visually similar distractors.

A second hypothesis is about the threshold. A text-to-image similarity score depends on the meaning of the target rather than on pixel-level appearance. Its decision threshold should therefore transfer from simulator to real footage better than a threshold on visual-template similarity. This is the part we are least sure of, and the experiments are designed so that either outcome is informative.

Relation to Prior Work

Two papers motivate the use of language against drift, and both treat the description as fixed. The Siamese Natural Language Tracker in Siamese Natural Language Tracker turns a sentence embedding into a convolution kernel and shifts weight toward that kernel when the visual head becomes uncertain, measured by the entropy of its response map. It reports gains of 3 to 7 points in precision and success over SiamFC and SiamRPN-family baselines on LaSOT and OTB-99-LANG. The tracker in Real-time Visual Object Tracking with Natural Language Description matches region proposals to a description and rejects proposals that fit the text but break trajectory consistency. Both mix language into the localization score. Neither uses language to decide what enters memory, which is where drift is born.

Visual Language Tracking with Multi-modal Interaction supplies a constraint. Its VLT-MI benchmark injects corrected boxes and updated descriptions when drift occurs, and it finds that current vision-language trackers memorize the first-frame prompt and handle mid-sequence text updates poorly. Our design avoids this failure by construction. The text is never trained into the tracker's weights. It lives only in the gate, so replacing the description simply swaps the anchor.

The same benchmark also reports a tension we must respect. Detailed descriptions reduce interventions on the cluttered MGIT data but increase them on OTB99_Lang. This tension is why we borrow the two-level concept design of StructCBM, originally for few-shot text classification. That paper uses short prototypical concepts to say what a class typically looks like and discriminative concepts to separate it from its nearest confusers. In our gate, the prototypical phrase is always active. Discriminative phrases are consulted only when a distractor is present. As a result, uncluttered scenes see only concise text. StructCBM's quadratic growth in discriminative pairs does not apply here, because a tracker contrasts one target against the few distractors in view, not every class pair.

SAMURAI supplies the host architecture and the strongest motion-based competitor. It adds a Kalman filter to Segment Anything Model 2, a promptable video segmenter that keeps a memory bank of past frames. Memory commits are allowed only when a motion-overlap score, a mask score, and an object score all clear thresholds. Its authors note that the motion model is valid only if the target was seen within the last $T_{kf}$ frames, and that is exactly the regime we target. The older approach of freezing updates when correlation response quality drops, then running a sliding-window HOG-SVM re-detector (Long-term target tracking combined with re-detection), represents handcrafted confidence gating. The occlusion survey in The State-of-the-Art in Handling Occlusions for Visual Object Tracking documents the failure we aim to prevent: ungated updates during occlusion overwrite the target model with features of the occluding object.

On the simulation side, two papers constrain how far simulation results can be trusted. Drone active tracking in Open-World Drone Active Tracking with Goal-Centered Rewards and visuo-haptic pose tracking in V-HOP both show that results obtained in simulation degrade on real data. Both also use purely geometric targets with no language. We use AirSim, CARLA and GRADE as the controllable lab. We treat the reality-gap concern raised in The Reality Gap in Simulation Based Evaluation as a hypothesis to measure, not an assumption to accept or ignore.

Research Program

The gate. The method is a gate inserted into SAMURAI's memory-commit step. At initialization we form a prototypical text embedding $t_p$ from the target's category and attribute phrase. LaSOT, OTB99-Lang and TNL2K provide such descriptions. In simulation we generate them from the asset's metadata.

We also form discriminative embeddings $t_d^{(j)}$, one for each salient non-target object found in the first frame. Each comes from a single language-model call per object that writes a contrastive phrase. This costs a few calls per sequence, in the spirit of StructCBM's one-time 173-call construction budget on AGNews.

For each candidate commit at frame $k$, we compute an image embedding $v_k$ of the masked crop with the same frozen encoder. The candidate is scored as

$$ g_k = \cos(v_k, t_p) - \lambda \max_j \cos(v_k, t_d^{(j)}), $$

where $\lambda \ge 0$ weights the contrastive term and is set to zero when no distractor is present. A commit requires SAMURAI's three existing tests and $g_k > \tau$, for a single threshold $\tau$. When a VLT-MI intervention supplies new text, we recompute $t_p$ and leave everything else unchanged.

The simulation lab. We render sequences in AirSim and CARLA (vehicles, pedestrians, drones) and GRADE (indoor dynamic scenes). Two factors are varied on a grid. The first is occlusion duration, set as multiples of $T_{kf}$. The second is distractor similarity, using the same asset with recolouring and then an identical asset. Ground-truth masks come free from the renderer. This lets us measure the wrong-commit rate directly, defined as the fraction of committed memory entries whose mask overlap with the true target falls below 0.5. Real benchmarks with box-only annotations allow only approximate versions of this measure.

We also run a closed-loop variant in AirSim, where the tracker's box steers the camera, so drift changes what the tracker sees next. The threshold $\tau$ and weight $\lambda$ are chosen in simulation only.

Real-world validation. We freeze $\tau$ and $\lambda$ and evaluate on real video. We bin real sequences by the same two factors. Occlusion length comes from annotated absence frames in LaSOT and LaSOText. Distractor presence comes from attribute tags and MGIT. Within each bin we compare sim and real on three things: the ranking of methods, the size of the gain, and the occlusion length at which motion gating fails.

A real closed-loop test with a physical drone is a stretch goal at small scale. The primary real-world evidence is offline, and we say so plainly.

Baselines

The naive reference point is SAM 2 with its default fixed-window memory and no gating. SAMURAI's authors identify this as propagating errors during occlusion and crowding. It shows how much drift there is to remove.

The strongest existing method is SAMURAI itself, run from its released configuration with our gate as the only change. Both use identical backbones, inputs and sequences, and we tune nothing of SAMURAI's beyond its published thresholds. The file reports SAMURAI's results only as relative gains: a 7.1% AUC improvement on LaSOText and a 3.5% average-overlap gain on GOT-10k over prior zero-shot trackers. It reports no absolute AUC. We will therefore rerun it ourselves and report both. Beating it means a statistically reliable AUC gain on LaSOText together with a lower wrong-commit rate in the occlusion-longer-than-$T_{kf}$ and distractor bins, with no loss on the easy bins.

For language-guided tracking we include SNLT, which reports 3 to 7 points over Siamese baselines on LaSOT and OTB-99-LANG. It shows whether language works better in the score or in the gate. We also include the trackers evaluated in VLT-MI, which report a LaSOT success-rate rise from 53.2 to 56.3 under interaction. Under the VLT-MI protocol, beating them means a lower normalized intervention count (R-AMI). The file contains no reported R-AMI values, so we compute them for all methods.

The classical reference is response-quality gating with re-detection from Wang et al., which reports 81.4% precision and 59.9% success on OTB-2015 and 65.2% precision AUC and 46.1% success AUC on UAV123. It is relevant mainly for UAV123, where target loss is frequent.

The key ablation replaces the text anchor with a frozen image embedding of the first-frame target crop, using the same encoder, the same gate form, and the same simulation-only threshold search. This isolates whether the benefit comes from language or simply from having any frozen external reference. A second ablation removes the discriminative term ($\lambda = 0$). A third swaps in a long, detailed description in place of the prototypical phrase, which directly re-tests the VLT-MI granularity finding.

Why not just…?

Why not just rely on the Kalman filter gate in SAMURAI? For short occlusions, it may be enough, and we expect our gate to add little there. SAMURAI's own authors state that the motion model breaks once the target has been unseen for more than $T_{kf}$ frames. SNLT documents that look-alike distractors moving along similar paths defeat appearance-plus-motion reasoning. The occlusion-duration grid in simulation shows directly whether a crossover exists and where it sits. If the language gate shows no advantage beyond $T_{kf}$, the core claim is false.

Why not just gate on a frozen frame-1 image crop instead of text? This is the most serious threat. A frozen image anchor also breaks the self-grading loop and needs no language at all. Text should win when the target's appearance changes substantially through rotation, deformation or lighting, because a category description is indifferent to viewpoint while a first-frame crop is not. Text should also win when a threshold set on rendered images must transfer to real footage. The image-anchor ablation settles both questions. If it matches the text gate on real benchmarks and in threshold transfer, we will report that the mechanism is "frozen reference gating" and that language is not needed, and we will reframe the contribution around that finding.

Why not just validate on LaSOT and skip the simulator? Real benchmarks are where final claims must be made, and we make them there. They cannot isolate occlusion duration or distractor similarity, they lack per-frame masks needed to measure wrong commits exactly, and their camera paths are recorded, so drift never alters what the tracker sees next. Simulation adds controlled variation, exact measurement and closed-loop feedback. The sim-to-real agreement test then shows whether those simulated conclusions can be trusted. If agreement is poor, that negative result is itself useful to anyone validating trackers in AirSim or CARLA.

Why not just prompt a large multimodal model on every frame? SNLT runs at 50 frames per second and the description-matching tracker of Feng et al. runs above 30. Querying a large multimodal model on every frame is far too slow for closed-loop use. VLT-MI also finds that large tracking models adapted to text memorize prompts. Our gate calls a language model only at initialization and at interventions.

Proposed Experiments and Decisive Checks

On real data we report AUC, precision and success on LaSOT, LaSOText, UAV123, OTB99-Lang and GOT-10k, and R-AMI and AMSL under VLT-MI on LaSOT, TNL2K and MGIT. All results are broken down by occlusion-length and distractor bins. In simulation we report wrong-commit rate, frames until first drift, and the probability of recovering after an occlusion of length $L$, both open-loop and closed-loop.

The project succeeds if four results hold together. First, the language gate lowers wrong-commit rate, and that reduction accounts for most of its AUC gain over SAMURAI. Second, the gain concentrates where occlusions exceed $T_{kf}$ and where distractors are present. Third, text beats the frozen-image anchor on at least the appearance-change and transfer tests. Fourth, a threshold chosen only in simulation comes within a small margin of one tuned on real validation data, and does so more closely for the text gate than for the image gate.

Separately, we report the rank correlation (Kendall's tau) between simulated and real method orderings within each bin. This is the evidence on whether simulation is a valid proxy for drift. It is reported whichever way it comes out.