Compile the Sentence Once: Language-Weighted Hybrid Features for Real-Time Natural-Language Tracking

In tracking by natural language, a system is given a sentence such as "the red car turning left behind the bus" and must follow that object through a video. Current unified trackers pay for the sentence on every frame. They run a transformer that jointly attends over the text tokens, stored templates and the current search region. Yet the sentence never changes during a sequence. We propose to pay the cross-modal cost once, at the first frame. At that point the sentence and the grounded target are "compiled" into a small set of weights over a cheap bank of features. That bank mixes classical handcrafted descriptors, such as Color Names (a descriptor built from eleven basic linguistic color terms) and gradient histograms, with a small distilled convolutional stream. From then on, a correlation filter (a tracker that finds the target by fast Fourier-domain template matching) runs on the reweighted features at correlation-filter speed, with no per-frame text processing.

The bet is that language can be fused into cheap features early and statically, rather than into expensive features late and repeatedly. The handcrafted channels matter here for a specific reason. Many attribute words in tracking sentences, like colors, "striped" or "bright", name exactly what those channels measure. So a projection from language onto them is a real, checkable translation, not the opaque trigger that projectors into large language models turn out to be. If this works, a vision-language tracker could run above 100 frames per second (FPS) on a desktop GPU and at usable rates on a CPU, while keeping most of the accuracy of per-frame fusion.

Core Claim

For a fixed sentence, most of the benefit of per-frame cross-modal attention can be captured by a sequence-level early fusion. A sentence embedding $s$ is projected once into channel gates $g_k(s)$ and a coarse spatial prior over a hybrid feature bank $\{\phi_k(x)\}$ computed on frame $x$. The tracker then operates on

$$z(x) = \sum_k g_k(s)\, \phi_k(x),$$

where $k$ ranges over handcrafted channels (Color Names, HOG, intensity histograms) and a few distilled deep channels. A correlation filter is learned on $z$ at the first frame and updated online as usual. The gates are recompiled only when tracking confidence collapses. This is a cheap reweighting, not a handoff to a heavy grounding model.

The claim is falsifiable in three ways. It fails if the gates carry no information beyond what the first-frame box already provides. It fails if the handcrafted channels add nothing once distilled deep channels are present. It fails if recompilation cannot keep up with semantic appearance change.

Relation to Prior Work

Two tracking-with-language systems motivate the problem. JointNLT unifies grounding and tracking in one transformer that fuses sentence, templates and search features at every frame. It uses the sentence's global embedding to filter historical templates, and reaches 54.6% area under the success curve (AUC) on TNL2K at 39 FPS on an RTX 3090. We keep its lesson that language should shape the visual representation rather than sit in a separate module. We drop its per-frame fusion. AdaSwitcher introduced TNL2K and couples a SiamRPN++ local tracker with a BERT-plus-CNN global grounder that fires when an anomaly detector flags a "confidence trap". It scores 0.422 success. Our recompilation also fires on low confidence, but it only rewrites a vector of gates and keeps a single tracker. Nothing is switched.

The efficiency side supplies our method. TRACA compresses deep features with a bank of category-specific autoencoders chosen at the first frame, reaching 101.3 FPS and 89.8% precision on OTB-50. Its choice of expert from the initial region is a coarse, vision-only form of what we do with language. It also notes failure under drastic appearance change, which constrains our design and motivates recompilation. Feature Distilled Tracking trains a four-layer student to mimic VGG-19 conv4_4, running at 42.5 FPS on GPU and 7.8 FPS on CPU versus 8.2 and 1.5 FPS for hierarchical convolutional features. We reuse this recipe for the deep channels in our bank.

OursCAFF shows that carefully fused handcrafted features in a correlation filter can beat deep RGB-thermal trackers on VOT-RGBT2019 (accuracy 0.569). Its authors also state that handcrafted features have hit a ceiling. We treat both statements as true. Handcrafted channels alone are not the proposal; they are cheap, nameable anchors next to a distilled deep stream. The hybrid-fusion literature (Boosting-like Deep Learning, Feature Fusion Net) supports feeding such channels into small networks, but none of it conditions the channels on language.

On the projection side, LightningDOT is the closest structural precedent. It moves cross-modal interaction into training so that inference is a cheap dot product, gaining speedups of 1,927x on COCO. It targets static retrieval, still pays for Faster R-CNN region extraction, and produces no spatial response. We apply the same "fuse offline, stay cheap online" logic to dense, per-frame localization.

Cross-Modal Projection in Multimodal LLMs Doesn't Really Project Visual Attributes to Textual Space finds that projector outputs lose attribute information after fine-tuning, with probe F1 falling 27.49% on PlantDoc. This constrains us: a learned projection can look useful while encoding nothing interpretable. So we project in the opposite direction, from text onto named visual channels, and measure whether the gates track the attribute words.

Research Program

The system has three parts.

The first is the feature bank. It holds Color Names, HOG, a local intensity histogram, and a distilled student of a modern tracker backbone trained FDT-style with a shift-and-stitch step. All channels share one resolution.

The second is the compiler. A frozen text encoder produces $s$. A small network maps $s$, together with features pooled from the first-frame box, to channel gates $g(s) \in [0,1]^K$ and a low-resolution spatial prior. When no box is given, the prior is supplied by running a grounding model once at the first frame. The compiler is trained on TNL2K training sequences. The correlation filter is learned in closed form per sequence, and the loss is back-propagated through the filter's first-frame solution into the compiler, in the style of DCFNet. An auxiliary loss asks the Color Names gates to agree with color words parsed from the sentence. This loss is used only in one arm, so we can test whether interpretability costs accuracy.

The third is recompilation. When the filter's peak-to-sidelobe ratio stays low for several frames, the compiler reruns. Its inputs are the sentence plus features pooled from the top few response peaks, and it picks gates that best separate the sentence-consistent peak. The cost is one small forward pass, amortized over the sequence.

The analysis half of the program asks where language actually helps. We slice TNL2K results by attribute, especially distractors, adversarial samples and modality switch, and by whether the sentence contains color, texture or relational words. Relational phrases like "behind the bus" should gain little from channel gates, and the method predicts as much. That is an honest limit, stated in advance.

Baselines

The naive reference is the identical correlation filter and hybrid bank with uniform gates. This is a fast bounding-box-only tracker that ignores the sentence and runs at exactly our speed. It isolates what language contributes. Beating it by a clear AUC margin on TNL2K, especially on the distractor subset, is the minimum bar. No reported TNL2K result was found on Lacuna for this configuration, so we will run it ourselves.

The strongest existing method is JointNLT, at 54.6% AUC on TNL2K and 39 FPS on an RTX 3090. We will run its released model on the same hardware we use, including a CPU and an embedded GPU, and report accuracy and FPS side by side. "Beating" it does not mean higher AUC. It means dominating it on the accuracy-latency curve: within about three AUC points at more than twice the FPS on the same GPU, and real-time on CPU where JointNLT is not.

AdaSwitcher, at 0.422 success on TNL2K, is the modular reference; we will cite its reported number and rerun it where the code allows. Fast bounding-box-only transformer trackers indexed alongside TNL2K, such as MixFormerV2 and ZoomTrack, are the most dangerous competitors. No reported TNL2K number was found on Lacuna for either, so we will run their public checkpoints under our timing protocol.

Two ablations isolate the key ingredients. The first removes the handcrafted channels and gates only the distilled deep stream. The second replaces sentence-conditioned gates with TRACA-style gates selected from the first-frame box alone. All learned variants share one training set (TNL2K train) and the same hyperparameter search budget, and are timed with the same batch size of one on the same hardware.

Why not just…?

Why not just use a fast bounding-box tracker and ignore the sentence? This is the real threat. When a box is given, much of JointNLT's accuracy may come from its visual backbone, not the text. The uniform-gate baseline and the transformer trackers above settle this directly. If sentence gates do not beat uniform gates on the distractor and semantic-change subsets, the claim is wrong for box-initialized tracking. The proposal would then shrink to the language-only initialization setting, where a bounding-box tracker cannot start at all.

Why not just distill JointNLT into a small student? FDT-style distillation shrinks the backbone, but it keeps a per-frame fusion transformer over text and templates. That is the cost we argue is unnecessary. We will run this comparison: a distilled JointNLT with a matched parameter count. If it matches our AUC at our FPS, the "compile once" idea buys nothing, and we will report that.

Why not just re-ground with a heavy model when tracking fails, as AdaSwitcher does? Its reported 0.422 trails JointNLT's 54.6% AUC, which suggests that switching between disjoint modules loses context. Our recompilation keeps one tracker and one filter state. The recompilation-versus-switch arm, with the grounder called at the same trigger points, tests whether reweighting is enough.

Why not just gate deep features and skip handcrafted descriptors, given the ceiling OursCAFF reports? Possibly the right answer. The deep-only ablation decides it. The handcrafted channels have to earn their place in one of two ways. They can improve CPU latency at equal accuracy, or they can make the gates interpretable, measured as agreement between Color Names gates and color words, without losing AUC. If neither holds, we drop them.

Proposed Experiments

The main experiment is on TNL2K with a standard split. It reports AUC and precision overall and per attribute, together with FPS on an RTX-class GPU, a laptop CPU and an embedded GPU. It runs two initialization modes: sentence plus box, and sentence only with first-frame grounding. OTB-100 serves as a sanity check that the hybrid bank with uniform gates reproduces FDT-level precision; FDT reports a distance precision of 84.6%.

The decisive checks are three. Sentence gates must beat uniform gates on TNL2K distractor sequences. The method must sit on or beyond the JointNLT accuracy-latency frontier on common hardware. In an interpretability probe, Color Names gates must predict the sentence's color word well above chance. This probe mirrors, in reverse, the analysis in the alignment-paradox study.

A final stress test swaps each sentence for one describing a different object in the same video. A method that truly uses language should drift to that object. One that ignores language will not.