Trackers that fuse a color camera with a second sensor, such as a thermal, depth, or event camera, face a quiet inefficiency. The second sensor matters only some of the time. Thermal imaging rescues a target at night or in fog but adds little on a clear afternoon. Depth helps separate a target from an occluder but is redundant when the target is isolated. Current parameter-efficient multimodal trackers still process the auxiliary stream identically on every frame, using the same learned injection weights for every scene. We propose a tracker built on a frozen color-only vision transformer that does three things on each frame. It reads the transformer's own intermediate-layer features, which it computes anyway, to judge whether and at which depths the auxiliary stream should be injected. It builds the injected signal as a per-frame mixture of a small bank of learned vectors. It skips the auxiliary pathway entirely when the color stream is sufficient.
The bet is that the two things people usually buy with extra modules come almost for free from reusing the frozen backbone's multi-layer states. The first is instance-adaptive conditioning, meaning prompts that change with each input. The second is multi-scale awareness, meaning information from shallow, high-resolution layers as well as deep, semantic ones. A learned per-frame gate then turns that conditioning into a measurable drop in average compute. It should also make the tracker more robust when the auxiliary sensor is missing or corrupted.
Background and the gap
Visual prompt tuning adapts a frozen pre-trained model by adding a few learnable vectors, called prompts, to its input tokens instead of updating its weights. Visual Prompt Tuning introduced the shallow variant (prompts added only at the first layer) and the deep variant (fresh prompts at every layer), matching or beating full fine-tuning while updating about 0.5% of parameters. Visual Prompt Multi-Modal Tracking (ViPT) carried this idea to trackers that combine color with another sensor. It freezes an RGB tracker built on a vision transformer (OSTrack) and adds small prompter blocks that turn auxiliary-sensor features into prompts injected into every block. It tunes 0.84M parameters and reports a DepthTrack F-score of 59.4%.
ViPT supplies our backbone and training recipe. It also defines the gap: its prompters are static per dataset, run on every frame at every depth, and never ask whether the auxiliary signal is useful here and now.
Two lines of work show that instance-adaptive prompts help, and both also show what they cost. Conditional Prompt Learning for Vision-Language Models (CoCoOp) adds a small network that shifts prompts per image. It raises unseen-class accuracy from 63.22% to 71.69%, but the authors note that per-instance prompts break batching and inflate memory and time. Visual Instance-aware Prompt Tuning (ViaPT) generates instance prompts with a two-layer CNN on raw pixels and reaches 76.36% mean accuracy on VTAB-1k against 69.43% for deep VPT, but it is unimodal and ignores intermediate backbone features. Conditional Prompt Tuning for Multimodal Fusion (MoPE) is the closest dynamic multimodal work. It routes prompts from a pool of experts using a single pooled vector of the guiding modality, reaching 73.85% on SNLI-VE. Its authors acknowledge that a single global descriptor is poorly suited to dense spatial tasks. These three papers motivate our proposal, and each points to a design we avoid: a separate generator network, per-instance prompts that cannot be batched, and global-only conditioning.
Multi-scale fusion work supplies the reason to condition on intermediate layers rather than the final one. CLIPCEIL fuses features from all layers of a vision transformer with a one-layer adapter. It raises DomainBed average accuracy from 70.2% (zero-shot CLIP) to 77.3%, and it argues that intermediate layers carry structure that is more robust to shift. Multi-Prompt with Depth Partitioned Cross-Modal Learning (PMPO) assigns different prompts to different depth ranges but keeps them static and pays one encoder pass per prompt. That cost is a negative lesson, and our design must not multiply forward passes.
The efficiency literature constrains the goal. MultiBench reports that multimodal transformers take over 12× the inference time of unimodal models, and also that plain late fusion is often competitive. ELiTe and Modality-specific Distillation cut cost by distilling offline and discarding modalities at test time. That throws away the auxiliary sensor exactly when it would help. We keep the sensor available at runtime but pay for it only when needed.
Core Claim
In RGB-plus-X tracking with a frozen transformer backbone, we claim that conditioning auxiliary prompts on a running summary of the backbone's own intermediate color features is enough for two things. It matches or exceeds ViPT's static prompters and a ViaPT-style pixel generator in accuracy. It also supports a per-frame gate that skips auxiliary injection on a large fraction of frames with little accuracy loss.
The claim can be falsified in three ways. If a capacity-matched static prompter does as well, the dynamic part is unnecessary. If a gate that opens at random with the same frequency does as well, the conditioning carries no information about when the second sensor matters. If the auxiliary pathway turns out to be a negligible share of total cost, the efficiency argument collapses and only the robustness argument remains. We will measure all three directly.
Research Program
The backbone is ViPT's frozen OSTrack. The auxiliary frame passes once through a light patch embedding, as in ViPT, and is average-pooled into a small pyramid at two or three resolutions. At transformer block $l$ we form a running summary $s_l$ of the color stream. This is a learned weighted average of pooled token features from blocks $1$ through $l$, so shallow high-resolution layers and deeper semantic layers both contribute and no extra pass is needed. From $s_l$ a tiny head predicts two things: mixing weights over a bank of $K$ learned prompt maps, and a gate value. The prompt added to the tokens at block $l$ is
$$p_l = g_l \sum_{k=1}^{K} \alpha_{l,k}(s_l)\, B_{l,k} \odot M(a),$$
where $B_{l,k}$ are the learned basis prompt maps for block $l$, $\alpha_{l,k}$ are softmax mixing weights predicted from the summary $s_l$, $M(a)$ is a spatial attention mask computed from the auxiliary pyramid $a$ (ViPT's smoothed spatial softmax), $\odot$ is element-wise multiplication, and $g_l \in \{0,1\}$ is the gate for block $l$.
This basis-plus-coefficients form comes from UCIP, which modulates super-resolution features with a small bank of prompt basis vectors weighted by coefficients derived from each input. It keeps batching intact, because each frame differs only in $K$ scalars per block rather than in a whole prompt tensor. That answers the batching cost CoCoOp reports.
The gate has a simple structure. Each frame chooses one entry depth $l^\ast$, or "never". Blocks before $l^\ast$ run as the plain color tracker. The auxiliary mask and prompts are computed only from $l^\ast$ on, and when the gate says "never" the auxiliary embedding is skipped too. We train with a straight-through estimator and a compute penalty:
$$\mathcal{L} = \mathcal{L}_{\text{track}} + \lambda\, \mathbb{E}\big[C_{\text{aux}}(l^\ast)\big],$$
where $\mathcal{L}_{\text{track}}$ is OSTrack's standard tracking loss and $C_{\text{aux}}(l^\ast)$ is the measured FLOPs of the auxiliary pathway given entry depth $l^\ast$. Sweeping $\lambda$ traces an accuracy-versus-compute curve rather than a single operating point. To prevent the mixing weights from collapsing onto one basis map, a failure mode noted for mixture-of-experts prompt routing, we reuse MoPE's importance loss.
Before building anything, we will profile ViPT to measure what share of per-frame FLOPs and wall-clock time the auxiliary embedding and prompters consume on a single GPU and on an embedded board. This profile sets the ceiling on any efficiency gain and decides how the paper is framed.
Baselines
The do-nothing reference is the frozen color-only OSTrack run on the same sequences with the auxiliary stream ignored. It marks both the accuracy floor and the compute floor. A gated model that falls to this accuracy has learned nothing about the second sensor.
The strongest existing method is ViPT, run from its official code on DepthTrack, LasHeR, RGBT234, VisEvent and VOT-RGBD2022. The benchmark numbers we can cite from the file are ViPT's 59.4% F-score on DepthTrack and its reported 10.5% success-rate improvement on LasHeR. We found no absolute LasHeR success rate for ViPT on Lacuna, so we will reproduce it ourselves. For efficiency we will measure ViPT's FLOPs and frames per second on the same hardware, since the file reports its parameter count (0.84M tunable) but no speed. Beating ViPT means either exceeding its reproduced accuracy at equal compute, or matching it within seed-to-seed variation at no more than 60% of its auxiliary-pathway compute. Both models get the same training data, schedule, epochs and hyperparameter search budget.
The cheapest multimodal reference is late-fusion max pooling as in DepthTrack's DeT, which reports an F-score of 0.532 at 36.88 FPS. On LasHeR, the LasHeR benchmark paper reports mfDiMP at 0.399 success and 38.2 FPS. These anchor the low-compute end of the curve.
Three ablations isolate our ingredients. The first is a capacity-matched static prompter: ViPT with its prompter widened so its tunable parameters equal ours. The second is the always-on dynamic model, with the gate fixed open, which isolates the value of skipping. The third is a random gate that opens with the same per-frame frequency and the same distribution of entry depths as the learned gate, which tests whether the gate knows when the second sensor matters.
Two further variants replace our conditioning signal while keeping everything else fixed. One conditions on a single final-layer pooled vector, as MoPE does. The other uses a ViaPT-style two-layer CNN applied to raw color and auxiliary pixels. These variants test whether reading the backbone's multi-layer states is better, cheaper, or both.
Why not just…?
Why not just use ViPT with a bigger prompter? Extra static capacity could explain any gain we find. ViaPT's comparison with deep VPT (76.36% versus 69.43% on VTAB-1k) suggests that instance conditioning matters beyond parameter count, but that result is unimodal classification, not tracking. The capacity-matched static baseline settles the question directly. If it ties our always-on dynamic model, we drop the claim about dynamic conditioning and keep only the gating contribution.
Why not just use late fusion? MultiBench is a real threat here, because it finds late fusion surprisingly strong and cheap. For trackers, though, late fusion needs a second feature extractor or accepts coarse interaction, as DeT's 0.532 F-score against ViPT's 59.4% on DepthTrack suggests. We place late fusion on the same accuracy-versus-FPS plot. If it dominates our curve anywhere, we will report that.
Why not just drop the auxiliary sensor at test time, or distill it away? ELiTe and Modality-specific Distillation show this can recover much of the accuracy of a multimodal model. It cannot help on frames where only the second sensor sees the target, such as thermal at night. We break down results by LasHeR's environmental attributes and DepthTrack's occlusion sequences. A distilled color-only student trained on the same data should lag on precisely those attributes. If it does not, the runtime-sensor argument is weak.
Why not just use a learned or random modality dropout without conditioning? The random-gate ablation answers this. If a random gate matches the learned one at the same open rate, then the backbone's intermediate features carry no usable signal about when the second sensor helps, and the core claim fails.
Proposed Experiments
The main result is an accuracy-versus-compute curve on DepthTrack and LasHeR, produced by sweeping the penalty weight $\lambda$. It reports F-score, success rate and precision against measured FLOPs and frames per second on a desktop GPU and an embedded board. RGBT234, VisEvent and VOT-RGBD2022 serve as held-out confirmation.
The first decisive check is whether the learned gate's open rate tracks conditions where the second sensor should matter. We expect it to be higher on LasHeR's low-illumination sequences than on normal-light ones, and higher on DepthTrack frames with occlusion than on unoccluded frames. A flat open rate across these conditions would undercut the claim even if average accuracy holds.
The second check is robustness to a missing or corrupted sensor. Direction 3's open problem is that static fusion topologies cannot adapt to a missing or noisy auxiliary stream. We will zero out or add noise to the auxiliary input on a fraction of test frames and compare how much each method degrades. The gated model should fall toward the color-only floor rather than below it, while ViPT, which always injects, may be pulled below the floor by bad prompts.
The third check is the profiling result. If ViPT's auxiliary pathway is under roughly 10% of per-frame cost, we will present the work primarily as adaptive, robust fusion rather than compute reduction. We will also test whether letting the gate pause the auxiliary sensor itself, a real saving on power-limited hardware, changes that picture.
A secondary transfer test moves the conditioning mechanism, without the gate, onto MoPE's SNLI-VE setup, where MoPE reports 73.85%. This asks whether multi-layer conditioning helps beyond tracking.
References
- Visual Prompt Tuning
- Visual Prompt Multi-Modal Tracking
- Conditional Prompt Learning for Vision-Language Models
- Visual Instance-aware Prompt Tuning
- Conditional Prompt Tuning for Multimodal Fusion
- CLIPCEIL: Domain Generalization through CLIP via Channel rEfinement and Image-text aLignment
- Multi-Prompt with Depth Partitioned Cross-Modal Learning
- MultiBench: Multiscale Benchmarks for Multimodal Representation Learning
- ELiTe: Efficient Image-to-LiDAR Knowledge Transfer for Semantic Segmentation
- Modality-specific Distillation
- UCIP: A Universal Framework for Compressed Image Super-Resolution Using Dynamic Prompt
- DepthTrack : Unveiling the Power of RGBD Tracking
- LasHeR: A Large-Scale High-Diversity Benchmark for RGBT Tracking