When a tracked object disappears behind an obstacle, most trackers commit to one guess about where it is. They either extrapolate a single box forward or stop and wait. Yet the truth is usually split among a few possibilities: the person may have come out the left side of the pillar or the right side, or may still be standing behind it. We propose to keep a small population of box hypotheses alive during occlusion. A learned motion model proposes them. A handful of gradient-guided sampling steps then corrects them, using three signals. The first is motion plausibility. The second is the "you would see it if it were here" evidence from visible pixels. The third is a frozen language description of the target's permanent attributes. When the object reappears, the tracker picks among these hypotheses instead of searching blindly.
The sampling machinery is short-run Langevin dynamics, a Markov chain Monte Carlo (MCMC) method that moves samples downhill on an energy function while adding noise. Short-run chains are only affordable when started near the answer. Here the learned motion model plays the role of an amortized initializer, meaning a network trained to predict good starting points so the chain does not have to search from scratch. The target's location during full occlusion is never labeled, so the only training signal arrives at re-emergence. That signal is delayed and non-differentiable, which makes this a policy-gradient problem with severe variance. We handle it with a self-critical baseline: the reward the naive deterministic extrapolator would have earned on the same clip. The main claim is that at re-emergence, a learned proposal plus a few Langevin steps plus a frozen semantic anchor recovers targets that single-guess extrapolators and threshold-triggered language re-detection lose. It should do so at real-time cost.
What exists and what is missing
The failure is documented on both the geometric and the semantic side. MambaTrack replaces the Kalman filter with a learned state-space motion predictor. It bridges gaps by running that predictor autoregressively on its own outputs, which is deterministic and drifts over long occlusions. We reuse its style of motion model, but as a sampler's initializer rather than as the answer. RELO trains the localization head with reinforcement learning on frame-level IoU (intersection-over-union overlap) and sequence-level AUC (area under the success curve). It shows an emergent widening of its spatial distribution under occlusion. However, it chooses greedily frame by frame, keeps no belief over the hidden state, and uses no variance-reduction baseline. That observation motivates our work: the widening signals that a belief is needed, and the greedy choice discards it.
On the semantic side, AdaSwitcher/TNL2K switches between local tracking and global language grounding with a learned anomaly detector. MSRTrack invokes a vision LLM on confidence thresholds and runs at 10.3 to 17.8 FPS on an A100. Both treat language as a way to re-search after failure, not as a constraint on where the hidden target can be.
Two findings constrain our design. First, CLDTracker updates its text features over recent frames and acknowledges that this absorbs background during out-of-view spans. Second, MSRTrack reports that inherent attributes such as markings help while transient ones such as posture can hurt. We therefore freeze the anchor and restrict it to inherent attributes.
The sampling method comes from two sources. In CoopFlow, a normalizing flow proposes and 30 Langevin steps refine, with the flow retrained on refined samples. Approximate Thompson Sampling with Langevin shows that warm-starting from the previous round's sample needs only a constant number of steps per round. Its guarantee assumes a single-peaked (log-concave) target, which our split posteriors violate. We treat that as an empirical question rather than claiming the bound.
The closest combination is Langevin Actor-Critic. It warm-starts short-run Langevin from a Gaussian generator for control actions. It has never been applied to visual state estimation, and its single-peaked generator is exactly what occlusion breaks. Our proposal network is a mixture over hypotheses, and its energy includes visual negative evidence and language.
Core mechanism
At each occluded frame $t$, the tracker holds $K$ box hypotheses $b^{(k)}$, where each box $b = (x, y, w, h)$. The energy of a box is
$$E_t(b) = -\log p_\theta(b \mid h_t) + \lambda_v V_t(b) - \lambda_s S(b).$$
Here $p_\theta(b \mid h_t)$ is the learned motion model's density given the track history $h_t$. $V_t(b)$ is the visible-evidence penalty: how strongly the host tracker's response map shows no target inside the visible part of $b$. It is high when a box sits in open view yet nothing is there, which pushes hypotheses into occluded regions. $S(b)$ is the similarity between the box's pooled visual features and a frozen text embedding of the target's inherent attributes. The weights $\lambda_v$ and $\lambda_s$ are scalars.
All three terms are read off per-frame dense maps by bilinear interpolation, so their gradient with respect to box coordinates is cheap. No crop is re-encoded per step. Hypotheses are drawn from the motion model, which is warm-started from the previous frame's refined set, and then refined for $L$ steps (a small number, between 3 and 10):
$$b \leftarrow b - \eta \nabla_b E_t(b) + \sqrt{2\eta}\,\epsilon, \qquad \epsilon \sim \mathcal{N}(0, I),$$
where $\eta$ is the step size.
At re-emergence, the host tracker scores detections near each hypothesis. It locks onto the candidate with the best combination of detection score, $S$, and hypothesis support.
The motion model is trained two ways. The first is the CoopFlow recipe: fit $p_\theta$ to its own refined samples, which is cheap and dense. The second is a policy gradient on the re-acquisition reward $R$, the IoU of the locked box at the first visible frame and over a short window afterwards. For a clip $c$, the advantage is
$$A(c) = R(\text{sampled belief}) - R(\text{deterministic patch}),$$
where the deterministic patch is the autoregressive mean rollout of the same $p_\theta$, in the manner of MambaTrack. This is the self-critical baseline of SCST, chosen for a specific reason: the gradient becomes nonzero only on clips where keeping multiple hypotheses changes the outcome relative to the naive method. Most clips are routine occlusions, where the difference is zero and would otherwise contribute pure noise.
Research Program
We first build the belief module as a plug-in for one public, retrainable transformer single-object tracker. The host is frozen except for a lightweight projection that produces the semantic similarity map.
Second, we test whether the warm start is what makes short-run chains viable. We compare amortized initialization against starting from the last visible box plus noise, at equal step counts, and plot re-acquisition against $L$.
Third, we measure policy-gradient variance directly. For the zero baseline, a learned value critic, and the self-critical baseline, we record the variance of minibatch gradient estimates, the frequency of gradient-norm spikes, and seed-to-seed spread in final re-acquisition. AIC motivates the concern about critics: it finds that critic signals destabilize high-capacity visual encoders. That concern is one reason to avoid a critic here.
Fourth, we examine where the semantic term helps. We expect it to help at distractor-rich re-emergences and not during full occlusion. We will check whether a frozen anchor beats a CLDTracker-style updated one specifically on out-of-view sequences.
Baselines
The do-nothing reference is the host tracker alone, run on identical checkpoints and inputs. Every gain is reported relative to it.
The classical reference is the host plus a constant-velocity Kalman filter that re-centers the search window during absence, in the manner of the motion module in WebUOT-1M/OKTrack.
The strongest geometric competitor is the host plus our own motion model rolled out deterministically, in the style of MambaTrack's tracklet patching. It shares architecture, training data, and training epochs with our proposal. Any difference therefore comes from sampling, refinement, and the reward, not from a better motion network.
The strongest sample-based competitor is a particle filter. Its proposal is $p_\theta$, its importance weights are $\exp(-E_t)$, and it has the same $K$. It gets the same number of energy evaluations per frame as our $K \times L$ Langevin evaluations, which isolates whether gradient-guided refinement beats reweighting and resampling.
The strongest semantic competitor has two variants. The first is AdaSwitcher-style threshold-triggered global language re-detection using the same frozen text embedding. The second is RELO itself.
Reported numbers set the bar. RELO-L256 reports 75.1% AUC on LaSOT and 57.5% AUC on LaSOText. CLDTracker reports 74.0% AUC on LaSOT. MSRTrack-L reports 78.7% success on LaSOT, but with GPT-4.1 in the loop. We treat it as a latency-unconstrained reference rather than a like-for-like competitor. On TNL2K, the file reports 0.42 for AdaSwitcher; no reported RELO number was found on Lacuna for TNL2K.
Overall AUC is dominated by visible frames, so it will move little. We count as beating a baseline two things. The primary outcome is a statistically significant gain in re-acquisition success on post-absence windows, with confidence intervals by sequence bootstrap. The secondary requirement is no loss in overall AUC and throughput above 30 FPS on a single consumer GPU.
The ablations remove one piece at a time. They drop the semantic term, drop the visible-evidence term, and replace amortized with naive initialization. They also swap the self-critical advantage for a zero baseline or a learned critic, and train with the CoopFlow-style self-fit alone.
Why not just…?
Why not just use a Kalman filter and re-detect when the target comes back? This is the default, and on short occlusions it will likely match us. MambaTrack reports that a learned motion model beats Kalman-based ByteTrack by nearly 10 HOTA points on SportsMOT (HOTA is a multi-object tracking accuracy score), which suggests linear extrapolation fails on agile motion. That evidence concerns association in multi-object tracking, not single-target re-acquisition. We will therefore stratify results by occlusion length and report where the Kalman baseline is sufficient. If it wins on long occlusions too, the premise fails.
Why not just run a particle filter with the same energy? This is the most serious threat. A particle filter represents split posteriors natively and needs no gradients. Our bet is that the visible-evidence term produces sharp edges along occluder boundaries. Gradient steps can move hypotheses across those edges, whereas reweighting a fixed set only kills hypotheses that landed on the wrong side, leaving few survivors. The equal-evaluation comparison settles this directly. If the particle filter matches us, the right conclusion is that the learned proposal and the energy are the contribution and Langevin refinement is not. We would report it that way.
Why not just have the motion network output a mixture of boxes and train it on the re-emergence box? A mixture density head trained by likelihood on the first visible box is differentiable and needs neither MCMC nor policy gradients. We include it. Its weakness is structural: it cannot use the current frame's visible evidence or the semantic map during occlusion unless those are fed as inputs. The refinement step conditions on them at test time for free. The ablation that removes $V_t$ and $S$ from our energy is the matched comparison. If the mixture head equals our full system, the sampling stage is unnecessary.
Why not just call a vision-language model to re-find the target? MSRTrack shows this is accurate but runs at 10.3 to 17.8 FPS on an A100 even when invoked sparsely. It also still relies on short-term visual memory during the gap. Our aim is a cheap belief that makes such calls rarer or unnecessary. We report re-acquisition per unit of compute rather than claiming to beat its raw success number.
Proposed Experiments
We train on the standard LaSOT, GOT-10k, and TrackingNet training splits. We evaluate on LaSOT, LaSOText (unseen categories with heavy out-of-view), and TNL2K (natural-language descriptions, adversarial and appearance-change cases). The language anchor for each target is its benchmark sentence, filtered to inherent attributes by a fixed rule so that no model call happens at test time.
Primary metrics are re-acquisition success and frames-to-reacquire on windows following frames the ground truth marks as target-absent. Secondary metrics are overall AUC and precision, FPS, and the gradient-variance measurements described above.
As a stress test of the identity question, we evaluate re-emergence on DanceTrack clips where two similarly dressed dancers cross behind one another. This is exactly the case where Occlusion Geodesics says pure geometry fails. We will compare the full system against the no-semantic ablation and against MambaTrack's reported 56.8 HOTA only as context, since our setup targets one identity at a time.
Three checks are decisive. First, sampled beliefs must beat the matched deterministic patch on long-occlusion windows. Second, Langevin refinement must beat the equal-budget particle filter. Third, the self-critical advantage must give measurably lower gradient variance and tighter seed spread than a learned critic without lowering final re-acquisition. If the first check fails, the idea fails. If the second fails, we drop MCMC and keep the rest. If the third fails, we report a negative result on variance reduction for delayed spatial rewards, which RELO's dual-level reward design also leaves unresolved.
References
- MambaTrack: A Simple Baseline for Multiple Object Tracking with State Space Model
- RELO: Reinforcement Learning to Localize for Visual Object Tracking
- Towards More Flexible and Accurate Object Tracking With Natural Language: Algorithms and Benchmark
- MSRTrack: LLM-Powered Object Tracking with Motion and Semantic Reasoning
- CLDTracker: A Comprehensive Language Description for visual Tracking
- A Tale of Two Flows: Cooperative Learning of Langevin Flow and Normalizing Flow Toward Energy-Based Model
- On Approximate Thompson Sampling with Langevin Algorithms
- Langevin Policy for Safe Reinforcement Learning
- Self-Critical Sequence Training for Image Captioning
- Stabilizing Visual Reinforcement Learning via Asymmetric Interactive Cooperation
- WebUOT-1M: Advancing Deep Underwater Object Tracking with A Million-Scale Benchmark
- Occlusion Geodesics for Online Multi-object Tracking