Where Could It Be Hiding? Scene-Wide Reachability Fields for Re-Linking Occluded Tracks

When a multi-object tracker loses a person behind someone else, it guesses where that person will reappear by extrapolating their last motion. That guess is local. It grows an uncertainty blob around the last known position and has no view of the rest of the scene. The tracker ignores the most informative fact it has: the detector is not firing. If the person were standing in plain view, they would be detected. So they must be somewhere occluded, and the occluded regions are defined by the other objects and structures in the frame, which are often far from the lost track's last box. We propose to predict, for every lost track and every frame, a probability map over the image. The map says where the hidden target could have moved and still be invisible. It is computed in a single pass by an attention layer that lets every location in the scene influence every other. The tracker then uses this map to decide which newly appearing detection continues the old track.

Put simply, trajectory fragmentation (one physical path split into several track IDs) is treated as a problem of limited reach. A motion filter only reaches outward from the last position. Answering "where can it be and remain unseen?" requires scene-wide reasoning, the same reasoning that lets a planner route around distant obstacles in one step. We build that reasoning as a small, cheap module that plugs into the strongest existing learned association system.

Core Claim

Occlusion-induced fragmentation is partly a receptive-field failure in the association stage, separate from any failure in the image backbone. Constant-velocity filters and learned temporal encoders over a track's own history both predict re-emergence from information local to that track. The true hidden position is constrained by non-local evidence: which regions are covered by visible occluders, how those occluders are moving, and where the detector would have fired had the target been exposed.

We define a reachability field for lost track $k$ at frame $t$:

$$p_{k,t}(x) \propto \exp\big(g_\theta(x \mid s_k, S_t)\big)\,\big(1 - v_t(x)\big)$$

The terms are as follows. $x$ is a cell of a spatial grid over the image. $s_k$ is the lost track's last state and elapsed gap. $S_t$ is a rasterized description of the current scene: visible detections, their apparent depth order from box bottom height, and frame borders. $g_\theta$ is a learned score. $v_t(x)$ is the probability that a target at $x$ would have been detected, so the second factor encodes the negative evidence. We compute $g_\theta$ with recurrent criss-cross attention, which gives full-grid reach at cost well below dense attention. The field is trained to assign high likelihood to the annotated hidden position.

We expect three results. The field should reduce identity switches most on long occlusion gaps. The gain should vanish if $g_\theta$ is restricted to a local receptive field of matched capacity. The gain should also vanish if the negative-evidence factor is removed.

Relation to Prior Work

The closest ancestor is Occlusion Geodesics for Online Multi-object Tracking. It re-links tracks by searching minimal-cost paths through an occlusion confidence map that penalizes visible areas where the detector should have fired. It reports 98.1% MOTA on PETS'09 S2L1. The negative-evidence idea comes from it. Its cost fields, however, are hand-built, its occluder masks are static and pre-annotated, and it assumes a fixed camera over flat ground. We learn the field from data, let occluders be other moving people, and make no camera assumptions. This is what DanceTrack, SportsMOT, and MOT20 require.

Two lines of work supply the mechanism. Differentiable Spatial Planning using Transformers shows that self-attention propagates value between any two cells of an obstacle map in one step. Convolutional planners need depth proportional to map size. The attention planner reports 7–19% absolute path accuracy gains over convolutional planners and about 95% versus about 60% out-of-distribution accuracy on Gibson maps. It assumes a static map and never models time or occlusion. CCNet: Criss-Cross Attention for Semantic Segmentation shows that two stacked row-and-column attention passes reach the whole image with 11× less memory and 85% fewer FLOPs than a dense non-local block. That lets us run the field at a finer grid than dense attention would allow, where it would scale as the fourth power of grid side length. We reuse both techniques. Neither paper applied them to tracking.

Four tracking papers motivate the gap, each by what it leaves out.

TrackFormer: Multi-Object Tracking with Transformers keeps inactive track queries alive for a patience window. It attends over global image features, but the queries receive no update while the target is hidden. As a result, they drift over long gaps.

More Than Meets the Eye: Enhancing Multi-Object Tracking Even with Prolonged Occlusions (MOTE) does update hidden tracks. It warps features forward along RAFT optical flow with softmax splatting. That propagation is local, costs about 160 ms per frame, and is unreliable for small or distant targets. Our field needs no optical flow and reasons about where a target cannot be.

CAMELTrack: Context-Aware Multi-cue ExpLoitation for Online Multi-Object Tracking learns association with a temporal encoder over each tracklet's cue history and group-aware attention across tracklets and detections. It is trained cheaply on pre-extracted features. Its attention runs over object tokens, not over the spatial layout of the scene. This is the system we extend.

UniTrack: Differentiable Graph Representation Learning for Multi-Object Tracking adds flow-conservation and reachability penalties as a training loss. It reports up to 53% fewer identity switches. Its "unreachable" notion is distance-based and blind to occluders.

A constraint comes from DanceTrack: Multi-Object Tracking in Uniform Appearance and Diverse Motion. There, appearance re-identification actively hurts association when everyone looks alike, and trackers fall from roughly 60–70 HOTA on MOT17 to 40–50 on DanceTrack. The field must therefore be built from geometry and detection evidence, not appearance. We deliberately keep appearance out of it.

Research Program

The field module takes a coarse grid, for example 1/8 of input resolution. Each cell carries three kinds of channels: rasterized visible detections with confidence and a depth-order proxy, a learned frame-border embedding, and an embedding of the lost track's last box, velocity, and gap length. Two criss-cross attention passes followed by a small head produce $g_\theta$.

The visibility term $v_t$ is predicted by a separate lightweight head. It is trained on frames where annotated targets are either detected or marked with low visibility; MOT17 annotations include per-box visibility for occluded pedestrians. Wherever a dataset annotates boxes through occlusion, we supervise the field with the negative log-likelihood of the true hidden box center. Where it does not, we supervise it only at re-emergence frames.

At association time, each lost track contributes a scalar cue for each candidate detection: the field's log-probability at that detection's center. This enters CAMEL as an extra token alongside box, motion, and pose cues. The field can therefore be combined with, or overridden by, the learned association rather than acting as a hard gate. Because the inputs are boxes and detector outputs, the module fits CAMELTrack's image-free training and keeps its low training cost.

Analysis matters as much as accuracy. We will visualize fields on DanceTrack crossings to check whether probability mass concentrates behind the occluding dancer and moves with that dancer. We will also measure how often the argmax of the field lies outside a Kalman gate yet still at the true re-emergence point. Those are exactly the cases a local predictor cannot recover.

Baselines

The naive reference is CAMELTrack's own track handling with the patience window extended, with no new cue. It tells us how much fragmentation is fixed simply by not deleting tracks.

The strongest existing method on the uniform-appearance, non-linear-motion benchmarks is CAMELTrack. It reports 69.3 HOTA on DanceTrack and 80.4 HOTA on SportsMOT at 13 FPS. It is the right competitor because its temporal encoder could, in principle, learn re-emergence from track history alone. We run it and our variant with identical detections, identical pre-extracted features, the same training data, and the same number of hyperparameter trials, reusing released configurations where possible. Beating it means a HOTA gain on DanceTrack test beyond run-to-run variance, measured over three seeds. The gain should be concentrated in association accuracy rather than detection accuracy, and it should come from the long-gap strata.

The occlusion-specialized competitor is MOTE, which reports 93.2 MOTA on DanceTrack and 82.0 MOTA / 66.3 HOTA on MOT17. No DanceTrack HOTA for MOTE was found on Lacuna. We therefore compare it on MOTA and identity switches, and report our speed next to its roughly 6 FPS. Matching MOTE's long-occlusion identity preservation at CAMELTrack-level speed would count as a win even without exceeding its MOTA.

On MOT17 and MOT20 we also report TrackFormer, at 74.1 MOTA on MOT17 private detections and 68.6 MOTA on MOT20, as the end-to-end attention reference.

Three ablations isolate the key ingredient. The first replaces criss-cross attention in $g_\theta$ with a convolutional network of matched parameters and matched local receptive field; this tests the reach claim. The second removes the $(1 - v_t)$ factor; this tests the negative-evidence claim. The third replaces the whole field with a learned, scene-blind anisotropic Gaussian whose spread grows with gap length; this tests whether the scene matters at all.

Why not just…?

Why not just let CAMELTrack's temporal encoder learn re-emergence from longer histories? This is the most serious threat. The encoder already sees each tracklet's box trajectory and, through group-aware attention, other objects' tokens. It might infer "hidden behind that dancer" implicitly. Our concern is that object-token attention lacks the explicit spatial layout needed to represent a region, such as the silhouette of an occluder that is moving, rather than a point. The extended-history CAMELTrack baseline, trained with the same budget, settles it. If it matches our model on the long-gap stratum, the field adds nothing, and we will report that.

Why not just inflate the Kalman gate or keep tracks alive longer? A growing gate is the local predictor this proposal argues against. The H3D baseline in The H3D Dataset for Full-Surround 3D Multi-Object Detection and Tracking in Crowded Urban Scenes shows filtering holding 76.2 MOTA on cars but only 36.8 MOTA on pedestrians. Pedestrians stop and turn while hidden, which is consistent with the concern. A wider gate also admits more wrong candidates in crowds. The scene-blind Gaussian ablation, whose spread growth is learned, is the fair version of this alternative and is run directly.

Why not just use optical flow to push hidden tracks forward, as MOTE does? Flow propagates a feature from where it was. It does not constrain where a fully hidden target can go, and MOTE's authors note that it fails for small and distant targets at about 160 ms per frame. Our module uses no pixels beyond detector outputs. The MOT20 comparison, dominated by small and dense targets, and the speed comparison will show whether dropping flow costs accuracy.

Why not just use hand-built occlusion geodesics? Occlusion Geodesics needs static occluder masks and a static camera. SportsMOT and DanceTrack have neither. Where its assumptions hold, we expect our learned field to recover similar behavior. Visualizing fields on static-camera MOT17 sequences checks this qualitatively.

Proposed Experiments

We evaluate on DanceTrack, SportsMOT, MOT17, and MOT20. We report HOTA with its detection and association components, IDF1, and identity switches. The decisive metric is a gap-stratified re-link rate: for ground-truth tracks that disappear for a given number of frames, the fraction whose pre- and post-occlusion segments receive the same ID. Gaps are binned as short (under 10 frames), medium, and long (over 30 frames). The prediction is specific: improvements over CAMELTrack should grow with gap length and be near zero for short gaps.

The field itself is evaluated separately on annotated hidden positions. We report the negative log-likelihood of the true center, and the rank of the true re-emergence detection among candidates, against the Kalman prediction and the scene-blind Gaussian.

Three results would falsify the proposal. The first is a convolutional $g_\theta$ of matched capacity matching the criss-cross version, which would mean reach is not the bottleneck. The second is removing the negative-evidence factor leaving performance unchanged, which would mean the "unseen" reasoning is not doing work. The third is extended-history CAMELTrack closing the long-gap gap. Any of these would redirect the work. The reach and negative-evidence claims are the intended contribution. A plain HOTA increment without them is not.