Anchoring World-Grounded Trackers to a Reconstructed Background

Multi-object trackers struggle when people look alike and move erratically: dancers in matching costumes, players in the same kit. Appearance features stop telling targets apart, and simple motion predictors make things worse. A recent answer is to track in 3D. A monocular depth network estimates each target's distance from the camera, so two people who overlap in the image can still be separated by depth. That depth, however, is estimated one frame at a time. Its scale drifts and flickers, and it is expressed relative to a camera that pans, zooms and shakes. The "3D position" of a target standing still can therefore jump from frame to frame. Our proposal is to give such a tracker a stable world to live in. It keeps an online 3D reconstruction of only the static background. Each frame's depth map is recalibrated against that reconstruction, and camera motion is removed using it. The tracker's own target masks in turn tell the reconstruction which pixels to ignore.

The claim is deliberately narrow. We argue that for identity association, consistent geometry of the static background matters more than reconstructing the moving targets. We therefore avoid full 4D reconstruction of dancers and athletes, which is expensive, offline, and built on motion priors that tracking benchmarks show to be wrong. In our design, dynamic-scene reconstruction supplies the machinery for cleanly separating static from moving content. Cross-modal fusion, meaning the merging of 2D image features with 3D geometry, happens at the level of tracking queries. Tracking accuracy is the test.

Core Claim

Beyond the Image Plane (PLANET) lifts 2D tracking queries into 3D with Depth Anything 3 point maps and reports 72.1 HOTA on DanceTrack. By its own account, it treats depth as uncoupled per-frame features and does not integrate camera pose into online association. We claim that much of the error that remains in this kind of tracker comes from two sources: per-frame depth scale inconsistency and uncompensated camera motion. Both can be removed without modelling the targets at all. The tool is a transient-free background reconstruction, updated online and fed with the tracker's masks.

The key step is a per-frame affine recalibration of monocular depth. Let $D_t$ be the raw monocular depth at frame $t$ and $R_t$ the depth rendered from the background model at the current camera pose. Let $B_t$ be the set of pixels outside all tracked target masks. We fit a scale $s_t$ and shift $b_t$ with a robust loss $\rho$:

$$ (s_t, b_t) = \arg\min_{s,b} \sum_{p \in B_t} \rho\big(s\,D_t(p) + b - R_t(p)\big). $$

The recalibrated depth $s_t D_t + b_t$ is then used to lift each target's 2D anchor into the world frame defined by the background model.

Each lifted anchor also carries a variance $\sigma_i^2$. It is estimated from the alignment residuals in a ring of background pixels around target $i$. The reasoning is that where calibration is poor near a target, that target's 3D position should count less in association. Association attention scales geometric distances by $1/\sigma_i$. This borrows the idea behind PCF-Lift, which models lifted semantic features as Gaussians so that uncertain observations are down-weighted. PCF-Lift applies this to static panoptic lifting at high cost. We apply it cheaply, to one position per target, in moving scenes.

Relation to Prior Work

Three groups of papers bear on this, each in a different role.

PLANET motivates the work and supplies the query architecture. It shows that lifting helps under crossovers, and it names monocular depth flicker and ignored camera dynamics as open problems. We keep its Anchored 3D Localization and Dual-Resolution Temporal Memory and change only the geometry it consumes. GSGTrack is further motivating evidence from single-object 6DoF tracking. There, feeding raw monocular depth into an open loop collapses DROID-SLAM to 4.64% ADD-S on HO3D, while maintaining explicit Gaussian geometry recovers 64.60%. Our setting is multi-object and non-rigid, and we maintain geometry only for the background.

The second group supplies the method for the background. RobustSplat shows that 3D Gaussian Splatting spawns Gaussians to fit moving objects, which then scar the static map. It fixes this by delaying densification and bootstrapping transient masks from image features. NeRF On-the-go instead predicts per-pixel uncertainty from DINOv2 features and discards distractors. Both discard moving content and must infer from photometry alone what is moving. We reverse that relationship. The tracker already knows where the people are, so its masks replace mask bootstrapping, and the reconstruction in return provides calibration. This matters for tracking in a specific way. Densification scars form exactly around the targets, which is where rendered background depth is needed to calibrate them.

The third group constrains the design. Split4D, DynaGSLAM and 4D Gaussian Splatting SLAM do reconstruct moving objects. They rely respectively on linear per-Gaussian velocity, cubic Hermite splines, and an MLP deformation field, and Split4D is offline. DanceTrack and SportsMOT show that constant-velocity and smooth motion priors hurt on exactly these targets. That is why we refuse to model target motion geometrically and leave it to the learned tracker. 4D Gaussian Splatting SLAM does contribute one design principle we keep: estimate camera pose only against static primitives.

Research Program

The system has three parts running online.

The first is a lightweight background model. It is a 3D Gaussian map built from keyframes, with densification and optimization restricted to pixels outside the tracker's current masks, dilated by a margin. Camera pose, and focal length for broadcast zoom, are estimated against this map each frame.

The second is the calibration layer. It solves the robust affine fit above, lifts each detection's anchor into the world frame, and attaches a variance to it.

The third is the PLANET-style tracker. It receives these world-frame anchors and variances in place of raw per-frame point maps.

The loop closes in the obvious way: at each frame, the tracker's masks from frame $t-1$ gate background updates at frame $t$. To limit the risk of a feedback spiral, background updates are only accepted where two conditions hold. The pixel must have been outside all masks for several consecutive frames, and its rendered depth must agree with the recalibrated monocular depth.

We will analyse where the gains come from along three axes. The first is camera motion, measured as background optical flow magnitude. The second is crossover events, meaning overlapping boxes of different identities. The third is out-of-plane motion such as jumps, lifts and falls. The central hypothesis predicts that gains concentrate where camera motion and crossovers coincide.

Baselines

The naive reference point is PLANET's recipe unchanged: raw Depth Anything 3 per-frame point maps lifted into the camera frame of each image, with no calibration or ego-motion removal. Any gain we report must be relative to this, trained on identical data, schedule and backbone.

The reported figures for this recipe come from PLANET: 72.1 HOTA and 78.0 IDF1 on DanceTrack, 73.7 HOTA on SportsMOT, and 72.6 HOTA on BFT. The file does not say whether PLANET's code is released. If it is not, we will reimplement it and first reproduce its DanceTrack number within about one HOTA point before comparing anything. On SportsMOT the strongest reported number in the file is not PLANET but MixSort-OC, at 74.1 HOTA on the test set (SportsMOT). This 2D tracker with strong motion handling is the bar to clear there. On VisDrone the reference is MM-Tracker, with 44.7% MOTA and 58.3% IDF1 at 31.1 FPS. It models platform motion implicitly with a state space model.

Beating the baselines means improving HOTA and IDF1 over the reproduced PLANET on all three datasets, and exceeding 74.1 HOTA on SportsMOT, while staying within PLANET's reported 12–20 FPS range. We will also compare against an offline variant that replaces the online background with RobustSplat run on the whole sequence using tracker masks. This variant is an upper bound on how good the background geometry can get.

Two ablations isolate the key ingredients. The first removes the depth recalibration but keeps camera ego-motion compensation, and the second does the reverse. A third removes the per-anchor variances. A fourth ablation isolates the closed loop by building the background from RobustSplat-style self-bootstrapped transient masks instead of tracker masks.

All variants share detector, backbone, training data, tuning budget of hyperparameter trials, and hardware.

Why not just…?

Why not just use a temporally consistent video depth model instead of a reconstruction? Smoothing depth over time would reduce flicker. It does not fix a pure scale drift that changes slowly, and it leaves positions relative to a camera that pans. It is a real threat on static-camera DanceTrack sequences, and we will run it as a baseline there, fed into the same tracker. If it matches our full system on DanceTrack but not on SportsMOT and VisDrone, the honest conclusion is that ego-motion compensation is the active ingredient. The paper would then say so.

Why not just run off-the-shelf masked SLAM for camera poses and fit depth to its sparse points? This is the closest competitor, and it may work nearly as well. GSGTrack's HO3D results, where DROID-SLAM reaches only 4.64% ADD-S, suggest that monocular SLAM degrades when moving content dominates the frame. That setting is not ours, however. We therefore include masked DROID-SLAM with sparse-point depth alignment as a baseline. Our reconstruction must justify itself through dense rendered depth near targets, which sparse SLAM points rarely cover. The ring-residual variances depend on that density.

Why not just project feet onto a ground-plane homography, as sports analytics often does? Courts and dance floors are roughly planar, so this is a strong, cheap baseline on SportsMOT and DanceTrack. It fails when feet are occluded in crossovers and when targets leave the ground, which is common in volleyball and dance lifts. We will run it and stratify results by frames with hidden feet and airborne targets. If it ties overall, our claim narrows to those strata and to non-planar backgrounds.

Why not just reconstruct the targets too, as Split4D does? Split4D is offline, and it assumes linear per-Gaussian velocity. The Kalman-filter penalty documented on DanceTrack and SportsMOT predicts that such priors break on these motions. Target reconstruction would also reintroduce the memory growth and floaters seen in dynamic Gaussian SLAM. The offline RobustSplat upper-bound comparison shows how much remains from better background geometry alone.

Proposed Experiments and Decisive Checks

The primary evaluation is HOTA, AssA and IDF1 on DanceTrack and SportsMOT against the baselines above, with MOTA, IDF1 and FPS on VisDrone. Because these tracking datasets lack 3D ground truth, we validate the geometry separately on the Bonn RGB-D Dynamic dataset. It has metric depth and moving people with a handheld camera. There we measure two things. The first is depth error on the human pixels for raw monocular depth versus recalibrated depth. The second is camera trajectory error, with 4D Gaussian Splatting SLAM's reported 3.6 cm average absolute trajectory error as the reference.

Four outcomes would settle the claim.

The first concerns where the gains fall. On SportsMOT, stratified by background flow magnitude, the improvement over reproduced PLANET should grow with camera motion. On static-camera DanceTrack clips, only the recalibration ablation should contribute.

The second is a null prediction. On BFT, where the background is mostly sky and there is little static structure to align to, the method should fall back to near-PLANET performance. It should not improve there, and it should not degrade either, because the variances inflate. A gain on BFT would suggest the improvement comes from something other than the stated mechanism.

The third tests robustness to injected noise. Synthetic per-frame scale jitter added to Depth Anything 3 should sharply degrade the naive baseline. The calibrated system should be largely unaffected, since the affine fit absorbs global scale.

The fourth is the closed-loop ablation. Background built from tracker masks should yield lower background depth error near targets than background built from self-bootstrapped masks. If it does not, the coupling between tracker and reconstruction is not pulling its weight and should be dropped.