When a camera or LiDAR samples a street at 2 frames per second, a car can move farther than its own length between snapshots. Trackers that match detections by box overlap then lose all signal, because the boxes no longer intersect. The 3D reconstruction systems that build drivable "digital twins" of such scenes need clean object tracks as input, so they inherit every identity swap the tracker makes. We propose to reverse that dependency. Each tracked object keeps a small 3D Gaussian model of itself, a set of colored, semi-transparent ellipsoids that can be rendered from any viewpoint. A new detection is matched to a track by asking how well that track's model, re-posed at the detection and rendered, explains the new image and LiDAR points. This test does not care how far the object moved. It also accounts for the fact that, after a long gap, the object is usually seen from a different angle.
The opportunity is that the two literatures fail in complementary ways. Tracking at low frame rates breaks because geometric overlap vanishes and 2D appearance features are ambiguous. Dynamic reconstruction at low frame rates breaks because it trusts upstream tracks and because its optimizer has nothing to anchor large motions to. An object model that is used for association and refined by the associations it accepts could fix both. The test case is nuScenes, which already runs at 2 FPS.
Core Claim
Our claim is that a reconstruction-based association cost beats box-overlap and 2D appearance-embedding costs, and that the margin grows as the frame rate drops. We define the cost as the photometric and depth residual left after briefly refining the pose of a track's object-centric Gaussian model against a candidate detection. The cost is
$$c(k,j) = \min_{\xi \in \mathcal{N}(\xi_j)} \; \| I_t - R(G_k, \xi) \|_{M_j} + \lambda \, d_{\text{LiDAR}}(G_k, \xi, P_j),$$
where $G_k$ is the Gaussian model of track $k$ in its own object frame and $\xi_j$ is the 6-degree-of-freedom pose implied by detection $j$. The set $\mathcal{N}(\xi_j)$ is a small neighborhood searched by gradient steps. $R$ renders the model into the current image $I_t$, $M_j$ restricts the image comparison to the detection's region, $d_{\text{LiDAR}}$ is a one-sided Chamfer distance to the LiDAR points $P_j$ inside the box, and $\lambda$ balances the two terms.
Three properties make this cost useful. It is invariant to displacement magnitude. It is viewpoint-aware, because the model can be rendered from the new angle. It is shape-aware, so a white sedan and a white SUV score differently even when 2D embeddings confuse them. A second claim follows: if reconstruction is driven by these self-produced tracks, with retroactive correction when an association is revised, it should approach the rendering quality that reconstruction gets from ground-truth boxes.
Relation to Prior Work
Several papers motivate the gap. EagerMOT shows that 3D overlap fails on nuScenes at 2 FPS. It replaces overlap with an orientation-weighted Euclidean distance plus 2D image-plane matching, but it never builds object shape. Towards Frame Rate Agnostic Multi-Object Tracking shifts weight from motion cues to appearance embeddings as frame rate falls. We keep that insight and change what "appearance" means: a renderable 3D model instead of a 2D feature vector. DSC-Track matches using stable relative geometry between neighboring objects, a cue that is complementary to ours and that we can combine with ours.
On the reconstruction side, DrivingGaussian reconstructs nuScenes with static background Gaussians plus per-vehicle Gaussians in local frames. It assumes accurate box trajectories are given, and that is exactly the dependency we remove. Split4D avoids external trackers by carrying identities along Gaussian velocity trajectories. Its streaming design assumes smooth frame-to-frame motion, which a 2 FPS gap breaks. Sparse4DGS handles sparse frames with texture-anchored regularization of a single scene-wide deformation field. It has no object instances, so it cannot answer "which car is which."
Other papers supply the method. GSGTrack refines a single object's pose by differentiable silhouette rendering against its Gaussian model. It shows that tracking and geometry refined together beat decoupled pipelines on HO3D (64.60% ADD-S versus 4.64% for DROID-SLAM). We reuse that pose refinement, but as a scoring function over many track–candidate pairs rather than for one known object. M5D-GS inflates Gaussian scales early in training so that large displacements still produce gradients. We use the same trick inside the pose search $\mathcal{N}(\xi_j)$, where a 2 FPS jump would otherwise leave the rendered model and the observation with no overlap. OC-SORT retroactively re-updates track state once a lost object is re-observed. We extend that idea from a Kalman state to the object's Gaussian model and pose history.
Other work constrains the design. RobustSplat shows that moving content makes Gaussian densification spawn floating clutter. Buffer inflation of the kind C-BIoU uses in matching is known to raise false associations in dense scenes. Both shape what we do. Inflation is applied only inside a bounded object frame, and only during the scoring search, never to the shared scene. Densification of an object model is allowed only after an association is accepted with high confidence.
Research Program
The system runs online. Each track carries three things: an object-frame Gaussian model, initialized from accumulated LiDAR points as DrivingGaussian does; a pose history; and a confidence value. At each new frame, a cheap gate keeps the top few candidates per track. The gate uses EagerMOT's orientation-aware distance with a wide radius scaled by the measured time gap. The gate exists only for compute: it bounds the number of render-and-compare evaluations. Each surviving pair gets a few dozen pose-refinement steps with inflated-then-annealed scales, which yields $c(k,j)$. Hungarian matching on these costs assigns detections to tracks. Unmatched detections spawn new tracks, and unmatched tracks coast.
When a coasting track is re-matched, an OC-SORT-style pass happens. It interpolates the missing poses along the object's trajectory and re-fits the object model with the new view added. Densification is gated on confidence. The background follows DrivingGaussian's incremental static Gaussians, with RobustSplat-style delayed growth so that unassociated moving pixels do not scar it.
The analysis has three parts. The first measures how the association advantage depends on frame rate. The second stratifies errors by viewpoint change between matched observations, to test whether rendering from the new angle is the active ingredient. The third measures the cost per frame, to check whether the method stays online. The full-batch Non-Markovian solver does not.
Baselines
The naive reference point is EagerMOT run on the same detections at each frame rate. It is the strongest well-documented low-frame-rate 3D tracker in the file: it reports 0.68 AMOTA on nuScenes, at the native 2 FPS. The strongest existing tracker on nuScenes in the file is DSC-Track, at 70.3% AMOTA (DSC-Track). We will run both from released code on identical detector outputs, with no extra hyperparameter tuning beyond what we give our own method. On the nuScenes validation split and on time-subsampled KITTI, beating them means higher AMOTA and fewer identity switches at 2 FPS and below. We expect parity at 10 Hz, and anything else there would suggest a confound.
For reconstruction, the reference is DrivingGaussian fed ground-truth boxes; it reports 28.74 PSNR on nuScenes. We take that as an upper bound. DrivingGaussian fed EagerMOT or DSC-Track tracklets is the realistic competitor. Beating it means closing most of the gap to the ground-truth-box result.
The key ablation keeps our whole pipeline, including gating, Hungarian matching, and retroactive correction, and swaps $c(k,j)$ for three alternatives. The first is a learned 2D re-identification embedding, following FAPS's appearance-weighted strategy. The second is rendering without pose refinement. The third is refinement without the inflation warmup. This isolates whether re-rendering, rather than the surrounding machinery, carries the gain. For per-frame time-subsampled KITTI there is no reported result found on Lacuna for any of these trackers, so all such numbers will be produced by us under one protocol, in the style of FAPS's frame-rate subsampling.
Why not just…?
Why not just run a strong low-frame-rate tracker and feed its tracks to DrivingGaussian? This decoupled pipeline is the main competitor, and we run it directly. The evidence against it is indirect. Split4D reports that a tracker-dependent baseline (SA4D) reaches 0.592 mIoU against 0.893 without that dependence on Multi-Human. GSGTrack's HO3D results point the same way. Neither result is on driving data at 2 FPS, so the decoupled-pipeline baseline above is what settles it.
Why not just use a better 2D re-identification embedding, as FAPS does at low frame rates? This is a real threat. Vehicles are fairly distinctive, and a strong embedding may already resolve most ambiguities. The ablation answers it by swapping only the cost function. We predict our advantage concentrates where the viewpoint changes by more than roughly 30 degrees between matched observations, and where same-colored vehicles sit close together. If the stratified results show no such concentration, the rendering cost is not worth its compute. In that case we would report this and fall back to embeddings for association, keeping the object models only for reconstruction.
Why not just interpolate the video to 30 FPS and use dense-frame methods? Interpolation must invent content across gaps where objects share no overlap and move non-linearly. RobustSplat shows that photometric inconsistencies of this kind trigger floating-Gaussian densification. We will include one interpolation-then-DrivingGaussian run on nuScenes as a sanity check, with no extra claims attached.
Why not just solve association globally, offline? The Non-Markovian approach reports a 37% reduction in identity switches on MOT16, but it needs the full sequence and heavy solvers. Our cost could serve as edge weights in such a solver, so the two approaches are complementary rather than competing. Our contribution is the cost, and we test it online.
Proposed Experiments
The primary benchmark is nuScenes at its native 2 FPS. Tracking is reported as AMOTA and identity switches; reconstruction as PSNR on held-out camera views, with DrivingGaussian's 28.74 as the ground-truth-box ceiling. The frame-rate sweep uses KITTI, subsampled from its native rate to 5, 2, and 1 Hz and reported in HOTA and sAMOTA. EagerMOT's full-rate 94.94 sAMOTA for cars serves as the high-frame-rate anchor where all methods should converge.
The first decisive check is the shape of the curve. Our method's margin over EagerMOT and DSC-Track should be near zero at full rate and should widen monotonically as frames are removed. A flat margin would mean the gain comes from something other than displacement invariance. The second check is the viewpoint-stratified identity-switch rate from the ablation. The third is reconstruction: PSNR with self-produced tracks should land closer to the ground-truth-box result than DrivingGaussian fed baseline tracklets does. We also report the number of floating Gaussians outside object boxes, to confirm that confining inflation to object frames avoids RobustSplat's failure mode. Wall-clock time per frame is reported throughout, so that the online claim can be checked against the gate size.
References
- EagerMOT: 3D Multi-Object Tracking via Sensor Fusion
- Towards Frame Rate Agnostic Multi-object Tracking
- Delving into Dynamic Scene Cue-Consistency for Robust 3D Multi-Object Tracking
- DrivingGaussian: Composite Gaussian Splatting for Surrounding Dynamic Autonomous Driving Scenes
- Split4D: Decomposed 4D Scene Reconstruction Without Video Segmentation
- Sparse4DGS: 4D Gaussian Splatting for Sparse-Frame Dynamic Scene Reconstruction
- GSGTrack: Gaussian Splatting-Guided Object Pose Tracking from RGB Videos
- Motion Decoupled 3D Gaussian Splatting for Dynamic Object Representation
- Observation-Centric SORT: Rethinking SORT for Robust Multi-Object Tracking
- RobustSplat: Decoupling Densification and Dynamics for Transient-Free 3DGS
- Hard to Track Objects with Irregular Motions and Similar Appearances? Make It Easier by Buffering the Matching Space
- Non-Markovian Globally Consistent Multi-object Tracking