TL;DR
This paper introduces a modular, efficient approach to metric 3D point tracking that operates in absolute meters rather than arbitrary scale. By combining frozen front-ends for optical flow and depth with a tiny (<1M parameter) refiner based on the Mamba-3 state space model, the authors achieve state-of-the-art absolute metric accuracy on a single commodity GPU budget.
What Problem Does This Paper Address?
Tracking physical points in 3D space is fundamental for robotics and autonomous driving, where decisions must be made in meters, not pixels. However, current 3D point tracking faces two major hurdles:
- Scale Ambiguity: Standard benchmarks often use scale-invariant metrics, meaning a model can top the leaderboard while being metrically wrong in the real world.
- Hardware Demands: The strongest existing trackers rely on massive temporal windows, precise camera poses, or heavy video-diffusion backbones that require high-end server GPUs, making them inaccessible to most practitioners.
The authors aim for a "pose-free, monocular budget" tracker that is metrically accurate and reproducible on a standard laptop GPU.
Methods
The authors propose a deliberately modular and interpretable architecture, as shown in the pipeline below:
Key components include:
- Frozen Front-ends: They use off-the-shelf models for 2D correspondence (SEA-RAFT or WAFT) and monocular metric depth (Depth-Anything-3).
- Mamba-3 Refiner: Instead of using memory-hungry Transformers, they use a State Space Model (SSM). Mamba-3 summarizes a track's history in a fixed-size recurrent state, meaning memory cost stays constant regardless of video length—a huge advantage over the linear memory growth of Transformer KV caches.
- VSSD-2pool: The authors derive a novel non-causal operator for Mamba-3. By using two independently-read "pools" (vectors) to collapse the temporal mask, they recover the "$\beta$-band" capacity (temporal context) that standard VSSD methods lose.
- Conditioning: The refiner is conditioned on DINOv3 appearance features and local depth patches to ensure 2D corrections don't accidentally "slide" points across depth discontinuities.
Datasets
The model is evaluated on the TAPVid-3D benchmark, specifically the minival split. This includes three diverse subsets:
- DriveTrack: Outdoor driving scenes (10–50m range).
- Panoptic Studio: Indoor multi-person motion capture (1–5m range).
- Aria Digital Twin: Egocentric indoor motion (~1m range).
Training was performed on the larger full_eval split of TAPVid-3D.
Key Insights and Ideas
- Depth is the Bottleneck: The authors observe that once a point's 2D trajectory is fixed, 3D accuracy is entirely governed by the depth along its pixel ray. This simplifies the learning task to a 1D refinement problem.
- Efficiency via SSMs: At a sequence length of 1025 tokens, the Mamba-based operator uses only 30–54% of the peak memory required by standard softmax attention.
- Bias vs. Flicker: A fascinating finding is that the larger DA3-g depth model, while more accurate per-frame, suffers from unlearnable "scale flicker" between frames. The smaller DA3-l model has a systematic "far-field bias" which, crucially, is learnable. Consequently, the refined smaller model often outperforms the refined larger one.
- Learned Visibility: Replacing standard forward-backward flow checks with a learned visibility head allows the tracker to "re-find" points after they emerge from behind occlusions, which accounts for 52–66% of ground-truth points in the dataset.
Key Results
- SOTA Absolute Accuracy: The best configuration (WAFT flow + DA3-l depth + vmamba3-2pool refiner) achieved a mean metric Average Jaccard (AJ) of 0.256. This is the highest reported for any method under an identical single-GPU budget.
- Compactness: The learned refiner module contains fewer than 1 million parameters, yet it significantly improves the baseline metric-AJ from 0.147 to 0.256.
- Speed: The entire pipeline runs at 6–12 FPS on an RTX 4080 Laptop GPU, making it 1.6–2x faster than competitors like SpatialTrackerV2.
Limitations and Future Work
The authors note that their per-track refiner is "blind" to global frame-wide errors. Because it only looks at one track at a time, it cannot correct for per-frame scale drift (flicker) that affects all points simultaneously.
Future work includes:
- Developing a depth backbone that combines the temporal stability of smaller models with the far-field correctness of larger ones.
- Extending the non-causal cross-attention to full multi-view 3D tracking, allowing the model to observe shared per-frame quantities that a per-track refiner currently misses.
Summary updated