3D Point Tracking with State Space Models

Masahiro Ogawa, Qi An, Atsushi YamashitaarXiv 2026 · arXiv

TL;DR

This paper introduces a modular, efficient approach to metric 3D point tracking that operates in absolute meters rather than arbitrary scale. By combining frozen front-ends for optical flow and depth with a tiny (<1M parameter) refiner based on the Mamba-3 state space model, the authors achieve state-of-the-art absolute metric accuracy on a single commodity GPU budget.

What Problem Does This Paper Address?

Tracking physical points in 3D space is fundamental for robotics and autonomous driving, where decisions must be made in meters, not pixels. However, current 3D point tracking faces two major hurdles:

  1. Scale Ambiguity: Standard benchmarks often use scale-invariant metrics, meaning a model can top the leaderboard while being metrically wrong in the real world.
  2. Hardware Demands: The strongest existing trackers rely on massive temporal windows, precise camera poses, or heavy video-diffusion backbones that require high-end server GPUs, making them inaccessible to most practitioners.

The authors aim for a "pose-free, monocular budget" tracker that is metrically accurate and reproducible on a standard laptop GPU.

Methods

The authors propose a deliberately modular and interpretable architecture, as shown in the pipeline below:

Figure 2: Overall pipeline. The flow front-end (SEA-RAFT or WAFT) and the metric-depth backbone (DA3-l or DA3-g, §4) are both frozen and swappable: swapping either only changes which frozen network fills these two grey boxes, and re-training uses an identical architecture, only the frozen depth cache changes. Flow chaining composes the front-end’s consecutive-frame flow into a full-length 2D track and, by a forward–backward consistency check, sets each point’s visibility v—the “+vis” of its output box (§3.4). Grey = frozen / training-free; the blue box is the only learned module (under 1M params); its internals are the Mamba-3 refiner (Fig. 5(a)) and the vmamba3 refiner (Fig. 6(a)). The Mamba-3 refiner leaves the frozen flow front-end’s 2D track ut (and visibility) untouched; the vmamba3 refiner additionally applies a small bounded 2D correction to it.

Key components include:

  • Frozen Front-ends: They use off-the-shelf models for 2D correspondence (SEA-RAFT or WAFT) and monocular metric depth (Depth-Anything-3).
  • Mamba-3 Refiner: Instead of using memory-hungry Transformers, they use a State Space Model (SSM). Mamba-3 summarizes a track's history in a fixed-size recurrent state, meaning memory cost stays constant regardless of video length—a huge advantage over the linear memory growth of Transformer KV caches.
  • VSSD-2pool: The authors derive a novel non-causal operator for Mamba-3. By using two independently-read "pools" (vectors) to collapse the temporal mask, they recover the "$\beta$-band" capacity (temporal context) that standard VSSD methods lose.
  • Conditioning: The refiner is conditioned on DINOv3 appearance features and local depth patches to ensure 2D corrections don't accidentally "slide" points across depth discontinuities.

Datasets

The model is evaluated on the TAPVid-3D benchmark, specifically the minival split. This includes three diverse subsets:

  • DriveTrack: Outdoor driving scenes (10–50m range).
  • Panoptic Studio: Indoor multi-person motion capture (1–5m range).
  • Aria Digital Twin: Egocentric indoor motion (~1m range).

Training was performed on the larger full_eval split of TAPVid-3D.

Key Insights and Ideas

  • Depth is the Bottleneck: The authors observe that once a point's 2D trajectory is fixed, 3D accuracy is entirely governed by the depth along its pixel ray. This simplifies the learning task to a 1D refinement problem.
  • Efficiency via SSMs: At a sequence length of 1025 tokens, the Mamba-based operator uses only 30–54% of the peak memory required by standard softmax attention.
  • Bias vs. Flicker: A fascinating finding is that the larger DA3-g depth model, while more accurate per-frame, suffers from unlearnable "scale flicker" between frames. The smaller DA3-l model has a systematic "far-field bias" which, crucially, is learnable. Consequently, the refined smaller model often outperforms the refined larger one.
  • Learned Visibility: Replacing standard forward-backward flow checks with a learned visibility head allows the tracker to "re-find" points after they emerge from behind occlusions, which accounts for 52–66% of ground-truth points in the dataset.

Key Results

  • SOTA Absolute Accuracy: The best configuration (WAFT flow + DA3-l depth + vmamba3-2pool refiner) achieved a mean metric Average Jaccard (AJ) of 0.256. This is the highest reported for any method under an identical single-GPU budget.
  • Compactness: The learned refiner module contains fewer than 1 million parameters, yet it significantly improves the baseline metric-AJ from 0.147 to 0.256.
  • Speed: The entire pipeline runs at 6–12 FPS on an RTX 4080 Laptop GPU, making it 1.6–2x faster than competitors like SpatialTrackerV2.

Limitations and Future Work

The authors note that their per-track refiner is "blind" to global frame-wide errors. Because it only looks at one track at a time, it cannot correct for per-frame scale drift (flicker) that affects all points simultaneously.

Future work includes:

  • Developing a depth backbone that combines the temporal stability of smaller models with the far-field correctness of larger ones.
  • Extending the non-causal cross-attention to full multi-view 3D tracking, allowing the model to observe shared per-frame quantities that a per-track refiner currently misses.

Summary updated

Abstract

Tracking any point of a dynamic scene in metric 3D - in absolute meters, not up to an unknown scale - underpins 3D and 4D reconstruction, robot navigation, and autonomous driving, where decisions are made in meters, not pixels. Our objective is a 3D point tracker accurate in those absolute terms and operating within a single commodity GPU, pose-free, monocular budget. Our method rests on one observation: once a point's 2D image trajectory is fixed, the quantity that governs its metric accuracy is the depth along its pixel ray. Rather than learning tracking end-to-end, we therefore compose two frozen front-ends - dense optical flow for 2D correspondence and a monocular metric-depth network for the third dimension - and learn only the residual they cannot supply: that depth, refined by a compact state space model (Mamba-3) conditioned on appearance features (DINOv3). A state space model rather than the transformers the strongest 3D trackers adopt is what makes a single-GPU budget attainable: it summarises a track in a fixed-size recurrent state whose memory cost is constant in the number of frames, whereas attention requires a key-value cache that grows linearly with them. On the TAPVid-3D minival benchmark our best configuration attains the highest absolute metric accuracy among methods evaluated under identical conditions (mean metric Average Jaccard, 0.256), exceeding strong feed-forward trackers, while a companion analysis, reproduced with each competitor's own evaluator, explains why several published trackers lose most of their accuracy under this budget.
BibTeX
@misc{ogawa2026d,
  author = {Masahiro Ogawa and Qi An and Atsushi Yamashita},
  title = {{3D Point Tracking with State Space Models}},
  year = {2026},
  eprint = {2609.34035},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2609.34035},
}
Figure