KV-Tracker: Real-Time Pose Tracking with Transformers

Marwan Taher, Ignacio Alzugaray, Kirill Mazur, Xin Kong, Andrew J. DavisonarXiv 2025 · DOI · arXiv

Introduction

Recent foundation models for 3D vision, such as DUSt3R and $\pi^3$, have significantly advanced scene reconstruction from images. However, these models are computationally intensive due to their reliance on global self-attention, where the computational cost grows quadratically ($O(N^2)$) with the number of input frames. This complexity makes them unsuitable for real-time applications like robotics or augmented reality, where processing a continuous stream of frames is required. KV-Tracker addresses this bottleneck by introducing a caching mechanism that enables real-time 6-DoF tracking and online reconstruction at 27 FPS.

Methodology

KV-Tracker decouples the reconstruction process into two interleaved loops: Mapping and Tracking. This separation allows the system to maintain global consistency while providing the low-latency updates necessary for live tracking.

The KV-Tracker architecture interleaves a Mapping process for global consistency and a Tracking process for real-time performance.

Mapping and Tracking Loops

  1. Mapping: The system identifies "keyframes" based on camera movement thresholds. A full multi-view transformer processes these keyframes to establish a consistent 3D map. During this phase, the system caches the Key-Value (KV) pairs from the transformer's global attention layers. These cached pairs serve as a learned, implicit representation of the scene.
  2. Tracking: For each new "live" frame, the system avoids re-processing the entire history. Instead, it computes self-attention for the current frame and then utilizes cross-attention against the cached KV pairs of the keyframes. This approach ensures the system remains "anchored" to the established map, preventing the drift and "memory poisoning" often seen in recurrent models.

The KV-Cache Mechanism

The primary technical contribution of KV-Tracker is the transition from all-to-all self-attention to query-to-cache cross-attention.

Comparison of global self-attention (left) vs. the proposed attention with KV-cache (right), illustrating the reduction in computational complexity.

In a standard transformer, every pixel patch in every image attends to every other patch. By caching the KV pairs of keyframes, the tracking phase only needs to compute Queries for the new frame. This reduces complexity from quadratic to linear relative to the number of keyframes, resulting in a 15× speedup. Furthermore, the authors implemented decoder pruning during tracking—disabling the point map and confidence heads—to further optimize performance without sacrificing pose accuracy.

Experiments and Results

The authors evaluated KV-Tracker on several benchmarks, including TUM RGB-D and 7-Scenes for camera tracking, and ARCTIC and OnePose++ for object-level tracking.

Performance and Accuracy

  • Speed: KV-Tracker achieves up to 27 FPS on an NVIDIA RTX 4090, a significant improvement over the baseline of recomputing full attention blocks.
  • Trajectory Accuracy: On the 7-Scenes dataset, the system outperformed the TTT3R baseline by 44% in Absolute Trajectory Error (ATE). Notably, on the challenging "fire" sequence, it reduced error from 0.124m to 0.042m.
  • Object Tracking: In egocentric manipulation tasks (ARCTIC), it achieved an ATE of 0.228m, significantly better than streaming models like CUT3R (0.305m). It also demonstrated robustness on low-texture objects, such as a white fan, where traditional methods often fail.

Discussion and Conclusion

KV-Tracker demonstrates that high-dimensional features stored in a transformer's memory can serve as an effective, implicit scene representation. This model-agnostic strategy allows for zero-shot object tracking without requiring pre-built CAD models, provided the object can be segmented (e.g., using SAM 2).

However, the system faces limitations regarding memory, as the KV-cache grows linearly with the number of keyframes. Currently, it is best suited for spatially confined environments. Future work will focus on cache pruning and incremental updates to scale the system for larger environments and handle dynamic scenes with multiple moving objects.

Summary updated

BibTeX
@misc{taher2025kv,
  author = {Marwan Taher and Ignacio Alzugaray and Kirill Mazur and Xin Kong and Andrew J. Davison},
  title = {{KV-Tracker: Real-Time Pose Tracking with Transformers}},
  year = {2025},
  eprint = {2512.22581},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2512.22581},
}
Figure