Mamba3R: Fixed-Capacity Selective State Memory for Long-Sequence 3D Reconstruction

Zhenxiang Xiong, Tianchen Deng, Nailin Wang, Lei Zhao, Guole Shen, Weidong ChearXiv 2026 · arXiv

TL;DR

Mamba3R introduces a compact, trainable "state-memory adapter" for frozen recurrent 3D reconstruction models. By leveraging selective state-space models (Mamba) to manage a fixed-capacity auxiliary memory, it significantly reduces trajectory drift and improves geometric accuracy in long sequences while maintaining a constant, real-time computational footprint on consumer-grade hardware.

What Problem Does This Paper Address?

Streaming 3D reconstruction—where a robot builds a map frame-by-frame as it moves—faces a fundamental tension between accuracy and efficiency.

Current approaches generally fall into two camps:

  1. Causal Attention Models: These provide high accuracy by attending to past frames, but their memory requirements grow linearly with time. Eventually, they exceed the GPU budget of a mobile platform.
  2. Recurrent Models: These use a fixed-size "summary" of the past, keeping computational costs constant. However, they often suffer from "drift" over long sequences, and their learned interactions are typically baked into a frozen backbone that can't easily be adapted to new environments.

The authors ask: Can we improve the geometric quality of these fixed-budget recurrent models without retraining the entire massive backbone?

Methods

The researchers introduce Mamba3R, which augments a frozen recurrent backbone (specifically CUT3R) with a lightweight adapter (only 1.28 million parameters).

Key innovations include:

  • Selective State-Space Mixer: Instead of standard attention, they use a 1D selective scan (borrowed from the Mamba architecture) to jointly process the backbone's native state and a new, fixed-capacity auxiliary memory. This allows the model to "selectively" decide what information from the past is relevant to the current frame.
  • Fixed-Capacity Auxiliary Memory: A dedicated memory buffer ($M_t$) that carries information across frames without growing in size.
  • Controlled Feedback Loop: To avoid destabilizing the pretrained backbone, the adapter's output is decomposed. The model calculates a "memory-induced increment" and attenuates it (using a factor $\alpha=0.5$) before adding it back to the main feature stream.
  • History-Only Decay: A specific decay protocol ensures that the auxiliary memory doesn't become stale over very long traversals.
Fig. 1. Overview of Mamba3R. A frozen encoder and recurrent decoder are augmented by the state–memory adapter: it mixes the auxiliary memory with the state carried into the call, its read residual corrects the current frame’s decoder feature before the heads run, and its memory half supplies the write read by the next frame (Section III; execution order in supplementary Algorithm S1).

Datasets

The model was evaluated on:

  • ScanNet: A standard indoor dataset for 3D reconstruction.
  • KITTI-360: A large-scale outdoor dataset used to test long-sequence trajectory accuracy.
  • Robot-Recorded Traversals: The authors collected their own data using a sensor-equipped wheelchair (LiDAR, RGB-D camera, and an onboard RTX 4060 GPU) to test real-world deployment.

Key Insights and Ideas

  • Mamba for 3D States: While Mamba is usually applied to sequences of pixels or text, this paper shows it is highly effective at mixing "slots" in a recurrent 3D state, even when those slots don't have a clear spatial grid arrangement.
  • The Power of Attenuation: A crucial finding was that injecting memory at "full strength" actually hurt performance. By regulating the feedback, the adapter acts as a corrective signal rather than an overwhelming one.
  • Constant Throughput: Because the memory capacity is fixed, the inference speed (FPS) and memory usage remain virtually identical whether the robot has traveled 10 meters or 2 kilometers.

Key Results

Mamba3R consistently outperformed existing recurrent baselines across multiple metrics:

  • Accuracy: On ScanNet, it improved Chamfer Distance (CD) by 9.6% and Trajectory Error (ATE) by 7.3% compared to the strong TTT3R baseline. On the more challenging KITTI-360 dataset, these gains grew to 25.1% and 11.7% respectively.
  • Long-Sequence Stability: In traversals up to 3,000 frames, Mamba3R maintained the lowest trajectory error, often with less than half the error of the original frozen backbone.
  • Real-Time Performance: On a consumer-grade RTX 4060, the model averaged 11.7 FPS, comfortably covering the 10 Hz rate of standard sensors while staying within a 5.1 GiB memory footprint.
Qualitative comparison showing how Mamba3R maintains a coherent map where other methods fragment.

Limitations and Future Work

While Mamba3R significantly reduces drift, it is still a "front-end" predictor. For perfect global consistency over very long loops, it still benefits from a "back-end" optimizer (like a pose-graph).

The authors note that their current 1D scan order is arbitrary (based on the backbone's token order). Future work could explore spatially-aware scanning patterns. Additionally, they aim to develop back-ends that are better optimized for the specific outputs of monocular streaming predictors to further reduce the latency of the complete system.

Summary updated

Abstract

Streaming 3D reconstruction must maintain geometric quality over long inputs within a fixed GPU budget. A pretrained recurrent model provides bounded state, but freezing its backbone also fixes its learned state-observation interactions. We introduce Mamba3R, a jointly designed state-memory adapter that provides a trainable adaptation path without updating the backbone or task heads. A selective state-space mixer jointly transforms native state features and a fixed-capacity auxiliary memory, explicit read and write paths carry information across frames, and a decomposed feedback path regulates what the added memory feeds back. Only 1.28 million adapter parameters are trained. Under a shared periodic-reset protocol, it lowers trajectory error and improves accumulated surface and depth accuracy over recurrent baselines on public indoor and outdoor datasets and robot-recorded traversals, gaining up to 40 percent on one metric. Matched comparisons support both the choice of the mixing operator and the utility of carrying the auxiliary memory across frames. Allocated memory remains unchanged and throughput moves by at most 1.4 percent from one to two thousand frames. Replaying its recorded traversals offline, the platform's RTX 4060 averages 11.7 frames per second within 5.1 GiB with the same checkpoint, a mean throughput covering the 10 Hz sensor rate.
BibTeX
@misc{xiong2026mamba,
  author = {Zhenxiang Xiong and Tianchen Deng and Nailin Wang and Lei Zhao and Guole Shen and Weidong Che},
  title = {{Mamba3R: Fixed-Capacity Selective State Memory for Long-Sequence 3D Reconstruction}},
  year = {2026},
  eprint = {2611.04221},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.04221},
}
Figure