TL;DR
Mamba3R introduces a compact, trainable "state-memory adapter" for frozen recurrent 3D reconstruction models. By leveraging selective state-space models (Mamba) to manage a fixed-capacity auxiliary memory, it significantly reduces trajectory drift and improves geometric accuracy in long sequences while maintaining a constant, real-time computational footprint on consumer-grade hardware.
What Problem Does This Paper Address?
Streaming 3D reconstruction—where a robot builds a map frame-by-frame as it moves—faces a fundamental tension between accuracy and efficiency.
Current approaches generally fall into two camps:
- Causal Attention Models: These provide high accuracy by attending to past frames, but their memory requirements grow linearly with time. Eventually, they exceed the GPU budget of a mobile platform.
- Recurrent Models: These use a fixed-size "summary" of the past, keeping computational costs constant. However, they often suffer from "drift" over long sequences, and their learned interactions are typically baked into a frozen backbone that can't easily be adapted to new environments.
The authors ask: Can we improve the geometric quality of these fixed-budget recurrent models without retraining the entire massive backbone?
Methods
The researchers introduce Mamba3R, which augments a frozen recurrent backbone (specifically CUT3R) with a lightweight adapter (only 1.28 million parameters).
Key innovations include:
- Selective State-Space Mixer: Instead of standard attention, they use a 1D selective scan (borrowed from the Mamba architecture) to jointly process the backbone's native state and a new, fixed-capacity auxiliary memory. This allows the model to "selectively" decide what information from the past is relevant to the current frame.
- Fixed-Capacity Auxiliary Memory: A dedicated memory buffer ($M_t$) that carries information across frames without growing in size.
- Controlled Feedback Loop: To avoid destabilizing the pretrained backbone, the adapter's output is decomposed. The model calculates a "memory-induced increment" and attenuates it (using a factor $\alpha=0.5$) before adding it back to the main feature stream.
- History-Only Decay: A specific decay protocol ensures that the auxiliary memory doesn't become stale over very long traversals.
Datasets
The model was evaluated on:
- ScanNet: A standard indoor dataset for 3D reconstruction.
- KITTI-360: A large-scale outdoor dataset used to test long-sequence trajectory accuracy.
- Robot-Recorded Traversals: The authors collected their own data using a sensor-equipped wheelchair (LiDAR, RGB-D camera, and an onboard RTX 4060 GPU) to test real-world deployment.
Key Insights and Ideas
- Mamba for 3D States: While Mamba is usually applied to sequences of pixels or text, this paper shows it is highly effective at mixing "slots" in a recurrent 3D state, even when those slots don't have a clear spatial grid arrangement.
- The Power of Attenuation: A crucial finding was that injecting memory at "full strength" actually hurt performance. By regulating the feedback, the adapter acts as a corrective signal rather than an overwhelming one.
- Constant Throughput: Because the memory capacity is fixed, the inference speed (FPS) and memory usage remain virtually identical whether the robot has traveled 10 meters or 2 kilometers.
Key Results
Mamba3R consistently outperformed existing recurrent baselines across multiple metrics:
- Accuracy: On ScanNet, it improved Chamfer Distance (CD) by 9.6% and Trajectory Error (ATE) by 7.3% compared to the strong TTT3R baseline. On the more challenging KITTI-360 dataset, these gains grew to 25.1% and 11.7% respectively.
- Long-Sequence Stability: In traversals up to 3,000 frames, Mamba3R maintained the lowest trajectory error, often with less than half the error of the original frozen backbone.
- Real-Time Performance: On a consumer-grade RTX 4060, the model averaged 11.7 FPS, comfortably covering the 10 Hz rate of standard sensors while staying within a 5.1 GiB memory footprint.
Limitations and Future Work
While Mamba3R significantly reduces drift, it is still a "front-end" predictor. For perfect global consistency over very long loops, it still benefits from a "back-end" optimizer (like a pose-graph).
The authors note that their current 1D scan order is arbitrary (based on the backbone's token order). Future work could explore spatially-aware scanning patterns. Additionally, they aim to develop back-ends that are better optimized for the specific outputs of monocular streaming predictors to further reduce the latency of the complete system.
Summary updated