Gamba: Marry Gaussian Splatting with Mamba for single view 3D reconstruction

Qiuhong Shen, Xuanyu Yi, Zike Wu, Pan Zhou, Hanwang Zhang, Shuicheng Yan, Xinchao Wang

TL;DR

Gamba is a novel end-to-end model that reconstructs high-quality 3D assets from a single image in just 50 milliseconds. By replacing traditional Transformer backbones with the linear-scaling Mamba architecture, the researchers successfully modeled 3D Gaussian Splatting as a sequential prediction task, achieving a 1,000x speedup over optimization-based methods.

What Problem Does This Paper Address?

Creating a 3D model from a single 2D image is a "holy grail" for AR/VR and autonomous driving, but current methods face a frustrating trade-off between quality and speed.

Existing approaches generally fall into two camps:

  1. Optimization-based methods: These use Score Distillation Sampling (SDS) to slowly "carve" a 3D shape. While they produce high-fidelity results, they take minutes or even hours per object.
  2. Feed-forward models: These use Transformers to predict 3D representations (like NeRFs or Gaussians) instantly. However, Transformers have a "quadratic bottleneck"—their memory usage explodes as you add more detail. To represent a complex 3D object, you need a lot of 3D Gaussians (tokens), but Transformers struggle to handle the sequence lengths required for high-resolution reconstruction.

The authors of Gamba set out to build a model that could handle a massive number of Gaussians (16,384) in a single pass without hitting a memory wall.

Methods

The core innovation is the GambaFormer, a backbone built on the Mamba architecture (a type of State Space Model). Unlike Transformers, Mamba scales linearly with sequence length, making it perfectly suited for processing the large number of tokens needed for 3D Gaussian Splatting (3DGS).

Key components include:

  • Sequential Prediction Insight: The authors observed that the iterative process of "densifying" Gaussians in traditional 3DGS reconstruction looks a lot like a sequential prediction task. Mamba excels at this type of context-dependent reasoning.
  • Gamba Block: This custom layer uses a "Prepend and Drop" mechanism. It attaches condition tokens (image and camera pose features) to the beginning of the Gaussian token sequence, processes them through a Mamba block, and then drops the conditions to pass only the updated 3D information to the next layer.
  • Radial Mask Constraint: Training these models usually requires 3D point cloud data for "warmup" supervision to prevent the model from collapsing. Gamba introduces a clever geometric loss that uses 2D masks and ray casting to constrain Gaussians within the object's silhouette, eliminating the need for expensive 3D ground truth during training.
Figure 2: Overall architecture of Gamba. Gamba takes a single view image and its camera pose as input to predict the 3D Gaussian Splatting of the given subject. Training supervision is only applied on the rendered multi-view images through reconstruction loss.

Datasets

  • Training: The model was trained on a filtered subset of Objaverse (specifically the LVIS subset), consisting of approximately 20,000 high-quality 3D objects.
  • Evaluation: Quantitative benchmarks were performed on the Google Scanned Objects (GSO) dataset, using 60 diverse 3D assets to test the model's ability to generalize to unseen shapes.

Key Insights and Ideas

  • Linear Scalability is the Game Changer: By moving away from Transformers, the researchers could predict 16,384 Gaussians in a single forward pass. Previous Transformer-based models often had to resort to multi-stage training or "super-resolution" tricks to reach that level of detail because of memory constraints.
  • 3DGS as a Sequence: Treating a set of discrete, non-structural 3D Gaussians as a sequence that can be "reasoned" over by a state-space model is a significant conceptual shift from treating them as a 2D grid or a point cloud.
  • Eliminating 3D Supervision: The radial mask constraint proves that we can achieve stable 3D geometry using only 2D image-space losses, which simplifies the training pipeline significantly.

Key Results

Gamba sets a new bar for efficiency in single-view reconstruction:

  • Speed: It completes a full reconstruction in 0.05 seconds on an NVIDIA A100 GPU. For comparison, optimization-based methods like Zero-1-to-3 take about 30 minutes (1,800 seconds).
  • Performance: On the GSO dataset, Gamba achieved a PSNR of 24.74, outperforming other feed-forward models like LRM (21.35) and TGS (22.68).
  • Qualitative Quality: As seen in the comparisons, Gamba maintains better geometric consistency and sharper textures than its predecessors, avoiding the "warped" look common in earlier large reconstruction models.
Figure 4: Qualitative Comparison with large reconstruction models.

Limitations and Future Work

Despite its speed, Gamba has a few hurdles to clear:

  • Input Sensitivity: The quality of the 3D output is highly dependent on the input image. If the photo lacks geometric cues, the reconstruction struggles.
  • The "Back-View" Problem: Because Gamba is a prediction model rather than a generative one (like a diffusion model), it tends to produce smoothed-out textures on the back side of objects that weren't visible in the original photo.
  • Model Scale: The current version uses only 14 blocks—a fraction of the size of modern LLMs. The authors suggest that scaling up the architecture could lead to even better results, though this remains unexplored.

Summary updated

Abstract

We tackle the challenge of efficiently reconstructing a 3D asset from a single image at millisecond speed. Existing methods for single-image 3D reconstruction are primarily based on Score Distillation Sampling (SDS) with Neural 3D representations. Despite promising results, these approaches encounter practical limitations due to lengthy optimizations and significant memory consumption. In this work, we introduce Gamba, an end-to-end 3D reconstruction model from a single-view image, emphasizing two main insights: (1) Efficient Backbone Design: introducing a Mamba-based GambaFormer network to model 3D Gaussian Splatting (3DGS) reconstruction as sequential prediction with linear scalability of token length, thereby accommodating a substantial number of Gaussians; (2) Robust Gaussian Constraints: deriving radial mask constraints from multi-view masks to eliminate the need for warmup supervision of 3D point clouds in training. We trained Gamba on Objaverse and assessed it against existing optimization-based and feed-forward 3D reconstruction approaches on the GSO Dataset, among which Gamba is the only end-to-end trained single-view reconstruction model with 3DGS. Experimental results demonstrate its competitive generation capabilities both qualitatively and quantitatively and highlight its remarkable speed: Gamba completes reconstruction within 0.05 seconds on a single NVIDIA A100 GPU, which is about $1,000\times$ faster than optimization-based methods. Please see our project page at https://florinshen.github.io/gamba-project.
BibTeX
@misc{shen2024gamba,
  author = {Qiuhong Shen and Xuanyu Yi and Zike Wu and Pan Zhou and Hanwang Zhang and Shuicheng Yan and Xinchao Wang},
  title = {{Gamba: Marry Gaussian Splatting with Mamba for single view 3D reconstruction}},
  year = {2024},
  eprint = {2403.18795},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2403.18795},
}
Figure