SGAM: Building a Virtual 3D World through Simultaneous Generation and Mapping

Yuan Shen, Wei-Chiu Ma, Shenlong WangNeurIPS 2022 · OpenReview

Executive Summary

Simultaneous Generation and Mapping (SGAM) is a novel framework designed to build massive, globally consistent 3D worlds starting from a single image observation. By tightly coupling a generative "imagination" engine with a real-time 3D mapping system, SGAM can unroll thousands of steps of a camera trajectory without suffering from the drifting or structural collapse that typically plagues traditional video generation methods.

The Challenge of Infinite 3D Worlds

Creating large-scale virtual environments is a monumental task. While AI has made huge strides in generating realistic images and small 3D objects, scaling this capability up to an "infinite" world remains incredibly difficult.

Existing approaches usually fall into two traps:

  1. 2D Video Generation: These models can create realistic sequences, but they lack a true understanding of 3D space. As the camera moves, the world "drifts"—objects change shape, morph, or disappear entirely when you look away and look back.
  2. View Synthesis: These methods excel at small movements but struggle to "hallucinate" entirely new, distant regions of a scene while maintaining global consistency.

The authors identify a critical missing link in current generative models: memory. To build a world that remains consistent regardless of camera movement, the AI requires a mechanism to store and update a persistent 3D map of everything it has generated so far.

The SGAM Framework: Imagination Meets Memory

The core of SGAM is an autoregressive loop inspired by SLAM (Simultaneous Localization and Mapping) in robotics. It consists of two main modules that work in tandem at every timestamp:

  1. Generative Sensor Module (Imagination): Given the current 3D map and a new camera pose, this module first renders what it already knows about the scene. Because the map is incomplete, this usually results in an image with holes. It then uses a refined VQ-GAN (Vector Quantized Generative Adversarial Network) to fill in those holes and refine the details, producing a complete, photo-realistic RGB-D (color and depth) image.
  2. Mapping Module (Memory): This module takes the newly generated RGB-D image and integrates it back into a global 3D representation. The authors use a voxel-hashing volumetric representation (specifically, Truncated Signed Distance Functions or TSDF), which allows the map to grow efficiently to cover massive areas.
The SGAM framework: Generative Sensing creates new views, while Mapping integrates them into a persistent 3D scene.

The system is trained in two stages: first, learning to represent clean 3D data, and second, learning to "complete" the partial renders generated during the unrolling process.

Core Innovations

  • Mapping as an Anchor: The most important insight is that explicit 3D mapping prevents "mode collapse" and drift. By grounding the generation process in a persistent map, the model is forced to remain consistent with its own past.
  • The Power of RGB-D: Unlike many generative models that only predict pixels, SGAM predicts depth alongside color. This geometric information is crucial for the mapping module to correctly place new "hallucinations" into the 3D world.
  • Efficiency via One-Shot Denoising: Unlike some recent diffusion models or sequential VQ-GANs that are computationally heavy, SGAM’s generative refinement is highly efficient, allowing for much faster unrolling.

Evaluation Datasets

The researchers evaluated SGAM on two large-scale benchmarks:

  • CLEVR-Infinite: A new synthetic dataset created for this paper using Blender. It features a vast plane populated with thousands of geometric objects, providing a "ground truth" for evaluating 3D consistency.
  • GoogleEarth-Infinite: Real-world data scraped from Google Earth. This provides a much more complex and diverse set of textures and geometries to test the model's performance in "the wild."

Performance and Results

SGAM significantly outperforms prior state-of-the-art methods like Infinite Nature and GFVS in long-range generation.

  • Consistency: In the CLEVR-Infinite tests, SGAM maintained structural integrity over 900+ frames. In contrast, baseline methods often collapsed into blurry messes or hallucinated impossible geometries after just a few dozen steps.
  • Speed: SGAM is remarkably fast, achieving a 150x speedup in runtime compared to implicit generative methods (like GFVS-Implicit), making it much more practical for interactive applications.
  • Visual Quality: Quantitatively, SGAM achieved the best scores in PSNR, SSIM, and FID on the CLEVR-Infinite validation set, indicating it produces images that are both more accurate to the ground truth and more realistic.
Qualitative comparison showing how SGAM maintains consistency over 900 frames while other methods drift or fail.

Limitations and Future Directions

While SGAM is a major step forward, the authors note a few areas for improvement:

  • Real-world Realism: On the Google Earth dataset, while SGAM is more consistent over long distances, some existing "one-step" prediction methods still produce slightly sharper textures for immediate neighbors.
  • Dynamic Scenes: Currently, SGAM assumes a static world. Extending this to handle moving objects (like cars or walking people) while maintaining a 3D map is a significant open challenge.
  • Semantic Understanding: The model generates geometry and texture but doesn't "know" it's building a house or a tree. Integrating semantic labels could lead to even more plausible world-building.

Summary updated

Abstract

We present simultaneous generation and mapping (SGAM), a novel 3D scene generation algorithm. Our goal is to produce a realistic, globally consistent 3D world on a large scale. Achieving this goal is challenging and goes beyond the capacities of existing 3D generation or video generation approaches, which fail to scale up to create large, globally consistent 3D scene structures. Towards tackling the challenges, we take a hybrid approach that integrates generative sensor model- ing with 3D reconstruction. Our proposed approach is an autoregressive generative framework that simultaneously generates sensor data at novel viewpoints and builds a 3D map at each timestamp. Given an arbitrary camera trajectory, our method repeatedly applies this generation-and-mapping process for thousands of steps, allowing us to create a gigantic virtual world. Our model can be trained from RGB-D sequences without having access to the complete 3D scene structure. The generated scenes are readily compatible with various interactive environments and rendering engines. Experiments on CLEVER and GoogleEarth datasets demon- strates ours can generate consistent, realistic, and geometrically-plausible scenes that compare favorably to existing view synthesis methods. Our project page is available at https://yshen47.github.io/sgam.
BibTeX
@inproceedings{shen2022sgam,
  author = {Yuan Shen and Wei-Chiu Ma and Shenlong Wang},
  title = {{SGAM: Building a Virtual 3D World through Simultaneous Generation and Mapping}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2022},
  url = {https://openreview.net/forum?id=19h5YqfJ-a},
}
Figure