Generative modeling for 3D data has historically struggled with a fundamental dilemma: geometry is inherently continuous and high-dimensional, yet our most powerful generative architectures—such as Transformers and Large Language Models (LLMs)—excel at processing discrete sequences of categorical tokens. Traditional 3D representations like raw point clouds, dense voxel grids, or continuous Signed Distance Fields (SDFs) either scale poorly or introduce rigid inductive biases that limit structural diversity.
To bridge this gap, a powerful paradigm has emerged: tokenizing 3D spatial data into a discrete latent space. By compressing continuous 3D structures into a finite "vocabulary" of geometric parts, researchers can treat 3D generation, completion, and understanding exactly like natural language processing.
The Core Intuition: Building a Visual Vocabulary
The core mechanism behind this paradigm is vector quantization, typically implemented via a Vector-Quantized Variational Autoencoder (VQ-VAE) or VQ-GAN. The process transforms a complex 3D asset into a sequence of discrete tokens through a two-step pipeline:
[Continuous 3D Input] ──> [Spatial Decomposition] ──> [Vector Quantization] ──> [Discrete Token Sequence]
(Points, Voxels, SDFs) (Patches, Triplanes) (Codebook Mapping) ([Idx 42, Idx 107, ...])
- Spatial Decomposition: The continuous 3D space is broken down into localized components. For instance, Yu et al. and Gao et al. partition unstructured point clouds into local spatial patches. Alternatively, Mittal et al. and Xiong et al. compress volumetric Signed Distance Fields, while Luo et al. and the authors of TAR3D project geometry into compact, multi-view tri-planar representations. To break free from rigid grids entirely, Zhang et al. introduce irregular latent grids that adaptively place tokens at arbitrary 3D positions based on shape density.
- Quantization via a Codebook: Each localized 3D feature vector is mapped to its closest neighbor in a learned dictionary of vectors, known as a codebook. As Li et al. describe, this process establishes a shared vocabulary of geometric parts across entirely different objects. A complex 3D structure is thus reduced to a sequence or grid of integer indices (tokens), preserving spatial locality while discarding high-frequency noise.
Generative Paradigms on Discrete Spaces
Once 3D geometry is serialized into a discrete token sequence, the generative task is decoupled from the geometric representation. This allows researchers to deploy standard sequence-modeling architectures:
Autoregressive Next-Token Prediction
Just as GPT models predict the next word in a sentence, a transformer can predict the next geometric part of a 3D shape. Ye et al. and Zhao et al. convert 3D shapes into "shape sentences," allowing native multimodal LLMs to generate and edit 3D assets via standard next-token prediction.
This sequential generation is highly effective at capturing diverse topologies. For example, Medi et al. utilize autoregressive transformers over Unsigned Distance Fields (UDFs) to generate complex, non-watertight structures—like the interior seats and steering wheels inside a car—without suffering from the mode collapse common to continuous point-flow or diffusion alternatives.
Masked Spatial Modeling
Mirroring the BERT paradigm in NLP, models can learn robust 3D representations by filling in missing spatial blanks. Yu et al. pre-train standard transformers by masking random point cloud tokens and forcing the network to predict the underlying geometric vocabulary. Gao et al. unify this approach by combining masked autoencoding (for shape understanding) and autoregressive generation (for shape synthesis) within a single architecture.
Discrete Diffusion
Instead of adding continuous Gaussian noise to coordinates, diffusion can be performed over categorical token distributions. Li et al. combine patch-level VQ-VAEs with a discrete diffusion generator, demonstrating a highly efficient shape prior capable of handling unconditional generation and point cloud completion simultaneously.
Why Discretization Scales Better
Shifting from continuous coordinates to discrete codebook indices provides several fundamental advantages that make 3D modeling scalable and robust:
- Training Stability and Mode Collapse Prevention: Training generative adversarial networks (GANs) on diverse, unaligned 3D data is notoriously unstable. As Sargent et al. demonstrate when training 3D-aware models on ImageNet, vector-quantized frameworks stabilize training by separating the task of learning 3D representations from the task of modeling their distribution.
- Capturing Complex Spatial Dependencies: In high-fidelity domains like medical imaging, Tudosiu et al. show that discrete token grids allow transformers to capture intricate anatomical dependencies across 3D brain MRIs better than traditional convolutional networks. This structural reliability is critical for safety; Graham et al. exploit these discrete token distributions to perform reliable out-of-distribution detection, preventing medical models from generating plausible-looking but entirely fictitious segmentations.
- Native Cross-Modal Alignment: Because text, images, and tokenized 3D shapes share a discrete sequence format, cross-modal reasoning becomes straightforward. Liu et al. project 3D voxel patches into a shared embedding space with text, allowing an LLM to perform text-driven volume completion directly.
This paradigm scales across vastly different dimensions of spatial data. At the macro scale, Zheng et al. compress 3D semantic occupancy grids into discrete tokens to build an autonomous driving "world model" that predicts how an entire traffic scene will evolve over time. At the micro scale, Gao et al. map continuous molecular coordinates to a discrete "structural alphabet," enabling transformer-based language models to generate valid 3D molecular structures up to 28 times faster than continuous diffusion models.
Key Foundational Work
FullFormer: Generating Shapes Inside Shapes (2023) — contributed 2 concept elements to this direction.
Learning Versatile 3D Shape Generation with Improved Auto-regressive Models (2023) — contributed 1 concept elements to this direction.
AutoSDF: Shape Priors for 3D Completion, Reconstruction and Generation (2022) — contributed 1 concept elements to this direction.