3D Generation via Latent Space Diffusion

33 concepts29 papers1 proposal

Generative diffusion models have revolutionized 2D image synthesis by breaking down the complex task of image creation into a sequence of simple denoising steps. However, directly translating this success to the 3D domain introduces a fundamental representation bottleneck.

Unlike 2D images, which are uniformly represented as regular grids of pixels, 3D assets can take many forms: voxels, point clouds, meshes, neural radiance fields (NeRFs), or Gaussian splats. Operating a diffusion process directly on these raw representations scales poorly. For instance, dense voxel grids suffer from cubic computational complexity ($O(N^3)$), while meshes and point clouds possess irregular topologies that break the spatial inductive biases of standard convolutional or transformer architectures.

To overcome these limitations, a powerful paradigm has emerged: 3D Latent Space Diffusion. The core insight is to decouple the problem into two distinct stages:

  1. Dimensionality Reduction: Compress complex, high-dimensional 3D geometric and volumetric data into a compact, highly structured latent space.
  2. Generative Modeling: Train a diffusion model exclusively within this low-dimensional latent space to navigate the manifold of valid 3D shapes and textures.

By diffusing in a latent space rather than raw geometric space, models can leverage highly optimized architectures, maintain strict multi-view consistency, and scale effectively with data.


Compressing Geometry into Latent Manifolds

The first step in latent 3D diffusion is establishing a continuous mapping between raw 3D data and a compact latent representation. Researchers achieve this primarily through two distinct strategies: variational autoencoders (VAEs) trained on geometric fields, or leveraging the pre-structured manifolds of generative adversarial networks (GANs).

Autoencoding Geometric Fields and Primitives

One approach is to train an autoencoder to compress explicit or implicit 3D representations into a lower-dimensional grid or token set. For example, in Diffusion-SDF: Text-to-Shape via Voxelized Diffusion, Muheng Li et al. compress voxelized Signed Distance Fields (SDFs) into a compact latent space. This retains the spatial layout of the 3D grid but strips away redundant high-frequency geometric data, allowing the subsequent diffusion process to focus on structural layout without the prohibitive memory cost of high-resolution volumes.

Instead of global grids, the compression can also operate on localized primitives or tokens. In 3DTopia-XL: Scaling High-quality 3D Asset Generation via Primitive Diffusion, Zhaoxi Chen et al. utilize a VAE to compress local primitive patches into a latent space. Similarly, Biao Zhang et al. introduce 3DShape2VecSet: A 3D Shape Representation for Neural Fields and Generative Diffusion Models, which encodes complex geometry into a fixed-size set of latent vectors. This tokenization strategy effectively turns a 3D shape into a sequence of discrete embeddings, making it perfectly compatible with sequence-modeling architectures.

This autoencoding strategy extends across diverse modalities. In molecular biology, Matthieu Kirchmeyer et al. demonstrate in Unified all-atom molecule generation with neural fields that continuous atomic densities can be compressed into spatially arranged feature maps. For explicit surfaces, the authors of EdgeRunner: Auto-regressive Auto-encoder for Artistic Mesh Generation extract a fixed-length latent space from mesh sequences, effectively isolating the geometric reconstruction challenge from the generative task.

Leveraging Pre-trained 3D GAN Spaces

An alternative to training a new autoencoder from scratch is to perform diffusion within the latent space of an already optimized 3D-aware GAN. GANs excel at producing photorealistic, multi-view consistent assets, but they suffer from limited distribution coverage and lack intuitive conditioning mechanisms.

By treating a pre-trained 3D GAN as a structural manifold prior, a diffusion model can learn to navigate this space to generate assets. In GANFusion: Feed-Forward Text-to-3D with Diffusion in GAN Space, Souhaib Attaiki distills the latent space of an unconditional 3D GAN into a text-conditioned diffusion model. This architecture bypasses test-time optimization entirely; the diffusion process simply samples a valid path through the GAN's latent space, ensuring that the output inherently respects the structural consistency of the underlying 3D generator.

This mechanism proves highly effective for domain-specific tasks like facial modeling. Jianhui Li et al. develop InstructPix2NeRF: Instructed 3D Portrait Editing from a Single Image, which performs diffusion directly within the style-code space of a 3D portrait GAN. Similarly, Chang Zhou et al. demonstrate in Controllable 3D Face Generation with Conditional Style Code Diffusion that diffusing within a GAN's structured manifold guarantees that generated samples remain within the distribution of realistic 3D shapes and textures, avoiding the chaotic structural failures that can occur when diffusing unconstrained pixels or voxels.


Why It Works: Denoising on the Manifold

Once the 3D data is mapped to a latent space, the diffusion model's job is to reverse a forward noise process applied to these latents. Why does this perform so much better than direct spatial diffusion?

Bypassing Structural Constraints via Alignment

Standard diffusion models assume data lies on a regular grid. 3D data often breaks this assumption due to geometric orientation and symmetries. By mapping data to an aligned latent space, we can use standard, highly parallelized architectures like Transformers without modification.

For example, in Scalable Non-Equivariant 3D Molecule Generation via Rotational Alignment, Yuhui Ding et al. show that rotating 3D molecular structures into a consistent orientation within a latent space allows standard, non-equivariant Transformer architectures to achieve state-of-the-art results. This eliminates the need for complex, computationally expensive equivariant network layers, allowing the model to fully benefit from the scaling laws developed for vision and language tasks.

Better Distribution Coverage

A common failure mode of 3D GANs is mode collapse, where the model fails to capture the full diversity of the training set. Latent diffusion models operating on the same spaces exhibit significantly better distribution coverage. As shown by Katja Schwarz et al. in WildFusion: Learning 3D-Aware Latent Diffusion Models in View Space, diffusion-based architectures are inherently more robust against mode collapse when scaling to unposed, real-world imagery, offering superior sample diversity while maintaining strict geometric fidelity.

[Raw 3D Data / Views] ──(3D Encoder / GAN Manifold)──> [Compact Latent Space]
                                                               │
                                                    (Diffusion Denoising)
                                                               │
                                                               ▼
[Generated 3D Asset] <──(Decoder / Splatting / NeRF)─── [Refined Latent]

Expressive Denoising Architectures

With the 3D asset reduced to a sequence of latent tokens or a compressed map, advanced denoising backbones become viable. Richard A. Newcombe et al. implement this in ShapeR: Robust Conditional 3D Shape Generation from Casual Captures, applying a rectified flow transformer to iteratively refine a noise distribution over latent vector sets. By operating on a tokenized latent space, the transformer's self-attention mechanism can freely learn global dependencies and complex spatial relationships across the asset without being constrained by rigid grid resolutions.


Multimodal Guidance and 3D Editing

A major advantage of 3D latent spaces is their capacity for semantic alignment with other modalities, such as text and images. Because the latent space compresses raw geometry into abstract features, it becomes much easier to map these features to the semantic spaces of large foundation models.

Alignment-Before-Generation

In Michelangelo: Conditional 3D Shape Generation based on Shape-Image-Text Aligned Latent Representation, Xin Chen et al. introduce an "alignment-before-generation" strategy. They first align the 3D shape latent space with pre-trained CLIP image and text spaces. When the latent diffusion model is subsequently trained within this shared space, it inherits the rich semantic relationships of the foundation model, allowing text prompts to guide the geometric synthesis process with high accuracy.

Similarly, Aashish Rai et al. demonstrate in Towards Realistic Generative 3D Face Models that linking 3D generative latent spaces with contrastive language-image pretraining enables precise, text-driven manipulation of 3D facial expressions and textures while preserving the underlying identity and structural integrity.

High-Level Controllers and Feed-Forward Synthesis

Instead of relying solely on joint embeddings, modern frameworks frequently employ large language models (LLMs) or 2D image priors as high-level controllers to guide the latent denoising process.

In UniUGG: Unified 3D Understanding and Generation via Geometric-Semantic Encoding, Yueming Xu et al. utilize an LLM to generate conditional features that guide a 3D latent diffusion model. This architecture allows the language model to handle abstract spatial reasoning and relationship modeling, while the diffusion process focuses entirely on the low-level mechanics of spatial denoising and reconstruction.

To achieve rapid, feed-forward generation without slow per-scene optimization, Yuanbo Yang et al. develop Prometheus: 3D-Aware Latent Diffusion Models for Feed-Forward Text-to-3D Scene Generation. By integrating 3D Gaussian Splatting into a latent diffusion paradigm, they leverage the massive 2D image priors embedded within Stable Diffusion to synthesize complete, high-quality 3D scenes in seconds.

Resolving Multi-View Inconsistencies in Editing

When performing 3D editing based on 2D image modifications, a common issue is that independent 2D edits are often view-inconsistent. Latent spaces provide a natural mechanism to resolve these contradictions.

In GenN2N: Generative NeRF2NeRF Translation, Xiangyue Liu et al. model the distribution of inconsistent 2D edits by mapping them into a generative VAE-GAN latent space. By sampling from this learned latent distribution, the model can synthesize diverse, completely multi-view consistent 3D scenes.

For refinement tasks, Or Patashnik introduces Sharp-It: A Multi-view to Multi-view Diffusion Model for 3D Synthesis and Manipulation, which takes a coarse 3D shape and refines its rendered views through a specialized multi-view diffusion process. By integrating text prompts via cross-attention layers, specific attributes like material properties or styles can be modified dynamically, using the underlying coarse 3D structure as a geometric anchor to ensure the final asset remains structurally sound and consistent from every angle.

Key Foundational Work

Controllable 3D Face Generation with Conditional Style Code Diffusion (2024) — contributed 2 concept elements to this direction.
Control3Diff: Learning Controllable 3D Diffusion Models from Single-view Images (2024) — contributed 2 concept elements to this direction.
InstructPix2NeRF: Instructed 3D Portrait Editing from a Single Image (2023) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_latent_space_diffusion_latent_diffusion__38181b1c,
  title = {latent space diffusion (latent diffusion for 3D)},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/latent-space-diffusion-latent-diffusion-for-3d/rdr_38181b1cfdb54dd8bb02eaae96dca680},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_latent_space_diffusion_latent_diffusion__38181b1c_v1,
  title = {latent space diffusion (latent diffusion for 3D)},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/latent-space-diffusion-latent-diffusion-for-3d/rdr_38181b1cfdb54dd8bb02eaae96dca680/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake