Efficient 3D Shape Generation via Hybrid Latent Representations

This direction is split into other directions: 3D representation trade-offs. Its last version stays readable and citable here.

Version 1 of 1183 papers190 conceptsfrozen 2026-10-04

In the transition from 2D image generation to 3D modeling, researchers have encountered a "cubic explosion." While a 2D image scales by the square of its resolution, a 3D voxel grid scales by the cube, leading to massive memory overhead and I/O bottlenecks. As Ka-Hei Hui and colleagues note in Make-A-Shape, this complexity has prevented 3D modeling from having its "Stable Diffusion moment."

To solve this, the field has moved toward hybrid latent representations: compact formats that compress 3D geometry into neural-friendly structures like tri-planes, implicit fields, or adaptive grids. These representations allow models to generate high-fidelity details without the computational cost of dense volumetric data.

The Tri-Plane Breakthrough

The most influential strategy for taming 3D data is the tri-plane representation. Instead of storing a full 3D grid, the object is projected onto three orthogonal 2D planes (XY, YZ, and XZ). To find the value of any point in 3D space, the model samples features from these three planes and aggregates them.

This approach, utilized by Simian Luo in the Improved Auto-regressive Model (ImAM), bridges the gap between 2D efficiency and 3D structure. By using tri-planes, models like DiffTF by Ziang Cao can leverage standard Transformer architectures to generate complex objects across hundreds of categories. The tri-plane acts as a "3D-aware" latent space that is far more compact than voxels but more structured than raw point clouds.

From Discrete Points to Continuous Fields

While tri-planes provide a structured backbone, the way we define the "surface" of an object has also shifted. Traditional methods used discrete points or voxels, which often result in jagged edges or "floating" artifacts. Modern research favors Neural Signed Distance Functions (SDFs).

An SDF is a continuous function that tells you how far any point is from the object's surface. Ruikai Cui and the team behind NeuSDFusion combine this with tri-planes to create a spatial-aware generative model. This hybrid approach allows for "infinite resolution"—because the surface is defined by a function, you can query it at any level of detail. Bo Jian Wu further refined this by learning a compact latent space specifically for these functions, allowing models to capture sharp, high-frequency details like clothing folds or mechanical parts.

Weight-Encoded Geometry

A more radical approach to compression is to treat the neural network itself as the representation. In Higher-Order Function Networks (HOF), Eric Mitchell and his team encode the geometry of an object directly into the weights of a tiny "mapping" network.

Instead of a latent vector representing a shape, the parameters of the network represent the shape. This "weight-encoding" can reduce storage requirements by 100x to 1000x compared to standard decoders. As explored in On the Effectiveness of Weight-Encoded Neural Implicit 3D Shapes, "purposefully overfitting" a small network to a single geometry provides a robust, scalable way to handle high-resolution surfaces that would otherwise overwhelm a system's memory.

Lagrangian Primitives and Adaptive Grids

The most recent frontier involves moving away from fixed grids entirely. Researchers are increasingly using Lagrangian representations, where the shape is composed of flexible primitives that move and deform.

  • Gaussian Splatting: Gaia Di Lorenzo and colleagues introduced Object-X, which compresses objects into a latent space that decodes into 3D Gaussian Splats. This achieves a 3,000x reduction in storage while maintaining high-fidelity textures.
  • Adaptive Grids: Dingdong Yang proposed GALA, which uses local grids that orient and scale themselves to match the object's surface. This combines the structural benefits of octrees with the detail of local grids, allowing the model to focus its "computational budget" only where the geometry is complex.

Diffusion as the Generative Engine

To actually generate these hybrid structures, the field has converged on Diffusion Models. By treating 3D generation as a denoising process, models can capture the "one-to-many" nature of 3D reconstruction (e.g., there are many possible 3D chairs that could match a single 2D image).

Linqi Zhou demonstrated this with Point-Voxel Diffusion (PVD), which uses a hybrid representation to handle both global structure and local detail. More recently, Zhaoxi Chen introduced 3DTopia-XL, which scales this "primitive diffusion" to generate professional-grade assets for gaming and film, proving that hybrid representations are the key to unlocking large-scale 3D synthesis.

Papers in this version

Showing 60 of 183 papers.

History & citationVersion 1 · 2026-10-04

Split into other directions · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_efficient_3d_shape_generation_via_hybrid_e1b9d17f,
  title = {Efficient 3D Shape Generation via Hybrid Latent Representations},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/efficient-3d-shape-generation-via-hybrid-latent-representations/rdr_e1b9d17f0bdc439f8f5e628099559ca0},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_efficient_3d_shape_generation_via_hybrid_e1b9d17f_v1,
  title = {Efficient 3D Shape Generation via Hybrid Latent Representations},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/efficient-3d-shape-generation-via-hybrid-latent-representations/rdr_e1b9d17f0bdc439f8f5e628099559ca0/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}