For years, 3D generative modeling was a field of specialists. If you wanted to generate a 3D car, you trained a model on cars; if you wanted a chair, you started over with a chair dataset. Unlike 2D image generation, which scaled rapidly to "open-domain" prompts, 3D generation remained trapped in category-specific silos.
The research direction emerging across these papers describes a fundamental shift: the transition from specialized 3D generators to large-vocabulary 3D foundation models. This shift is driven by new representations that compress 3D geometry into manageable latent spaces and training strategies that leverage massive, heterogeneous datasets.
The Representation Bottleneck
The primary obstacle to scaling 3D generation is the representation itself. Standard voxels are computationally expensive (scaling cubically), while meshes are topologically rigid and difficult for neural networks to "grow" from scratch.
To achieve large-vocabulary generation, researchers have moved toward hybrid representations that combine the structural regularity of grids with the detail of implicit functions. A recurring favorite is the triplane representation, which projects a 3D volume onto three orthogonal 2D planes. As shown in 3DGen by Anchit Gupta, mapping complex geometry into this continuous latent space allows a single model to handle the high variance in topology found across hundreds of thousands of unique objects.
Other approaches, like the Voxel-Point Progressive Representation (VPP) developed by Runpei Dong and colleagues, use a two-stage approach. By combining the structural strengths of voxels for coarse shape with the efficiency of point clouds for fine detail, VPP achieves high-fidelity generation across multiple categories in a fraction of a second.
From Denoising to Next-Part Prediction
Once an efficient representation is established, the "intelligence" of the model comes from the generative prior. Two paradigms from the 2D and language worlds are currently competing to dominate 3D:
- 3D Diffusion: Models like DiffTF, proposed by Ziang Cao and Ziwei Liu, apply denoising diffusion to 3D-aware Transformers. By training on a large vocabulary of categories simultaneously, these models learn a generalized prior of "3D-ness" that allows them to synthesize semantically rich assets far beyond simple geometric primitives.
- Autoregressive Generation: Taking a cue from Large Language Models, TAR3D by Xuying Zhang treats 3D generation as a "next-part prediction" task. By encoding shapes into a sequence of latent codes, the model can generate complex geometries part-by-part, bypassing the memory limitations that usually haunt high-resolution 3D grids.
Solving the 3D Data Scarcity
A model is only as diverse as its training data. The "ImageNet moment" for 3D has been delayed by the difficulty of acquiring high-quality 3D scans. To bridge this gap, researchers are looking toward two solutions: larger datasets and 2D-to-3D lifting.
Yushuang Wu and the team behind MVImgNet2.0 have provided a massive expansion of multi-view image data, covering over 500,000 objects. This scale is critical; as noted in their work, reaching a "critical mass" of categories allows models to learn generalized priors that improve performance even on unseen object types.
Alternatively, models like HOLODIFFUSION by Andrea Vedaldi demonstrate that we can train 3D diffusion models using only 2D images. By using differentiable rendering to "lift" 2D observations into a 3D feature grid, these models bypass the need for explicit 3D supervision, opening the door to training on the vast wealth of images available on the internet.
Disentanglement and Control
As these models scale to handle thousands of categories, the need for precise control becomes paramount. Philipp Schröppel and Jan Eric Lenssen introduced Neural Point Cloud Diffusion (NPCD) to address this. Their work, Neural Point Cloud Diffusion for Disentangled 3D Shape and Appearance Generation, shows that by performing diffusion on points where positions define geometry and high-dimensional features encode local details, we can swap the "skin" of an object without changing its "skeleton."
This disentanglement is the final piece of the puzzle for a universal 3D generator. It ensures that as we move toward open-domain synthesis, users can manipulate 3D assets with the same granularity they expect from professional modeling software, but with the speed and ease of a text prompt.