Early text-to-3D generation relied heavily on optimizing neural representations—such as Neural Radiance Fields (NeRFs) or 3D Gaussian Splatting—via 2D diffusion models using Score Distillation Sampling (SDS). While revolutionary, this paradigm frequently suffers from the "Janus problem" (multi-faced artifacts), over-smoothing, and poor text-to-geometry alignment.
To overcome these limitations, the field has shifted toward scaling 3D generation by incorporating advanced generative priors. By leveraging rectified flow transformers, autoregressive sequence modeling, and multi-view reconstruction pipelines, modern frameworks achieve significantly higher structural fidelity, faster convergence, and superior prompt alignment.
Straightening the Path: Rectified Flow Transformers
Traditional diffusion models operate on curved trajectories in the noise-to-data space. During 3D score distillation or direct generation, these curved paths introduce high variance and optimization instability, which manifests as blurry geometries or oversaturated colors. Rectified flow models solve this by learning straight, linear trajectories between the noise distribution and the target 3D asset distribution.
Curved Diffusion Path: [Noise] --(High Variance / Curved)--> [3D Asset]
Rectified Flow Path: [Noise] ---------------------------> [3D Asset]
(Straight Line / Stable)
Xiang et al. demonstrate in Structured 3D Latents for Scalable and Versatile 3D Generation (TRELLIS) that rectified flow transformers provide substantially better text-prompt alignment and structural consistency than traditional diffusion architectures. By straightening the generation path, the model preserves intricate geometric details across complex 3D shapes.
Similarly, Li et al. introduce FlowDreamer in FLOWDREAMER: EXPLORING HIGH FIDELITY TEXT-TO-3D GENERATION VIA RECTIFIED FLOW. They exploit the unique mathematical properties of rectified flow to perform an intelligent noise search. This directly addresses the mathematical root cause of over-smoothing in SDS, yielding sharper textures and faster convergence for both NeRF and 3D Gaussian Splatting representations. To further stabilize optimization, Lei et al. propose Linearized Lookahead Variational Score Distillation ($L^2$-VSD) in Advancing Text-to-3D Generation with Linearized Lookahead Variational Score Distillation, optimizing the interaction between the 3D asset and the score distillation model to ensure training stability.
3D as a Language: Autoregressive Tokenization
Another prominent paradigm treats 3D shapes as discrete sequences, translating the massive success of Large Language Models (LLMs) into the spatial domain. By tokenizing continuous 3D geometry into a discrete codebook, models can generate assets through next-token or next-part prediction. This discretizing step allows the model to learn complex, multi-modal structural distributions without suffering from mode collapse.
3D Mesh/Triplane ──> [VQ-VAE Encoder] ──> Discrete Tokens ──> [Transformer (GPT)] ──> Next-Token Prediction
In ShapeGPT: 3D Shape Generation With a Unified Multi-Modal Language Model, Zhao et al. show that tokenizing 3D shapes into a language-like format allows a single instruction-following model to natively handle text-to-shape, image-to-shape, and shape editing tasks. Instead of training specialized architectures for each task, the unified language model treats multi-modal prompts as conditioning tokens.
This autoregressive sequence strategy scales effectively to highly intricate, domain-specific geometries:
- Anime Hairstyles: He et al. introduce CHARM in CHARM: Control-point-based 3D Anime Hairstyle Auto-Regressive Modeling, treating control-point-based representations as a structural "hair language" to autoregressively synthesize complex hairstyles.
- Part-by-Part Synthesis: The authors of TAR3D (TAR3D: Creating High-Quality 3D Assets via Next-Part Prediction) utilize a 3D VQ-VAE to map shapes into a compact triplane representation, allowing a GPT-style transformer to autoregressively predict assets part-by-part for enhanced semantic alignment.
- Coarse-to-Fine Hierarchies: An et al. propose Fractal3DGen in FRACTAL3DGEN: COARSE-TO-FINE 3D GENERATION VIA FRACTAL HIERARCHICAL TRANSFORMERS, combining autoregressive generation with fractal theory. Their hierarchical transformer architecture uses adaptive pruning to generate shapes from coarse structures down to fine details, significantly reducing computational overhead.
Decomposition and Localization: Part-Based Latents
Holistic generation models often struggle with complex or articulated objects because a single global latent vector must compress all geometric nuances. Decomposing the generation process into localized regions or distinct parts provides a solution, allowing models to focus capacity on local fidelity.
The authors of CoPart demonstrate this in From One to More: Contextual Part Latents for 3D Generation by introducing a framework that models individual parts and their contextual relationships independently. This decomposition enables finer structural control and supports downstream applications like part editing and articulation. Similarly, for targeted modifications, the authors of ShapeFusion (Shapefusion: 3D localized human diffusion models) utilize a masked diffusion process to perform localized 3D shape manipulations, ensuring that modifications to specific regions do not corrupt the identity or realism of the remaining structure.
To inject structural awareness into models that lack explicit part tokenization, Sella et al. introduce a cross-entity attention mechanism in Spice·E: Structural Priors in 3D Diffusion using Cross-Entity Attention. This mechanism injects structural guidance directly into pre-trained 3D diffusion models, enabling rapid semantic shape editing and abstraction-to-3D transformations in seconds rather than hours.
Lifting 2D Priors via Multi-View Reconstruction
Rather than generating 3D structures directly from scratch—which is bottlenecked by the scarcity of native 3D training data—an alternative strategy leverages the highly mature capabilities of 2D diffusion models. These frameworks generate consistent multi-view images or videos first, and then lift them into 3D space using explicit geometric optimization.
Text Prompt ──> [2D Multi-View Diffusion] ──> Consistent Views ──> [Geometric Optimization (3DGS/NeRF)] ──> High-Fidelity 3D Asset
Gu et al. implement this bridge in Text-to-3D scene generation framework: bridging textual descriptions to high-fidelity 3D scenes (3DS-Gen). The framework first synthesizes a coherent multi-view video from a text prompt, then reconstructs the 3D scene using 3D Gaussian Splatting combined with an adaptive density optimization step to eliminate floating artifacts.
To prevent spatial distortions during this lifting process, Hou et al. introduce SceneWeaver in SceneWeaver: Text-Driven Scene Generation with Geometry-aware Gaussian Splatting. SceneWeaver refines depth maps to construct an initial point cloud before executing a geometry-aware optimization of the 3D Gaussians, ensuring global consistency. For interactive applications, Chen et al. leverage this multi-view generation-reconstruction pipeline in MVDrag3D: Drag-based Creative 3D Editing via Multi-view Generation-Reconstruction Prior to enable intuitive, drag-based structural editing of 3D assets.
When applied to highly constrained domains like human synthesis, these multi-view priors require specialized geometric and appearance constraints to maintain realism:
- InceptionHuman: Kao et al. (InceptionHuman: Controllable Prompt-to-NeRF for Photorealistic 3D Human Generation) combine controllable 2D diffusion with NeRFs, utilizing iterative view-refinement modules to ensure strict 3D consistency across poses.
- SEEAvatar: Xu et al. (Photorealistic Text-to-3D Avatar Generation with Constrained Geometry and Appearance) introduce self-evolving constraints for both geometry and appearance, producing detailed meshes and textures that seamlessly fit standard graphics pipelines.
Data Scaling and Automated Evaluation
The performance of these advanced generative models is fundamentally bound by the scale and quality of their training data, alongside the ability to reliably evaluate outputs. Recent work has focused on automating both dataset curation and model evaluation to bypass human bottlenecks.
View Selection for Dataset Scaling
To scale up 3D-text datasets without introducing low-quality or uninformative training pairs, Luo et al. introduce DiffuRank in View Selection for 3D Captioning via Diffusion Ranking. DiffuRank uses pre-trained diffusion models to evaluate and rank the informativeness of 2D views of 3D objects. By filtering out "atypical" views that lead to ambiguous captions, they corrected 200,000 existing captions and scaled the Cap3D dataset to over 1 million high-quality 3D-text pairs, significantly reducing hallucinations in downstream generative models.
Vision-Language Models as Evaluators
As models grow more sophisticated, traditional metrics like CLIP similarity often fail to capture complex geometric distortions or multi-view inconsistencies. To establish a scalable, human-aligned benchmark, Wu et al. demonstrate in GPT-4V(ision) is a Human-Aligned Evaluator for Text-to-3D Generation that multimodal large language models like GPT-4V can serve as automated evaluators. By systematically comparing rendered views of generated 3D assets against input prompts, this approach automatically assigns robust Elo ratings that correlate strongly with human judgment, providing a holistic framework for assessing model performance.
Key Foundational Work
TAPS3D: Text-Guided 3D Textured Shape Generation from Pseudo Supervision (2023) — contributed 1 concept elements to this direction.
TM-NET: deep generative networks for textured meshes (2021) — contributed 1 concept elements to this direction.