Introduction
Generating high-quality 3D assets from textual descriptions remains a significant challenge in computer vision, primarily due to the scarcity of large-scale 3D datasets compared to the billions of samples available for 2D image generation. Traditional approaches often rely on Score Distillation Sampling (SDS), which uses a 2D diffusion model to iteratively optimize a 3D representation. While effective, SDS is computationally expensive—often requiring 20 minutes to several hours per object—and prone to mode collapse. HexaGen3D introduces a feedforward alternative that leverages the rich priors of pretrained 2D models like Stable Diffusion to generate textured 3D meshes in approximately 7 seconds.
Methodology
HexaGen3D employs a multi-stage pipeline designed to bridge the gap between 2D generative priors and 3D spatial representations.
Triplanar VAE and Hexaview Guidance
The system first utilizes a Variational Autoencoder (VAE) to compress 3D objects into a triplanar format—three orthogonal 2D feature maps (XY, YZ, XZ). To generate these triplanes from text, the authors introduce Orthographic Hexaview Guidance. Instead of directly predicting abstract triplanar latents, the model is fine-tuned to generate six orthographic views (front, back, left, right, top, bottom) arranged in a 2x3 grid. This intermediate step allows the model to utilize its existing understanding of 3D structure through 2D projections.
Layout Conversion and Texture Baking
A specialized CNN-based layout converter maps the six orthographic views into the triplanar latent space. The lack of perspective distortion in orthographic views ensures a direct alignment with the triplanar axes, facilitating efficient conversion. Finally, to achieve high visual fidelity, the model "bakes" textures from the predicted 2D views onto the generated mesh.
Results and Discussion
HexaGen3D demonstrates a significant leap in both speed and diversity. By moving from optimization-based methods to a feedforward pass, it achieves a 1600x speedup over methods like MVDream. Furthermore, the use of SDXL as a foundation model significantly improves prompt adherence and visual quality compared to earlier iterations.
In user preference studies, HexaGen3D-SDXL outperformed feedforward baselines like Shape-E by a wide margin. Quantitatively, it achieves CLIP scores comparable to optimization-heavy methods while maintaining the ability to produce diverse outputs for the same prompt under different random seeds.
Limitations
Despite its efficiency, HexaGen3D faces challenges with extremely thin or intricate geometries, such as fine wires, which are difficult to capture in the triplanar format. Additionally, inconsistencies in the generated backgrounds can occasionally lead to "scaffolding" or box-like artifacts around the mesh. Future work aims to address these by exploring higher-resolution latents and improved VAE architectures.
Summary updated