Abstract
We explore the task of text to 3D object generation using CLIP. Specifically, we use CLIP for guidance without access to any datasets, a setting we refer to as pure CLIP guidance. While prior work has adopted this setting, there is no systematic study of mechanics for preventing adversarial generations within CLIP. We illustrate how different image-based augmentations prevent the adversarial generation problem, and how the generated results are impacted. We test different CLIP model architectures and show that ensembling different models for guidance can prevent adversarial generations within bigger models and generate sharper results. Furthermore, we implement an implicit voxel grid model to show how neural networks provide an additional layer of regularization, resulting in better geometrical structure and coherency of generated objects. Compared to prior work, we achieve more coherent results with higher memory efficiency and faster training speeds.
Introduction
Generating 3D content from text is a "holy grail" for digital media, but it faces a massive data bottleneck: while we have billions of paired images and captions, high-quality 3D datasets are scarce. To bypass this, researchers use CLIP (a model that understands the relationship between text and images) to "guide" a 3D representation like a Neural Radiance Field (NeRF) until its 2D renders match a text prompt.
However, this "pure CLIP guidance" (no 3D data) is notoriously difficult. CLIP is prone to adversarial generations—it often rewards "hallucinated" textures or noisy patterns that satisfy its mathematical objectives but look like nonsense to humans. Previous attempts, like Dream Fields, were computationally expensive and often produced "cloudy" or disconnected geometry.
Methodology
The proposed framework is based on Direct Voxel Grid Optimization (DVGO), which represents 3D space as a grid of values rather than a deep, slow-to-query neural network. Two variants are explored:
- VoxExp (Explicit): Density and color are stored directly in the voxel grid.
- VoxImp (Implicit): The grid stores features that are then processed by a shallow Multi-Layer Perceptron (MLP).
To stabilize the optimization, several "regularizers" are implemented:
- Augmentations: Background, perspective, and "DiffAug" (color jittering/translation) prevent CLIP from focusing on pixel-level noise.
- Spherical Prior (KL Loss): This encourages the model to concentrate density in the center of the scene, preventing the "floating cloud" problem.
- CLIP Ensembling: Instead of relying on one CLIP model, the system ensembles different architectures (e.g., ViT-B/16 and ViT-L/14). This prevents larger models from "overfitting" on adversarial textures.
Results and Key Insights
The VoxImp model achieved an R-Precision of 88.24% (evaluated with ViT-B/32), significantly outperforming Dream Fields at 59.8%. Efficiency is a major advantage; the model can be optimized on a consumer-grade RTX 2080 Ti in roughly 20 minutes using only 7GB of VRAM.
Several key insights emerged from this study:
- The "Adversarial" Trap: Larger CLIP models (like ViT-L/14) are more susceptible to adversarial attacks when used alone. Ensembling them with smaller, more "structural" models (ViT-B/16) forces the 3D model to maintain better global shape.
- Neural Networks as Regularizers: The implicit model (VoxImp) performed better than the explicit one. The MLP acts as a natural "geometric smoother," reducing jagged, non-coherent artifacts.
- The KL Loss "Seed": The KL loss provides the initial "nudge" needed for density to emerge in the center of the grid, which CLIP can then refine into an object.
Limitations and Future Work
Despite improvements, the model still faces the Janus Problem, where objects may have multiple faces (e.g., a cat with faces on all sides) due to CLIP's lack of inherent 3D consistency. Complex anatomical structures also remain challenging. Future work could involve incorporating 3D-aware priors or exploring how different training datasets for CLIP impact the diversity and quality of the 3D generations.
Summary updated