UNDERSTANDING PURE CLIP GUIDANCE FOR VOXEL GRID NERF MODELS

Han-Hung Lee, Angel X Chang

TL;DR

This paper investigates "pure CLIP guidance" for generating 3D objects from text using fast voxel grid radiance fields instead of heavy implicit MLPs. The authors systematically demonstrate how differentiable augmentations, loss scheduling, a spherical KL divergence prior, and multi-model CLIP ensembling prevent adversarial artifacts, yielding faster convergence and higher-fidelity 3D representations than prior baselines.

What Problem Does This Paper Address?

Generating 3D models directly from text is notoriously challenging because large-scale paired 3D-text datasets are scarce. While methods like Dream Fields demonstrated that zero-shot text-to-3D optimization is possible by using pretrained vision-language models (like OpenAI's CLIP) to evaluate rendered views of a NeRF (Neural Radiance Field), they suffer from two major drawbacks:

  1. Adversarial cheating: CLIP is easily fooled by high-frequency visual noise or textures that score highly on cosine similarity without looking like real 3D objects to humans.
  2. Computational overhead: Optimizing full coordinate-based MLPs (e.g., Mip-NeRF) per prompt is painfully slow and memory-intensive. Direct voxel grid methods (e.g., DVGO) converge orders of magnitude faster, but optimizing raw voxel densities under pure CLIP guidance often leads to empty space or severe artifacts.

Methods

The authors build on Direct Voxel Grid Optimization (DVGO) and analyze two representations:

  • $\text{Vox}_\text{Exp}$ (Explicit): Directly optimizes 3D density and color voxel grids without a neural network.
  • $\text{Vox}_\text{Imp}$ (Implicit): Predicts density and color from fixed Fourier positional encodings using small 3-layer coordinate MLPs, adding an inductive bias for geometric smoothness.

To stabilize training under pure CLIP guidance without any ground-truth 3D data, they introduce:

  • A Multi-Augmentation Pipeline: Renders are augmented using differentiable augmentations (DiffAug: color jitter, translation, cutout), background compositing (BackAug: noise, checkerboards, textures), and perspective distortions (PerspAug) before being fed to CLIP.
  • Spherical Prior KL Divergence Loss ($\mathcal{L}_{\text{KL}_s}$): Directs voxel mass into a unit sphere early in optimization, preventing empty gradients caused by augmentation-induced discontinuity.
  • Loss Scheduling: Crucially, $\mathcal{L}_{\text{KL}_s}$ and total variation loss ($\mathcal{L}_{\text{TV}}$) are active during initial/middle training to seed and smooth the geometry, then disabled near the end to let sharp details and textures emerge.
  • Multi-CLIP Ensembling: Combining different architectures (e.g., ViT-B/16 or ViT-B/32 paired with the larger ViT-L/14) to capture both coarse object structure and fine textural detail.

Datasets

  • Pretrained Guidance Models: Pretrained OpenAI CLIP models (RN50, RN50x4, ViT-B/32, ViT-B/16, ViT-L/14) and OpenCLIP models trained on LAION-400M and LAION-2B.
  • Evaluation Benchmark: A validation set of 153 text prompts originally introduced by Dream Fields (Jain et al., 2022) to evaluate zero-shot text-to-3D retrieval accuracy. No 3D dataset is used during training.

Key Insights and Ideas

  • Augmentations cause a "cold start" gradient failure: Adding heavy image augmentations flattens the CLIP similarity landscape early on, leading to near-zero gradients and empty grids. The spherical KL prior is essential because it forcibly seeds initial densities inside a bounded volume, giving CLIP meaningful rendered pixels to optimize against.
  • Bigger CLIP models aren't always better: Guidance with the largest model ($\text{ViT-L/14}$) on its own collapses into noisy, adversarial artifacts. Vision transformers appear to overfit high-frequency textures rather than macro-geometry.
  • Ensembling tames large CLIP models: Pairing a smaller model ($\text{ViT-B/32}$ or $\text{ViT-B/16}$) with $\text{ViT-L/14}$ prevents the larger model from falling into adversarial local minima, allowing the model to leverage $\text{ViT-L/14}$’s fine patch resolution safely.
  • Implicit MLPs act as structural regularizers: While explicit grids ($\text{Vox}_\text{Exp}$) capture textures well, geometry can be disjointed. Adding shallow MLPs ($\text{Vox}_\text{Imp}$) provides an inductive bias that naturally eliminates background floaters and produces complete, solid geometry even without perspective augmentation.

Key Results

  • Retrieval Accuracy (R-Precision@1): Evaluated with a held-out retrieval model ($\text{ViT-B/32}$) on 153 prompts, $\text{Vox}_\text{Imp}$ achieved 88.24% and $\text{Vox}_\text{Exp}$ achieved 83.01%, significantly outperforming Dream Fields (59.8%).
  • Memory & Speedup:
  • The smallest explicit model ($\text{Vox}_\text{Exp}$ with $\text{ViT-B/32}$) trained in roughly 20–35 minutes using 7–10 GB VRAM on an RTX 2080 Ti.
  • Dream Fields with $\text{ViT-B/16}$ ran out of memory (OOM) on an RTX A6000 (48 GB) and took over 2 hours 17 minutes on a TPU v2-8. In contrast, $\text{Vox}_\text{Imp}$ with $\text{ViT-B/16}$ completed in ~40 minutes on an RTX A6000.

Limitations and Future Work

  • The "Janus" Problem: For complex objects (especially quadrupeds or humans like cats and zebras), the guidance often repeats facial features across multiple sides due to lack of 3D viewpoint awareness in vanilla 2D CLIP.
  • Evaluation Limitations: The authors note that CLIP R-Precision is a flawed metric; objects with broken 3D geometry can still achieve high R-Precision scores as long as the surface texture superficially matches prompt keywords.
  • Voxel Memory Consumption: Current implementations evaluate the forward pass over the entire positional encoding grid regardless of ray occupancy, limiting scaling to higher grid resolutions without large-memory GPUs. Future work plans to prune unhit voxels dynamically.

Summary updated

Abstract

We explore the task of text to 3D object generation using CLIP. Specifically, we use CLIP for guidance without access to any datasets, a setting we refer to as pure CLIP guidance. While prior work has adopted this setting, there is no systematic study of mechanics for preventing adversarial generations within CLIP. We illustrate how different image-based augmentations prevent the adversarial generation problem, and how the generated results are impacted. We test different CLIP model architectures and show that ensembling different models for guidance can prevent adversarial generations within bigger models and generate sharper results. Furthermore, we implement an implicit voxel grid model to show how neural networks provide an additional layer of regularization, resulting in better geometrical structure and coherency of generated objects. Compared to prior work, we achieve more coherent results with higher memory efficiency and faster training speeds.
BibTeX
@inproceedings{lee2023understanding,
  author = {Han-Hung Lee and Angel X Chang},
  title = {{UNDERSTANDING PURE CLIP GUIDANCE FOR VOXEL GRID NERF MODELS}},
  booktitle = {International Conference on Learning Representations},
  year = {2023},
  url = {https://openreview.net/forum?id=3gZop22KWP},
}

Loading related content...

Figure