DreamAlign: Dynamic Text-to-3D Optimization with Human Preference Alignment

Gaofeng Liu, Zhiyuan Ma, Tao FangAAAI 2025 · DOI

Abstract

DreamAlign addresses the gap between technical correctness and aesthetic quality in text-to-3D generation. By utilizing the HP3D dataset and the D-3DPO optimization algorithm, it aligns 3D models with human preferences, improving lighting, perspective, and consistency.

Introduction

While text-to-3D generation has advanced rapidly through 2D lifting techniques like Score Distillation Sampling (SDS), these models often suffer from a "human preference dilemma." Even when a model follows a prompt, the resulting 3D object might have jarring flaws: asymmetric lighting, awkward perspective offsets, or "Janus-faced" artifacts (multiple heads/faces). Standard 2D diffusion priors used for 3D lifting aren't inherently tuned for aesthetics or human-centric preferences. DreamAlign addresses these challenges by introducing a more efficient, reward-free alignment pipeline.

Methodology

DreamAlign employs a two-stage approach to integrate human preferences into the 3D generation process:

1. DreamAlign-2D: Preference Learning

Instead of a complex RLHF setup, the framework utilizes Direct 3D Preference Optimization (D-3DPO). This reward-free method fine-tunes a multi-view diffusion model (MVDream) using LoRA adapters. It directly optimizes the model to increase the probability of generating preferred images over non-preferred ones based on a log-ratio objective, eliminating the need for expensive reward models.

2. DreamAlign-3D: Preference-Aware Lifting

During 3D generation, the standard SDS loss is replaced with Preference Contrastive Feedback (PCF) loss. This loss calculates gradients by contrasting the preference-aligned DreamAlign-2D model against the original base model. This contrastive approach "pushes" the 3D representation toward preferred aesthetic distributions while "pulling" it away from the flaws inherent in the base model.

Multi-view consistency and aesthetic alignment in a generated teddy bear.

Datasets

The study introduces HP3D (Human Preference 3D), a curated dataset of 1,300 3D entries with human preference annotations. Samples were selected from Cap3D, captioned using the multimodal LLM LLava, and ranked by human experts. Additionally, the Pick-a-Pic dataset was used to supplement 2D prior training, while Cap3D provided initial 3D assets.

Results and Discussion

DreamAlign demonstrates significant improvements in both quantitative and qualitative evaluations.

Quantitative Performance

DreamAlign-2D outperformed MVDream and Stable Diffusion v2.1 across multiple metrics, including CLIP text-image alignment and ImageReward. In a user study involving computer vision researchers, 68.6% of participants preferred DreamAlign’s results over baseline methods, and 60.3% found them more consistent with text descriptions.

Qualitative Quality

The assets generated show a marked reduction in artifacts. Common issues like "floating" shadows or inconsistent color patterns are resolved, resulting in clean, well-lit models that strictly adhere to prompt adjectives like "glossy" or "vibrant."

Qualitative results of DreamAlign across various prompts including characters, statues, and objects.

Key Insights

  • The Power of Contrast: Using the difference between a "good" and "reference" model focuses optimization specifically on attributes humans prefer, such as consistent shading and realistic materials.
  • Aesthetic Consistency: Consistency in 3D is not just geometric; it requires lighting and materials that make sense from every angle.

Limitations and Future Work

Current limitations include the relatively small scale of the HP3D dataset compared to 2D foundation datasets. Additionally, the SDS-based optimization remains slow, requiring approximately 10,000 steps per prompt. Future work will focus on scaling preference data and extending alignment to complex, multi-object scenes.

Summary updated

BibTeX
@inproceedings{liu2025dreamalign,
  author = {Gaofeng Liu and Zhiyuan Ma and Tao Fang},
  title = {{DreamAlign: Dynamic Text-to-3D Optimization with Human Preference Alignment}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2025},
  doi = {10.1609/aaai.v39i5.32577},
}
Figure