DAG: Depth-Aware Guidance with Denoising Diffusion Probabilistic Models

Gyeongnyeon Kim, Wooseok Jang, Gyuseong Lee, Susung Hong, Junyoung Seo, Seungryong Kim

Introduction: The Challenge of 3D Consistency

State-of-the-art generative models, particularly Denoising Diffusion Probabilistic Models (DDPMs), have achieved remarkable success in producing high-fidelity textures and semantic patterns. However, they often remain "novices of architecture." When generating complex indoor or outdoor scenes, these models frequently produce geometrically inconsistent layouts—walls that do not meet at right angles, furniture merging into floors, or perspective distortions that defy the laws of physics.

The fundamental issue is that standard guidance techniques, such as classifier-free guidance, prioritize semantic labels (e.g., "make a bedroom") over geometric configurations (e.g., "ensure the bed sits on the floor"). This paper introduces Depth-Aware Guidance (DAG), a framework designed to instill geometric awareness into diffusion models, ensuring that generated 2D pixels correspond to a coherent and plausible 3D space.

Methodology: Extracting Geometry from Noise

The DAG framework operates in two primary stages to bridge the gap between 2D generation and 3D structure without requiring a complete architectural overhaul of the diffusion model.

1. Label-Efficient Depth Estimation

Rather than training a separate, massive depth estimation model, the authors leverage the internal representations of the pretrained diffusion U-Net. They discovered that the U-Net’s features already encode significant geometric information. By training a shallow Multi-Layer Perceptron (MLP) on these frozen features using a limited amount of depth-labeled data, the system can predict depth maps for intermediate noisy images during the sampling process.

2. Dual Guidance Strategies

During the sampling phase, DAG employs two distinct gradient-based strategies to nudge the image toward better geometry:

  • Depth Consistency Guidance (DCG): Inspired by semi-supervised learning techniques like FixMatch, this method uses a "strong" branch (utilizing many U-Net layers) to create a pseudo-label that guides a "weak" branch (utilizing fewer layers). This internal alignment forces the model to resolve geometric ambiguities early in the denoising process.
  • Depth Prior Guidance (DPG): A separate, low-resolution diffusion model is trained specifically on depth maps. This "depth prior" acts as a critic, ensuring that the depth map extracted from the generated image aligns with the distribution of realistic 3D scenes.
The DAG architecture utilizes internal U-Net features and dual guidance branches (DCG and DPG) to ensure geometric consistency during sampling.

Key Insights: The U-Net as a Geometer

A striking finding of this research is that a U-Net trained solely to denoise images inherently learns monocular depth estimation. This suggests that geometric understanding is a byproduct of learning to represent natural images. To quantify this, the authors introduced dFID (Depth FID). While standard FID measures the realism of pixels, dFID measures the realism of the underlying 3D structure, providing a concrete metric for evaluating geometric plausibility.

Furthermore, the model utilizes self-conditioning via pseudo-labels. By treating its own "best guess" at depth as a target for its weaker internal representations, the model effectively self-corrects geometric errors as it denoises the image.

Results and Evaluation

The researchers evaluated DAG on the LSUN-Bedroom and LSUN-Church benchmarks. Since these datasets lack ground-truth depth, pseudo-labels from a pretrained Dense Prediction Transformer (DPT) were used for training the internal depth predictors.

Quantitative and Qualitative Improvements

On the LSUN-Bedroom dataset, DAG improved the dFID score from 15.71 (baseline) to 13.93, indicating significantly more realistic 3D layouts. Qualitatively, visualizations such as point clouds and surface normals demonstrate that DAG-generated images possess sharper boundaries and more logical perspective than those from unguided models.

Qualitative results showing generated images with room layout lines, estimated depth, and surface normals, highlighting the improved 3D coherence of DAG.

Downstream Utility

The study also demonstrated that DAG-generated images serve as superior training data. A separate depth estimator trained exclusively on synthetic images generated by DAG achieved an accuracy of 77.54%, compared to 72.55% for an estimator trained on standard diffusion-generated images. This suggests that DAG produces data that is not just visually pleasing, but structurally more accurate.

Limitations and Future Work

While DAG represents a significant step forward, it introduces certain trade-offs:

  • Computational Overhead: Guidance requires backpropagating through the U-Net to calculate gradients at every sampling step, increasing the time and memory required for generation.
  • Feature Selection: The choice of U-Net layers for the "strong" and "weak" branches is currently empirical; future work could involve automated selection methods.
  • Efficiency: Future iterations might explore more efficient sampling strategies or integrate depth-conditioning directly into the initial training phase, potentially reducing the need for post-hoc guidance.

Summary updated

Abstract

Generative models have recently undergone significant advancement due to the diffusion models. The success of these models can be often attributed to their use of guidance techniques, such as classifier or classifier-free guidance, which provide effective mechanisms to trade-off between fidelity and diversity. However, these methods are not capable of guiding a generated image to be aware of its geometric configuration, e.g., depth, which hinders their application to areas that require a certain level of depth awareness. To address this limitation, we propose a novel guidance method for diffusion models that uses estimated depth information derived from the rich intermediate representations of diffusion models. We first present label-efficient depth estimation framework using internal representations of diffusion models. Subsequently, we propose the incorporation of two guidance techniques based on pseudo-labeling and depth-domain diffusion prior during the sampling phase to self-condition the generated image using the estimated depth map. Experiments and comprehensive ablation studies demonstrate the effectiveness of our method in guiding the diffusion models towards the generation of geometrically plausible images.
BibTeX
@misc{kim2022dag,
  author = {Gyeongnyeon Kim and Wooseok Jang and Gyuseong Lee and Susung Hong and Junyoung Seo and Seungryong Kim},
  title = {{DAG: Depth-Aware Guidance with Denoising Diffusion Probabilistic Models}},
  year = {2022},
  eprint = {2212.08861},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2212.08861},
}
Figure