Enhancing Diffusion Models with 3D Perspective Geometry Constraints

Rishi Upadhyay, Howard Zhang, Yunhao Ba, Ethan Yang, Blake Gella, Sicheng Jiang, Alex Wong, Achuta Kadambi

TL;DR

Researchers have developed a novel geometric loss function that forces latent diffusion models to respect the laws of 3D perspective. By enforcing consistent vanishing points during image generation, the resulting models produce more realistic images and significantly better synthetic training data for downstream tasks like monocular depth estimation.

What Problem Does This Paper Address?

While modern text-to-image models like Stable Diffusion produce stunning visuals, they often fail at "basic" physics. If you look closely at an AI-generated city street, you’ll often find "wobbly" architecture: parallel lines that don't converge correctly, curved "straight" edges, and inconsistent vanishing points.

This isn't just an aesthetic issue. Perspective is a fundamental physical constraint that dictates object size, location, and depth. Because current models lack an explicit geometric prior, the synthetic data they generate is often "physically impossible," making it poor training material for computer vision systems (like those in self-driving cars) that rely on accurate 3D geometry to navigate the real world.

Methods

The authors introduce a perspective loss constraint ($L_{persp}$) into the training process of a latent diffusion model (specifically building on Stable Diffusion V2).

  • The "Sweep" Mechanism: To calculate the loss, the algorithm identifies vanishing points in an image. It then "sweeps" rays out from these points across the image.
  • Gradient Alignment: The model calculates the image gradients (edges) along these rays. The goal of the loss function is to ensure that the "edge-like" regions of the generated image align perfectly with the rays converging at the vanishing points.
  • Fine-Tuning: They fine-tuned the model on a dataset that provides ground-truth vanishing point data. During training, the model is penalized if its generated edges deviate from the mathematically correct perspective lines.
  • Differentiable Pipeline: Crucially, the entire process is implemented in PyTorch and is fully differentiable, allowing the geometric constraints to guide the neural network's weights directly.

Datasets

  • HoliCity: Used for fine-tuning the diffusion model. It contains over 50,000 real-world images of London with ground-truth vanishing points and CAD-derived layouts.
  • KITTI & DIODE: Used to evaluate downstream performance. These are standard benchmarks for autonomous driving and outdoor scene understanding.
  • Virtual KITTI 2 & SYNTHIA-AL: Used to generate large-scale synthetic datasets (155k+ images) to test how well the perspective-corrected images train other models.
  • Kaggle Landscapes/Animals: Used to verify that the model's creative "breadth" wasn't ruined by fine-tuning on cityscapes.

Key Insights and Ideas

  • Geometry as a Regularizer: The most important "aha" moment is that perspective isn't just an artistic choice; it's a mathematical constraint. By treating vanishing points as a "ground truth" anchor, we can prune the "gamut of possible images" the model might generate, forcing it toward physical plausibility.
  • Bridging the Sim2Real Gap: One of the biggest hurdles in AI is that models trained on synthetic data often fail in the real world. This paper shows that by fixing the geometry of synthetic data, we can significantly close this gap.
  • Human Preference for Physics: Interestingly, even when humans aren't consciously looking for vanishing points, they overwhelmingly prefer images that follow perspective rules. Perspective accuracy is a "hidden" component of what we perceive as "photo-realism."

Key Results

  • Human Subjective Trials: In head-to-head comparisons, human evaluators preferred images from the perspective-constrained model over the standard Stable Diffusion V2 model 70% of the time.
  • Downstream Performance: Monocular depth estimation models (like DPT-Hybrid) trained on this perspective-accurate synthetic data saw massive improvements when tested on real-world KITTI data:
  • 7.03% improvement in Root Mean Squared Error (RMSE).
  • 19.3% improvement in Square Relative error (SqRel).
  • Inpainting: The model proved significantly better at "filling in" missing parts of images (inpainting) while maintaining the structural integrity of the surrounding architecture.

Limitations and Future Work

  • Vanishing Point Dependency: The current training method requires a dataset with known vanishing points. While these can be estimated for many scenes, it limits the training data compared to the "wild" internet-scale datasets used by base models.
  • Missing Physical Priors: While perspective is fixed, the model still struggles with other physical laws. For example, the authors note that a generated "Big Ben" might have perfect perspective but still lack clock hands (semantic errors) or have inconsistent lighting/shadows.
  • Computational Cost: Generating these high-quality images is still relatively slow (~3 seconds per image on an RTX 3090), which limits the speed at which massive synthetic datasets can be produced.
  • Future Directions: The authors suggest that similar constraints could be developed for lighting, shadows, and more complex 3D relationships to eventually create a "physics-complete" generative model.

Summary updated

Abstract

While perspective is a well-studied topic in art, it is generally taken for granted in images. However, for the recent wave of high-quality image synthesis methods such as latent diffusion models, perspective accuracy is not an explicit requirement. Since these methods are capable of outputting a wide gamut of possible images, it is difficult for these synthesized images to adhere to the principles of linear perspective. We introduce a novel geometric constraint in the training process of generative models to enforce perspective accuracy. We show that outputs of models trained with this constraint both appear more realistic and improve performance of downstream models trained on generated images. Subjective human trials show that images generated with latent diffusion models trained with our constraint are preferred over images from the Stable Diffusion V2 model 70% of the time. SOTA monocular depth estimation models such as DPT and PixelFormer, fine-tuned on our images, outperform the original models trained on real images by up to 7.03% in RMSE and 19.3% in SqRel on the KITTI test set for zero-shot transfer.
BibTeX
@article{upadhyay2023enhancing,
  author = {Rishi Upadhyay and Howard Zhang and Yunhao Ba and Ethan Yang and Blake Gella and Sicheng Jiang and Alex Wong and Achuta Kadambi},
  title = {{Enhancing Diffusion Models with 3D Perspective Geometry Constraints}},
  journal = {ACM Trans. Graphics},
  year = {2023},
  url = {https://openreview.net/forum?id=MsT5Ln13Gt},
}
Figure