Enhancing Diffusion Models with 3D Perspective Geometry Constraints

Rishi Upadhyay, Howard Zhang, Yunhao Ba, Ethan Yang, Blake Gella, Sicheng Jiang, Alex Wong, Achuta Kadambi

Abstract

Modern latent diffusion models (LDMs) excel at generating high-fidelity imagery but frequently violate fundamental physical laws, particularly the rules of 3D perspective. This paper explores a novel geometric loss function designed to enforce perspective consistency during the image generation process. By integrating a differentiable perspective loss constraint ($L_{persp}$) that aligns image gradients with vanishing points, we demonstrate that generative models can produce physically plausible architecture and landscapes. Our results show that perspective-corrected synthetic data significantly improves the performance of downstream computer vision tasks, such as monocular depth estimation, and aligns more closely with human visual preferences.

Introduction

While text-to-image models like Stable Diffusion have reached a milestone in creative synthesis, they often struggle with structural integrity. A common artifact in AI-generated urban scenes is "wobbly" architecture, where parallel lines fail to converge at consistent vanishing points. This lack of an explicit geometric prior results in images that are aesthetically pleasing at a glance but physically impossible upon closer inspection.

For applications in autonomous driving and robotics, this is more than a visual glitch. Synthetic data is increasingly used to train perception systems, but if the training data lacks accurate 3D geometry, the resulting models fail to generalize to the real world. This research addresses the "Sim2Real" gap by forcing diffusion models to respect the mathematical constraints of perspective geometry.

Methodology

The core contribution of this work is the introduction of a perspective loss constraint ($L_{persp}$) into the fine-tuning pipeline of a latent diffusion model (specifically Stable Diffusion V2).

The Perspective Loss Framework

The proposed method utilizes a "sweep" mechanism to calculate geometric deviation. The algorithm first identifies or defines vanishing points within the scene. From these points, it projects rays across the image space. The loss function then evaluates the alignment between the generated image gradients (edges) and these rays.

Mathematically, the model is penalized when the "edge-like" regions of a generated image deviate from the rays converging at the vanishing points. This ensures that structural elements—such as the rooflines of buildings or the edges of a road—adhere to the laws of linear perspective.

Differentiable Implementation

To ensure the model can learn from these constraints, the entire pipeline is implemented as a differentiable process in PyTorch. This allows the geometric gradients to flow back through the neural network during training, directly guiding the weights of the diffusion model toward producing structurally sound outputs.

Experimental Setup and Datasets

The researchers utilized a variety of datasets to fine-tune and evaluate the model's performance across different domains:

  • HoliCity: A dataset comprising over 50,000 real-world images of London, complete with ground-truth vanishing points and CAD-derived layouts. This served as the primary training set for fine-tuning the perspective-aware diffusion model.
  • KITTI & DIODE: These standard benchmarks for outdoor scene understanding and autonomous driving were used to evaluate the utility of the generated images for downstream tasks.
  • Virtual KITTI 2 & SYNTHIA-AL: Used to generate large-scale synthetic datasets (exceeding 155,000 images) to benchmark the improvement in training other vision models.
  • Kaggle Landscapes/Animals: These datasets were used to ensure that the geometric fine-tuning did not result in "catastrophic forgetting" or a reduction in the model's general creative capabilities.

Key Insights

Geometry as a Regularizer

Perspective is not merely an aesthetic choice but a rigid mathematical constraint. By treating vanishing points as ground-truth anchors, the perspective loss acts as a powerful regularizer. It prunes the vast space of possible images the model might generate, steering the synthesis toward physical plausibility.

Closing the Sim2Real Gap

A significant hurdle in computer vision is the performance drop when models trained on synthetic data are deployed in real-world environments. This research demonstrates that by correcting the underlying geometry of synthetic imagery, we can create training data that more accurately reflects the physical world, thereby improving the robustness of downstream models.

Results

Human Subjective Evaluation

In head-to-head comparisons, human evaluators were asked to choose between images generated by the standard Stable Diffusion V2 and the perspective-constrained model. Evaluators preferred the perspective-accurate images 70% of the time, citing increased photo-realism and structural coherence.

Downstream Task Performance

The impact on monocular depth estimation was particularly pronounced. Models (such as DPT-Hybrid) trained on perspective-corrected synthetic data showed significant improvements when tested on the real-world KITTI benchmark:

  • 7.03% improvement in Root Mean Squared Error (RMSE).
  • 19.3% improvement in Square Relative error (SqRel).

Furthermore, the model demonstrated superior performance in image inpainting, maintaining the structural integrity of architectural features when filling in missing regions of an image.

Limitations and Future Work

Despite these advancements, several challenges remain. The current training method is dependent on datasets with known vanishing points, which are less abundant than the unlabelled "wild" data used for general-purpose models. Additionally, while perspective is addressed, other physical priors—such as consistent lighting, shadow casting, and semantic accuracy (e.g., ensuring a clock face has hands)—are still missing.

Future research will focus on developing similar differentiable constraints for lighting and complex 3D object relationships, moving toward a truly "physics-complete" generative framework.

Summary updated

BibTeX
@article{upadhyay2023enhancing,
  author = {Rishi Upadhyay and Howard Zhang and Yunhao Ba and Ethan Yang and Blake Gella and Sicheng Jiang and Alex Wong and Achuta Kadambi},
  title = {{Enhancing Diffusion Models with 3D Perspective Geometry Constraints}},
  journal = {ACM Trans. Graphics},
  year = {2023},
  url = {https://openreview.net/forum?id=MsT5Ln13Gt},
}
Figure