GeoDiffusion: A Training-Free Framework for Accurate 3D Geometric Conditioning in Image Generation

Phillip Mueller, Talip Uenlue, Sebastian Schmidt, Marcel Kollovieh, Jiajie Fan, Stephan Günnemann, Lars Mikelsons

Abstract

Modern generative models excel at creating visually stunning images but often lack the precision required for technical design tasks. GeoDiffusion introduces a training-free framework that incorporates 3D geometric constraints into 2D image generation. By utilizing simple 3D reference models to guide the diffusion process, GeoDiffusion enables mathematically accurate modifications—such as specific wheelbase extensions or height adjustments—while preserving object identity and visual fidelity.

Introduction

While models like Stable Diffusion have revolutionized content creation, they struggle with "technical" precision. Standard text prompts and 2D "dragging" interfaces often fail when a user requires exact geometric changes, such as making a car precisely 10% longer. The fundamental issue lies in the loss of 3D spatial information in 2D representations. Existing point-based editing methods frequently suffer from "drift" or "identity loss," where the object's appearance changes unintentionally during manipulation. Furthermore, 2D tools lack an inherent understanding of perspective, making it difficult to maintain structural integrity during deformations.

The GeoDiffusion Framework

GeoDiffusion bridges the gap between 3D modeling and 2D generation through a four-step, training-free pipeline.

Overview of the GeoDiffusion pipeline across different object classes, demonstrating the integration of 3D geometric priors and style prompts.

1. 3D Geometric Prior

The process begins with a class-specific 3D reference object (e.g., a generic car or chair) within a 3D environment. Users define "source points" and a "point translation function." When a geometric parameter is adjusted, the system calculates the precise 3D displacement and projects these points onto a 2D plane based on the camera viewpoint.

2. Style Transfer via PnP Diffusion

To transform the basic 3D render into a realistic image, GeoDiffusion employs Plug-and-Play (PnP) Diffusion. This technique allows the model to generate an image following a specific text prompt (e.g., "A Golden BMW SUV") while strictly adhering to the geometry of the reference render.

3. GeoDrag: Precise Point Manipulation

The core algorithmic innovation is GeoDrag, which enhances point-based editing through two key mechanisms:

  • Point Fixation: Standard dragging methods often experience drift because points "compete" for optimization. GeoDrag identifies points that have reached their target and "freezes" them, stabilizing the remaining optimization.
  • Copy-and-Paste Refinement: To accelerate convergence, latent features are directly copied from the source to the target destination once they are sufficiently close.

4. Image Refinement

An SDXL-refiner pass is used as a final step to polish the output, ensuring that any artifacts introduced during the dragging process are removed and the final image maintains high visual quality.

Key Insights

  • 3D Knowledge as a Structural Anchor: Complex 3D models are not necessary for high-quality results. Simple, generic 3D shapes provide sufficient perspective information to guide 2D diffusion models toward physically plausible deformations.
  • Stabilizing Optimization: Treating reached targets as anchors (Point Fixation) effectively solves the "tug-of-war" problem common in multi-point dragging tasks.
  • Practicality through Training-Free Design: By avoiding the need for task-specific adapters (like ControlNet), the framework is easily adaptable to new object categories with minimal setup (less than 10 seconds of LoRA fine-tuning per object).

Experimental Results

The authors evaluated GeoDiffusion using a new Geometry Guidance Benchmark (70 test cases across cars, bikes, airplanes, etc.) and the standard DragBench.

  • Accuracy: GeoDrag achieved a Mean Distance (MD) of 10.80 pixels on the Geometry Guidance Benchmark, an 8.24% improvement over the previous state-of-the-art, GoodDrag.
  • Efficiency: The method is 9.46% faster than GoodDrag despite its increased precision.
  • Identity Preservation: Qualitatively, GeoDiffusion maintains consistent color, texture, and part details even under significant geometric transformations.

Limitations and Future Work

The framework's performance is ultimately capped by the capabilities of the underlying pre-trained diffusion model. Additionally, while it excels at single-viewpoint consistency, it does not yet guarantee multi-view consistency. Future research could involve integrating GeoDiffusion into multi-view generative models to support full 3D design workflows.

Summary updated

BibTeX
@inproceedings{mueller2025geodiffusion,
  author = {Phillip Mueller and Talip Uenlue and Sebastian Schmidt and Marcel Kollovieh and Jiajie Fan and Stephan Günnemann and Lars Mikelsons},
  title = {{GeoDiffusion: A Training-Free Framework for Accurate 3D Geometric Conditioning in Image Generation}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2025},
  url = {https://openreview.net/forum?id=GOKwCNDc9a},
}
Figure