Abstract
Although recent 3D-native generators have made great progress in synthesizing reliable geometry, they still fall short in achieving realistic appearances. A key obstacle lies in the lack of diverse and high-quality real-world 3D assets with rich texture details, since capturing such data is intrinsically difficult due to the diverse scales of scenes, non-rigid motions of objects, and the limited precision of 3D scanners. We introduce Photo3D, a framework for advancing photorealistic 3D generation, which is driven by the image data generated by the GPT-4o-Image model. Considering that the generated images can distort 3D structures due to their lack of multi-view consistency, we design a structure-aligned multi-view synthesis pipeline and construct a detail-enhanced multi-view dataset paired with 3D geometry. Building on it, we present a realistic detail enhancement scheme that leverages perceptual feature adaptation and semantic structure matching to enforce appearance consistency with realistic details while preserving the structural consistency with the 3D-native geometry. Our scheme is general to different 3D-native generators, and we present dedicated training strategies to facilitate the optimization of geometry-texture coupled and decoupled 3D-native generation paradigms. Experiments demonstrate that Photo3D generalizes well across diverse 3D-native generation paradigms and achieves state-of-the-art photorealistic 3D generation performance.
Introduction
In the last two years, 3D generation has shifted from slow, iterative optimization methods to "3D-native" models like Trellis or Hunyuan3D that can generate objects in seconds. However, these models face a significant "realism gap." Most 3D-native models are trained on synthetic datasets like Objaverse, which provide clean geometry but often result in textures that look like plastic toys or cartoons.
Capturing high-quality, real-world 3D data remains a bottleneck due to lighting complexities, occlusions, and scanner limitations. Conversely, 2D image generators (like Flux or GPT-4o) have mastered photorealism but lack inherent 3D consistency. Photo3D bridges this gap by using the photorealism of 2D models to teach 3D models how to render realistic materials and details.
The Photo3D Pipeline
The Photo3D framework introduces a three-stage pipeline designed to inject realism into 3D generation while maintaining structural integrity. The process begins with generating a 3D asset from a text prompt using a standard model. These assets are then rendered into multi-view images and refined using a 2D foundation model.
1. Structure-Aligned Multi-View Synthesis
To create a training signal, the system renders orthogonal views of a 3D asset and feeds them into GPT-4o-Image with a specific refinement prompt. GPT-4o acts as a "master painter," adding fine-grained textures—such as fur, skin, or weathered wood—while attempting to keep the original structure intact.
2. Relaxed Detail Enhancement
A core challenge is that 2D models often introduce slight inconsistencies between views (e.g., a texture pattern shifting slightly). Using standard pixel-wise loss functions in this scenario would lead to blurry results. Photo3D solves this with a "relaxed" supervision scheme:
- Perceptual Adaptation ($L_{adapt}$): A CLIP-based loss applied to random crops, forcing the model to learn the style and texture of the real world without being anchored to specific, potentially inconsistent pixels.
- Semantic Structure Matching ($L_{match}$): Utilizing DINOv3 features to ensure that semantically important components (like eyes or wheels) remain aligned with the underlying 3D geometry.
3. Paradigm-Specific Training
The framework is designed to be "plug-and-play," supporting both coupled models (where geometry and texture are learned together) and decoupled models (where texture is applied to a pre-generated mesh).
Results and Dataset
The authors constructed the Photo3D-MV dataset, containing 10,000 objects across 373 categories. Each entry pairs 3D geometry with GPT-4o-enhanced photorealistic multi-view images. This dataset allows 3D-native models to learn from a much wider variety of realistic textures than previously possible.
Quantitative evaluations on the ImageNet benchmark show that Photo3D significantly improves realism and aesthetic scores. Specifically, when built on the Trellis architecture, it improved the MANIQA realism score from 0.438 to 0.470 and the aesthetic score from 4.682 to 4.927. Qualitative results demonstrate a massive jump in "materiality," where objects previously resembling smooth plastic now exhibit complex specularities and realistic surface imperfections.
Conclusion
Photo3D demonstrates that 2D foundation models can serve as powerful "super-resolution" teachers for 3D generation. By moving away from strict pixel-wise supervision toward perceptual and semantic matching, the framework enables the generation of 3D assets with movie-quality textures. Future work will likely involve integrating even more advanced 2D and video generators to further push the boundaries of photorealistic 3D synthesis.
Summary updated