End-to-End Fine-Tuning of 3D Texture Generation using Differentiable Rewards

AmirHossein Zamani, Tianhao Xie, Amir Aghdam, Tiberiu Popa, Eugene Belilovsky

TL;DR

This paper introduces a novel end-to-end framework for fine-tuning 3D texture generation using differentiable rewards, allowing human preferences and task-specific requirements to directly influence the output. By back-propagating these rewards through the entire generation pipeline, the method ensures textures are not only visually appealing but also geometrically coherent with the 3D mesh.

What Problem Does This Paper Address?

Generating high-quality textures for 3D models is challenging. Current methods often rely on 2D text-to-image models, which lack an inherent understanding of the 3D object's geometry. This can lead to textures that don't fit the shape well, fail to meet user preferences, or are unsuitable for specific tasks. While reinforcement learning from human feedback (RLHF) has shown promise in other domains, its application to 3D texture generation is limited, with existing approaches either not directly fine-tuning the generative model or neglecting geometric constraints. This paper aims to bridge this gap by enabling precise control over 3D texture generation through differentiable rewards that are sensitive to the underlying 3D structure.

Methods

The proposed framework operates in two stages:

  1. Texture Generation: This stage leverages existing techniques like depth-aware text-to-image diffusion models and differentiable rendering. It renders the 3D object from multiple viewpoints, generating depth maps, normal maps, and UV coordinates. A diffusion model, guided by a text prompt and depth information (via ControlNet), then paints each view. A dynamic partitioning strategy is used to ensure visual consistency across views. Crucially, mathematical modifications are introduced to make this stage differentiable.
  2. Texture Reward Learning: This is the core innovation. The framework fine-tunes the pre-trained diffusion model by maximizing a differentiable reward function. Gradients from this reward are back-propagated through the entire texture generation pipeline, making the process inherently geometry-aware. This allows the model to learn from feedback that can be expressed in either the 2D texture space or directly on the 3D surface. The authors introduce five novel geometry-aware reward functions, including:
  3. Texture Features Emphasis: Encourages texture patterns to highlight surface curvature.
  4. Geometry-Texture Alignment: Aligns texture gradient vectors with principal curvature directions of the mesh.
  5. Symmetry-Aware Texture Generation: Promotes symmetric textures across mirrored UV coordinates.
  6. Camera-Pose-Aware Texture Generation: Learns optimal camera viewpoints to maximize aesthetic rewards, making the generation process robust to initial camera settings.
  7. Geometry-Aware Texture Colorization: (Detailed in Appendix F)

To handle the significant memory requirements of back-propagating through multiple diffusion steps and viewpoints, the authors employ low-rank adaptation (LoRA) and gradient checkpointing.

Datasets

The paper does not explicitly mention specific datasets used for training the foundational diffusion models. However, the evaluation involves rendering textured 3D meshes from multiple viewpoints. The authors also mention using the LAION aesthetic predictor [32] for the camera-pose-aware reward, which is trained on LAION-style data.

Key Insights and Ideas

  • Geometry-Awareness is Crucial: The core insight is that 3D texture generation needs to be intrinsically aware of the object's geometry. Simply applying 2D image generation techniques without considering the 3D structure leads to suboptimal results.
  • Differentiable Rewards as a Control Mechanism: By formulating human preferences and task requirements as differentiable rewards, the framework can directly guide the generative process. This is more powerful than post-hoc editing or separate optimization steps.
  • End-to-End Fine-Tuning: The ability to back-propagate gradients through the entire pipeline, including the diffusion model and rendering process, is key to achieving geometry-aware texture generation.
  • Novel Geometry-Aware Rewards: The introduction of specific rewards like "Geometry-Texture Alignment" and "Symmetry-Aware Texture Generation" provides concrete mechanisms for achieving desired geometric and structural properties in textures.
  • Robustness through Learned Camera Poses: The "Camera-Pose-Aware" reward demonstrates a clever way to make the texture generation process more robust by learning optimal viewpoints, rather than relying on fixed or manually chosen ones.

Key Results

  • Quantitative Performance: The proposed method consistently outperforms baselines like InTeX [33], TEXTure [29], Text2Tex [5], and Paint3D [40] across various automatic metrics (Aesthetic, ImageReward, HPSv2, PickScore, CLIPScore) and inference time. For example, the "Ours - Cam-Pose-Aware Reward" achieved the highest Aesthetic score (4.9328 ± 0.0271) compared to InTeX (4.7467 ± 0.0381).
  • Qualitative Improvements: Visual results (e.g., Figure 3) show that textures generated by the fine-tuned model align much better with the mesh's curvature directions and exhibit desired properties like symmetry.
  • User Preference: A user study with 40 participants showed that textures generated by the fine-tuned method were significantly preferred over those from the pre-fine-tuned baseline (InTeX) across various criteria, with preference rates as high as 98% for certain rewards.

Limitations and Future Work

  • Computational Cost: While optimizations like LoRA and gradient checkpointing are used, the process still requires significant computational resources, making training challenging in typical academic environments.
  • Generalizability: The paper focuses on texture generation. The authors suggest that the framework is broadly applicable and extensible to tasks beyond texture generation, envisioning its application to general 3D content creation, including joint optimization of geometry and texture.
  • Reward Design: The effectiveness of the framework relies on well-designed differentiable rewards. While they introduce several novel ones, designing effective rewards for complex human preferences remains an ongoing challenge.

Summary updated

Abstract

While recent 3D generative models can produce high-quality texture images, they often fail to capture human preferences or meet task-specific requirements. Moreover, a core challenge in the 3D texture generation domain is that most existing approaches rely on repeated calls to 2D text-to-image generative models, which lack an inherent understanding of the 3D structure of the input 3D mesh object. To alleviate these issues, we propose an end-to-end differentiable, reinforcement-learning-free framework that embeds human feedback, expressed as differentiable reward functions, directly into the 3D texture synthesis pipeline. By back-propagating preference signals through both geometric and appearance modules of the proposed framework, our method generates textures that respect the 3D geometry structure and align with desired criteria. To demonstrate its versatility, we introduce three novel geometry-aware reward functions, which offer a more controllable and interpretable pathway for creating high-quality 3D content from natural language. By conducting qualitative, quantitative, and user-preference evaluations against state-of-the-art methods, we demonstrate that our proposed strategy consistently outperforms existing approaches. Our implementation code is publicly available at: https://github.com/AHHHZ975/Differentiable-Texture-Learning.
BibTeX
@inproceedings{zamani2026end,
  author = {AmirHossein Zamani and Tianhao Xie and Amir Aghdam and Tiberiu Popa and Eugene Belilovsky},
  title = {{End-to-End Fine-Tuning of 3D Texture Generation using Differentiable Rewards}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2026},
  doi = {10.1109/wacv61042.2026.00025},
}
Figure