Geometry-Aware Preference Learning for 3D Texture Generation

AmirHossein Zamani, Tianhao Xie, Amir Aghdam, Tiberiu Popa, Eugene Belilovsky

Generating high-quality textures for 3D models has come a long way, thanks to powerful 2D text-to-image diffusion models. However, there’s a persistent "uncanny valley" in 3D texturing: textures often look like 2D stickers slapped onto a 3D shape, ignoring the underlying geometry, symmetry, or human aesthetic preferences.

A new paper from researchers at Mila and Concordia University, titled "End-to-End Fine-Tuning of 3D Texture Generation using Differentiable Rewards," introduces a framework to fix this. By making the entire texturing pipeline differentiable, they can fine-tune generative models to respect the "bones" of a 3D mesh.

TL;DR

The authors present an end-to-end differentiable, reinforcement-learning-free framework that fine-tunes 3D texture generators using specific geometric rewards. By back-propagating signals through both rendering and appearance modules, the system produces textures that naturally align with a mesh’s curvature, symmetry, and aesthetic potential.

What Problem Does This Paper Address?

Most current 3D texturing methods rely on "lifting" 2D generative models (like Stable Diffusion) into 3D. They essentially "paint" the object from multiple viewpoints. The problem is that these 2D models have no inherent understanding of 3D structure.

This leads to several issues:

  1. Geometric Misalignment: Patterns (like tiger stripes) might ignore the flow of the muscles or limbs.
  2. Lack of Control: It is difficult to enforce specific requirements, like "make the edges look worn" or "ensure the left and right sides are perfectly symmetrical."
  3. Human Preference Gap: Standard models are trained on massive datasets but aren't easily "steered" toward specific aesthetic goals or task-specific needs without expensive manual prompting.

Methods

The core innovation is a two-stage, end-to-end differentiable pipeline:

  1. Texture Generation: They start with a depth-aware text-to-texture pipeline (building on works like InTeX). To make this step differentiable, they replaced "hard" binary operations (like discrete masks used to decide which parts of a mesh to paint) with "soft," continuous approximations using sigmoid functions.
  2. Texture Reward Learning: This is the "fine-tuning" phase. Instead of using complex Reinforcement Learning (RL), which is often unstable, they use a direct optimization approach. They define Differentiable Reward Functions and back-propagate the reward signal directly into the weights of the diffusion model (specifically using LoRA to keep memory requirements sane).

The Five Novel Rewards:

  • Texture Features Emphasis: Encourages higher contrast and detail in high-curvature areas (e.g., making the "cracks" in a brick pattern align with the folds of a mesh).
  • Geometry-Texture Alignment: Uses principal curvature directions to align texture gradients. This makes patterns "flow" along the surface.
  • Symmetry-Aware Reward: Uses PCA to find the mesh's symmetry plane and rewards textures that are consistent across that plane.
  • Camera-Pose-Aware Reward: Automatically learns the best camera angles to view the object to maximize an "aesthetic score" (using the LAION aesthetic predictor).
  • Geometry-Aware Colorization: Maps "warm" colors to high-curvature areas and "cool" colors to flat areas.

Datasets

The researchers evaluated their method using:

  • Standard 3D Meshes: Common benchmarks like the Stanford Bunny, a cow, a dragon, and various human head models.
  • Text Prompts: Sourced from datasets like LAION-5B and Pick-a-Pic to test a wide variety of styles (e.g., "a bunny with orange and black tiger stripes," "a wooden rabbit").
  • Evaluation Metrics: They used several "proxy" models for human preference, including ImageReward, HPSv2, PickScore, and CLIPScore.

Key Insights and Ideas

  • Differentiability is Key: By making the rendering and masking steps differentiable, the model can "learn" how its 2D painting choices affect the final 3D result.
  • RL-Free Optimization: The authors show that if your reward function is differentiable, you don't need the complexity of Reinforcement Learning from Human Feedback (RLHF). Simple gradient descent on the reward function is faster and more stable.
  • Curvature as a Guide: One of the "aha" moments is realizing that the mathematical "flow" of a surface (principal curvature) can be used as a direct supervisory signal for the "flow" of a texture.

Key Results

  • Quantitative Superiority: The fine-tuned models consistently outperformed the baseline (InTeX) across all metrics. For example, on the Aesthetic Score, their method achieved ~5.03 compared to the baseline's ~4.74.
  • User Preference: In a user study with 40 participants, the fine-tuned textures were preferred over the baseline in 88% to 98% of cases, depending on the reward type.
  • Memory Efficiency: By using LoRA and gradient checkpointing, they reduced the required GPU memory from an impossible 300GB down to a manageable 20GB, allowing the model to be trained on consumer-grade hardware like an RTX 4090.

Limitations and Future Work

  • Symmetry Failures: The symmetry reward relies on PCA to find a symmetry plane. If a mesh is highly irregular or has an uneven vertex distribution, PCA can pick the wrong plane, leading to "lopsided" textures.
  • Camera Initialization: The camera-pose optimization can get stuck in "local minima" if the initial camera positions are extremely poor (e.g., pointing away from the object).
  • Future Directions: The authors suggest moving toward joint optimization, where the geometry of the mesh and the texture are optimized simultaneously to create even more cohesive 3D assets.

This work marks a significant step toward 3D generative tools that don't just "hallucinate" textures, but actually understand the shapes they are decorating.

Summary updated

Abstract

Recent advances in 3D generative models have achieved impressive results but 3D contents generated by these models may not align with subjective human preferences or task-specific criteria. Moreover, a core challenge in the 3D texture generation domain remains: most existing approaches rely on repeated calls to 2D text-to-image generative models, which lack an inherent understanding of the 3D structure of the input 3D mesh object. To address this, we propose an end-to-end differentiable preference learning framework that back-propagates human preferences, represented by differentiable reward functions, through the entire 3D generative pipeline, making the process inherently geometry-aware. We demonstrate the effectiveness of our framework using four proposed novel geometry-aware reward functions, offering a more controllable and interpretable pathway for high-quality 3D content creation from natural language.
BibTeX
@inproceedings{zamani2026geometry,
  author = {AmirHossein Zamani and Tianhao Xie and Amir Aghdam and Tiberiu Popa and Eugene Belilovsky},
  title = {{Geometry-Aware Preference Learning for 3D Texture Generation}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2026},
  doi = {10.1109/wacv61042.2026.00025},
}
Figure