TL;DR
CaliTex introduces "geometry-calibrated attention" to solve the persistent problem of seams and visual inconsistencies in 3D texture generation. By explicitly aligning the model's attention mechanism with the underlying 3D structure—using semantic part grouping and geometry-conditioned routing—it produces high-fidelity, view-consistent textures that significantly outperform current open-source and commercial baselines.
Addressing the Challenge of Attention Ambiguity
Generating high-quality textures for 3D models is a holy grail for gaming and film, but current diffusion-based systems suffer from cross-view inconsistency. When a model "paints" a 3D object from multiple angles, the textures often fail to align perfectly. This results in visible seams, blurring, or "geometric confusion" (e.g., the model accidentally paints the left arm's texture onto the right arm because they look similar in 2D).
The authors identify the root cause as attention ambiguity. Standard Transformer-based models use "full attention," where every pixel (token) can look at every other token across all views and modalities. While powerful for 2D images, this unstructured approach allows the model to ignore 3D spatial constraints, leading to textures that look great from one angle but fall apart when projected back onto a 3D mesh.
Methods: Geometry-Calibrated Attention
CaliTex addresses this by "calibrating" the attention mechanism to be geometry-aware. The framework uses a two-stage Diffusion Transformer (DiT) architecture: a Single-View DiT to capture local details and a Multi-View DiT to ensure global consistency. The core innovations lie in two new attention modules:
- Part-Aligned Attention (PAA): The researchers use a tool called PartField to decompose the 3D mesh into semantic parts (e.g., head, torso, limbs). PAA restricts cross-view attention so that tokens representing a specific part (like the "left boot") only attend to tokens of that same part in other views. This prevents the model from confusing geometrically similar but distinct regions.
- Condition-Routed Attention (CRA): In standard models, noise tokens often "overfit" by directly copying patterns from a reference image, ignoring the 3D shape. CRA forces information to flow through a dual-pathway: one branch aligns the reference image with the geometry, and the other injects these geometry-aware features into the generation process. This ensures the texture actually "fits" the 3D surface.
Key Insights and Ideas
- Consistency as an Inherent Behavior: Rather than treating consistency as a post-processing "fix" or an optimization constraint, CaliTex makes geometric coherence an inherent part of the network's architecture.
- Breaking Symmetry Confusion: By using semantic part labels to guide attention, the model solves the "Janus problem" (where a model might put a face on both the front and back of a head) and symmetry-induced errors.
- Geometry as the Mediator: The core philosophy of Condition-Routed Attention is that appearance information should never reach the final output without first being "vetted" by the geometry condition. This prevents the "sticker-like" look where a 2D image is simply slapped onto a 3D shape.
Key Results
CaliTex achieved top-tier results across both quantitative metrics and human evaluation. It achieved the lowest FID (157.8) and LPIPS (0.2508), indicating the highest visual fidelity and similarity to ground truth. It also led in CLIP-FID, proving superior semantic alignment.
Visual comparisons show that CaliTex produces much sharper details and virtually eliminates the "seam" artifacts common in models like UniTEX or Hunyuan3D. In a study with 25 participants, CaliTex was rated significantly higher than all competitors in Overall Quality (4.53/5), Geometric Alignment (4.47/5), and Multi-view Consistency (4.52/5).
Limitations and Future Work
The effectiveness of Part-Aligned Attention relies on the quality of the initial 3D mesh segmentation. If the automated segmentation (via PartField) fails on a highly unconventional shape, the attention grouping might be less effective. Additionally, the multi-stage DiT architecture is more computationally intensive than simpler U-Net-based diffusion models, requiring roughly 600 GPU hours for training.
Future directions include expanding this geometry-aware attention design into more complex material properties, such as PBR maps (roughness, metallic, etc.) beyond just albedo textures, moving closer to professional-grade 3D asset creation.
Summary updated