Texture Completion in UV Space

This direction is no longer present in the current build. Its last version stays readable and citable here.

Version 1 of 134 papers41 conceptsfrozen 2026-10-04

When reconstructing a 3D object from a single 2D image, the most significant hurdle is "self-occlusion." If a camera captures a person from the front, the pixels representing their back are simply missing. Traditional image-to-image translation attempts to "hallucinate" these missing parts in the 2D frame, but this often leads to geometric distortions because the network doesn't understand the underlying 3D structure.

A more robust approach, pioneered by researchers like Jiankang Deng and Stefanos Zafeiriou in UV-GAN, shifts the problem from 2D image space to UV coordinate space. By "unwrapping" a 3D model into a flat 2D texture map (a UV map), researchers can treat 3D appearance as a standard image completion task that is inherently pose-invariant.

The UV Map as a Canonical Canvas

To understand why this works, imagine peeling an orange and laying the skin flat on a table. No matter how you rotate the orange before peeling it, the "map" of the skin remains a consistent 2D representation.

In 3D vision, we map 2D pixels from an input image onto a 3D morphable model (for faces) or a parametric mesh like SMPL (for humans). This creates a "partial" UV map where only the visible parts of the object are filled with color. The occluded parts—the back of the head or the side of the torso—appear as empty black holes.

As Natalia Neverova and Iasonas Kokkinos demonstrate in Dense Pose Transfer, performing texture completion on this flattened map is more effective than 2D inpainting. Because the UV space is "canonical"—meaning a specific coordinate always corresponds to the same anatomical point (e.g., the tip of the nose)—the neural network doesn't have to learn how to handle different camera angles. It only needs to learn the statistical patterns of what a complete "skin" looks like.

Hallucinating Missing Surfaces

Once the partial texture is mapped to UV space, generative models are used to "fill in the blanks."

  1. Generative Adversarial Networks (GANs): In UV-GAN, a generator learns to synthesize the missing regions of the facial UV map. By training on a large dataset of complete 3D scans, such as the MimicME database introduced by Athanasios Papaioannou, the network learns to maintain identity and symmetry. If it sees a left eye, it knows how to synthesize a matching right eye on the other side of the UV map.
  2. Diffusion Models: More recent work, such as Paint3D by Bin Fu and Seed3D by Fan Yang, uses diffusion-based inpainters. These models are often conditioned on mesh coordinates to ensure that the generated textures are seamless and align perfectly with the 3D geometry, avoiding the "seams" that occur when textures are simply projected from different angles.

Maintaining Global Consistency

A key challenge in UV completion is ensuring that the "hallucinated" back of an object matches the "real" front. Chang Yu and colleagues addressed this in the Appearance Reconstruction Model (ARM) by using neural networks with a global receptive field. By looking at the entire UV map at once, the model can ensure that the fabric pattern on the back of a shirt matches the logo on the front.

Similarly, in the context of virtual try-on systems, Sahib Majithia and Avinash Sharma showed in Robust 3D Garment Digitization that inpainting directly within the UV panels of a garment ensures that the reconstruction is "simulation-ready." Because the texture is completed on the 2D pattern used to build the 3D garment, the resulting model maintains structural integrity when the character moves.

Beyond Simple Color: PBR and Materials

The evolution of this field has moved from completing simple RGB colors to generating full Physically Based Rendering (PBR) materials. Models like Material Anything by Tengfei Wang use UV-space refinement to generate maps for roughness, metallicity, and normals.

By refining these properties in UV space, the system can resolve artifacts like baked-in shadows or highlights that would otherwise "break" the illusion of 3D when the lighting changes. This allows a 3D asset generated from a single photo to be placed in any virtual environment and react realistically to new light sources, a process Chang Yu refers to as creating "relightable" 3D generation.

Papers in this version

History & citationVersion 1 · 2026-10-04

No longer present in the current build · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_texture_completion_in_uv_space_f47b0ad7,
  title = {Texture Completion in UV Space},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/texture-completion-in-uv-space/rdr_f47b0ad702a94e4ba4a3c00d4ffe4e06},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-22},
}

This exact version (version 1, frozen text):

@misc{lacuna_texture_completion_in_uv_space_f47b0ad7_v1,
  title = {Texture Completion in UV Space},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/texture-completion-in-uv-space/rdr_f47b0ad702a94e4ba4a3c00d4ffe4e06/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-22},
}