Aligning 2D Textures with Imperfect 3D Geometry

27 concepts28 papers1 proposal

When we look at a photorealistic 3D model, our eyes are easily fooled by high-frequency surface details—the grain of wood, the brickwork of a facade, or the fabric of clothing. However, generating these assets requires marrying two fundamentally different modalities: 2D pixel grids (photographs or generative outputs) and 3D geometric representations (meshes, point clouds, or volumetric primitives).

In an ideal world, projecting a 2D image onto a 3D shape is a straightforward mathematical operation. In reality, this process is highly sensitive to minor discrepancies. When the geometry is slightly coarse or the camera tracking drifts, the texture breaks.

The Geometry-Calibration Gap: Ghosting and Blurring

Why do small errors in a 3D reconstruction cause catastrophic visual failures? Consider what happens when we try to blend multiple photographs of an object onto its reconstructed 3D mesh. As observed by Vagia Tsiminaki et al. in High Resolution 3D Shape Texture from Multiple Videos and Yanping Fu et al. in Texture Mapping for 3D Reconstruction With RGB-D Sensor, camera trajectory drift and geometric inaccuracies introduce sub-pixel misalignments between different viewpoints.

When traditional texture mapping pipelines attempt to average or blend these overlapping images onto a single surface, the high-frequency details do not line up. If pixel $(x,y)$ from Camera A contains a sharp edge, but due to a tracking error, that same edge maps to $(x+2, y)$ in Camera B, blending them acts as a low-pass filter. This turns crisp details into a "mushy" blur. Alternatively, if the coordinate systems are severely out of sync, it creates ghosting or double-printing artifacts, where distinct features like lines appear duplicated on the mesh surface, as detailed by Kazushi Yoshida et al. in Registration of Range Image Using Texture of High-Resolution Color Images.

Camera A Projection:   |--Edge--|
Camera B Projection:       |--Edge--|
Blended Result:        |---Blurred/Ghosted Edge---|

To combat this, early approaches focused heavily on rigid alignment. For instance, Alejandro J. Troccoli et al. in A Shadow Based Method for Image to Model Registration utilized the sun's position to match real shadows in outdoor photographs with simulated shadows on 3D geometry, effectively "locking" the texture into place.

Yet, even with perfect camera calibration, a fundamental problem remains: the 3D geometry itself is often a simplified proxy of the real world. As Dana Cobzas et al. in Tracking and Rendering Using Dynamic Textures on Geometric Structure from Motion and Weidan Xiong et al. in DiffTex: Differentiable Texturing for Architectural Proxy Models point out, mapping static textures onto simplified proxy planes or coarse structures inevitably breaks the visual illusion of depth, causing broken lines and structural distortions. As further noted by Yixing Lao et al. in Less Gaussians, Texture More: 4K Feed-Forward Textured Splatting and Andrew Gilbert et al. in Volumetric Performance Capture from Minimal Camera Viewpoints, the fidelity of projected textures is highly sensitive to the precision of the underlying geometric primitives; even minor depth estimation errors cause significant visual degradation during projection.

The Perspective Dilemma: Stretching and Bleeding

The second major failure mode stems from the geometric relationship between the camera viewpoint and the surface normal. Imagine shining a flashlight straight at a wall: it forms a tight, bright circle. If you shine it at an extreme angle along the wall, the circle stretches into a long, faint oval.

The same thing happens when projecting a square pixel onto a surface that faces away from the camera (a grazing angle). Mathematically, as the angle between the camera's viewing ray and the surface normal approaches 90 degrees, the footprint of a single pixel projects over an increasingly large spatial area on the 3D mesh. This causes severe texture stretching and blurring, a challenge addressed by Dave Zhenyu Chen et al. in Text2Tex: Text-driven Texture Synthesis via Diffusion Models and Elad Richardson et al. in TEXTure: Text-Guided Texturing of 3D Shapes.

In urban reconstruction, standard oblique aerial photography frequently warps critical architectural features like windows and doors because the flight paths fail to align with the building facades, as shown by Junhua Kang et al. in AUTOMATIC TEXTURE RECONSTRUCTION OF 3D CITY MODEL FROM OBLIQUE IMAGES and Weidan Xiong et al. in Aerial Path Planning for Urban Geometry and Texture Co-Capture.

Furthermore, perspective shifts introduce occlusion boundaries. When projecting a 2D image onto a 3D scene, color information from a foreground object can easily "spill" or "bleed" onto the background surface behind it due to slight geometric mismatches, an issue studied by Qi Wang et al. in RoomTex: Texturing Compositional Indoor Scenes via Iterative Inpainting. In orthogonal representations like tri-planes, this manifests as spatial texture leakage, where visual data from one side of a thin object erroneously mirrors onto the opposite side, as identified by Zhengqin Li et al. in LIRM: Large Inverse Rendering Model for Progressive Reconstruction of Shape, Materials and View-dependent Radiance Fields.

Smarter Lifting: Differentiable Rendering and Depth-Aware Synthesis

To bridge the gap between 2D pixels and 3D shapes, modern frameworks have moved away from naive homography projections toward geometry-aware lifting techniques.

One powerful intuition is to replace standard image warping with ray-casting. As demonstrated by Wenzhao Tang et al. in Texture2LoD3: Enabling LoD3 Building Reconstruction With Panoramic Images, ray-casting explicitly accounts for the 3D structure of the scene, making the projection robust against non-planar surfaces and calibration inaccuracies.

To eliminate blurring and ghosting on simplified models, the authors of DiffTex: Differentiable Texturing for Architectural Proxy Models introduced a differentiable texturing approach. Instead of optimizing direct RGB values—which succumbs to averaging—they use differentiable rendering to optimize texel-level blending weights across multiple source photographs. This ensures that only the sharpest, most direct views contribute to the final texture, discarding data from steep, oblique angles.

Oblique View (Steep Angle)  --> Discarded / Low Weight (Avoids Stretching)
Direct View (Normal Angle)   --> High Weight (Preserves Sharp Details)

In the realm of generative texturing (such as text-to-3D), diffusion models are increasingly constrained by geometric priors. Rather than generating a flat 2D image and blindly lifting it—which Ji Hou notes can result in severe stretching and distortions in ControlRoom3D: Room Generation Using Semantic Proxy Rooms—frameworks utilize depth-aware or depth-to-image diffusion models.

By dynamically partitioning the mesh into viewpoints and using the underlying depth maps to guide the denoising process, methods like those proposed by Chen et al. and Richardson et al. ensure that the synthesized textures naturally respect the surface curvature. This prevents the generation of features that would stretch when viewed from another angle, creating a seamless, perspective-consistent texture map.

Key Foundational Work

Registration of Range Image Using Texture of High-Resolution Color Images (2002) — contributed 1 concept elements to this direction.
A Shadow Based Method for Image to Model Registration (2004) — contributed 1 concept elements to this direction.
Tracking and Rendering Using Dynamic Textures on Geometric Structure from Motion (2002) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_texture_geometry_misalignment_bc42fd97,
  title = {texture-geometry misalignment},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/texture-geometry-misalignment/rdr_bc42fd9762384fa7ad01bee1fbdb46db},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_texture_geometry_misalignment_bc42fd97_v1,
  title = {texture-geometry misalignment},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/texture-geometry-misalignment/rdr_bc42fd9762384fa7ad01bee1fbdb46db/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake