Abstract
Recent advancements in Neural Fields (NFs) have enabled the learning of 3D representations from class-specific single-view image collections. However, these methods often fail to preserve high-frequency details and struggle to disentangle appearance from geometry, limiting their utility for tasks like texture transfer and editing. We introduce TEGLO (Textured EG3D-GLO), a framework for learning high-fidelity 3D representations from in-the-wild image collections. By creating a dense correspondence mapping to a 2D canonical space, TEGLO achieves near-perfect reconstruction (≥ 74 dB PSNR at $1024^2$ resolution) and enables seamless texture editing and transfer without requiring meshes with shared topology.
The Challenge: Spectral Bias and Disentanglement
Reconstructing high-fidelity 3D models from a single photograph is a significant challenge in computer vision. Standard Multi-Layer Perceptrons (MLPs) used in Neural Radiance Fields (NeRFs) suffer from spectral bias, naturally favoring low-frequency signals. This results in the loss of fine details such as skin pores, hair strands, or intricate jewelry.
Furthermore, existing methods often entangle geometry and appearance. Without a clear separation, tasks like texture transfer—applying the skin texture of one individual to the facial structure of another—become nearly impossible without expensive 3D scans or manual intervention.
Methodology: The TEGLO Pipeline
TEGLO addresses these issues through a two-stage approach:
- 3D Representation Learning: The model utilizes a tri-plane representation combined with Generative Latent Optimization (GLO). Unlike GANs, GLO optimizes a specific latent code for each image to ensure accurate reconstruction, focusing primarily on capturing the underlying 3D geometry.
- Dense Correspondence Mapping: TEGLO learns to map every 3D surface point to a 2D canonical coordinate space. This "unwrapping" process creates a learned UV map from 2D images without explicit 3D supervision.
Pixel-to-Texture Mapping: A key innovation is mapping input image pixels directly onto the learned texture space. By treating the input image as the primary source of high-frequency data rather than a target for generative synthesis, TEGLO achieves unprecedented sharpness. For novel views, the system queries this map using a K-d tree and Natural Neighbor Interpolation (NNI) to maintain 3D consistency.
Key Results and Editing Capabilities
TEGLO sets a new benchmark for fidelity in single-view 3D reconstruction. On the CelebA-HQ dataset, it achieves a PSNR of 89.5 dB for training images and 86.2 dB for test images at $256^2$ resolution, significantly outperforming previous state-of-the-art methods like LoLNeRF (~29 dB). Even at megapixel resolutions ($1024^2$), it maintains a high PSNR of 74.7 dB, preserving minute details.
The learned canonical mapping enables robust texture editing. As shown in the figure above, edits made in the 2D space (such as adding text) wrap correctly around the 3D model and remain consistent across different camera angles for various object classes, including humans, cats, and cars.
Limitations and Future Directions
While TEGLO achieves high fidelity, it relies on a multi-stage pipeline rather than an end-to-end formulation. Additionally, because it maps pixels from a single view, occluded regions (e.g., the back of a head) require interpolation to fill "holes," which may not always generate plausible new data. Future work aims to explore end-to-end training and the integration of generative models like StyleGANv2 to synthesize textures for unseen surfaces.
This summary is based on the research paper "TEGLO: High Fidelity Canonical Texture Mapping from Single-View Images".
Summary updated