TEGLO: High Fidelity Canonical Texture Mapping from Single-View Images

Vishal Vinod, Tanmay Shah, Dmitry Lagun

Abstract

TEGLO (Textured EG3D-GLO) is a novel framework that achieves near-perfect 3D reconstruction from a single "in-the-wild" image by disentangling texture from geometry. By mapping input pixels directly into a 2D canonical texture space, it preserves high-frequency details and enables seamless texture editing and transfer without requiring 3D meshes or expensive fine-tuning.

Introduction

While Neural Radiance Fields (NeRFs) have revolutionized 3D scene representation, applying them to "in-the-wild" collections of single-view images (like a dataset of random faces) remains difficult. Current methods face three main hurdles: the loss of high-frequency details due to the low-pass filtering effect of standard MLPs, the entanglement of appearance and geometry, and the heavy computational cost of per-image optimization. TEGLO aims to provide a way to reconstruct high-resolution, 3D-consistent objects from single views while keeping texture and geometry strictly separate for downstream editing.

Methods

The authors propose a two-stage pipeline. In the first stage, a conditional NeRF is trained using a tri-plane representation and Generative Latent Optimization (GLO). This "auto-decoder" setup allows the model to reconstruct images at arbitrary resolutions without being constrained by a discriminator's fixed resolution.

The second stage introduces dense correspondence learning, the core innovation of the framework. The model learns to map 3D surface points from the NeRF onto a 2D canonical coordinate space. A Texture Mapper "unwraps" input pixels onto this map, and Natural Neighbor Interpolation (NNI) is used to fill gaps caused by occlusions, effectively creating a high-resolution texture atlas.

Key Insights

A core innovation of TEGLO is the realization that for reconstruction, the input pixels themselves are the ground truth. By mapping them directly to a texture space, the model bypasses the "spectral bias" of MLPs that usually blurs fine details like skin pores or wrinkles. This creates a universal "atlas" for a given object class, making texture transfer and editing as simple as modifying a 2D image.

High-fidelity texture mapping allows for precise, view-consistent edits like virtual tattoos.

Key Results

TEGLO achieves unprecedented fidelity, with a reconstruction PSNR of $\ge$ 74 dB at $1024^2$ resolution, compared to previous state-of-the-art methods that hover around 26-29 dB. In tests on ShapeNet-Cars, it significantly outperformed baselines in multi-view consistency, achieving a PSNR of 67.52 compared to the next best at 30.48. Furthermore, the model demonstrates robust zero-shot editing capabilities, allowing for the addition of "virtual tattoos" or face paint that renders perfectly across all 3D views.

TEGLO results across various datasets, demonstrating multi-view consistency and zero-shot texture editing for faces, cats, and cars.

Conclusion

TEGLO represents a significant step forward in high-fidelity 3D reconstruction from single images. By disentangling texture and geometry through a canonical mapping, it enables high-resolution editing and consistent multi-view rendering without the need for complex 3D data. Future work will aim to integrate the pipeline into an end-to-end formulation and improve the handling of environmental lighting and occluded regions.

Summary updated

Abstract

Recent work in Neural Fields (NFs) learn 3D representations from class-specific single view image collections. However, they are unable to reconstruct the input data preserving high-frequency details. Further, these methods do not disentangle appearance from geometry and hence are not suitable for tasks such as texture transfer and editing. In this work, we propose TEGLO (Textured EG3D-GLO) for learning 3D representations from single view in-the-wild image collections for a given class of objects. We accomplish this by training a conditional Neural Radiance Field (NeRF) without any explicit 3D supervision. We equip our method with editing capabilities by creating a dense correspondence mapping to a 2D canonical space. We demonstrate that such mapping enables texture transfer and texture editing without requiring meshes with shared topology. Our key insight is that by mapping the input image pixels onto the texture space we can achieve near perfect reconstruction (>= 74 dB PSNR at 1024^2 resolution). Our formulation allows for high quality 3D consistent novel view synthesis with high-frequency details at megapixel image resolution.
BibTeX
@inproceedings{vinod2024teglo,
  author = {Vishal Vinod and Tanmay Shah and Dmitry Lagun},
  title = {{TEGLO: High Fidelity Canonical Texture Mapping from Single-View Images}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2024},
  doi = {10.1109/wacv57701.2024.00355},
}
Figure