Neural Neural Textures Make Sim2Real Consistent

Ryan Burgert, Jinghuan Shang, Xiang Li, Michael S. RyooarXiv 2022 · DOI · OpenReview

Abstract

Training robotic policies in simulation offers safety and scalability, yet the "reality gap" remains a significant barrier to deployment. While unsupervised image-to-image translation can narrow this gap, standard methods often lack temporal and surface consistency, leading to visual artifacts like texture "crawling" as objects move. We introduce TRITON (Texture Recovering Image Translation Network), a framework that ensures visual consistency by representing object textures as implicit neural functions. By incorporating novel unprojection consistency and texture realism losses, TRITON maintains a stable appearance for objects across different poses and viewpoints, significantly improving the reliability of sim-to-real transfer.

Introduction: The Challenge of Surface Consistency

The transition from simulation to reality (Sim2Real) is often hindered by visual discrepancies. Unsupervised image-to-image translation techniques, such as CycleGAN, attempt to bridge this gap by mapping simulated frames to the real-world domain. However, these methods typically process frames independently or with limited temporal context, failing to ensure that a specific point on an object's surface retains the same appearance throughout a sequence. This lack of surface consistency results in visual artifacts that can confuse downstream robotic controllers.

Figure 1: Overview of the TRITON pipeline, showing the translation from simulated UV maps and labels to realistic, surface-consistent "fake" photos.

The TRITON Framework

TRITON addresses these inconsistencies by grounding the image translation process in the underlying 3D geometry of the scene.

1. Neural Neural Textures

A core innovation of TRITON is the use of Neural Neural Textures. Unlike traditional neural textures that rely on discrete raster grids, TRITON employs a continuous, implicit representation. A Multi-Layer Perceptron (MLP) with Fourier features takes 2D UV coordinates as input and outputs RGB values. This continuous representation is more stable during optimization and naturally produces smoother, less noisy textures than grid-based approaches.

2. UVL Scene Representation

Instead of raw simulated images, TRITON operates on "UVL" scenes. These scenes consist of channels representing texture coordinates ($u, v$) and object labels ($l$). This representation provides the network with explicit geometric information, allowing it to map textures directly onto object surfaces.

3. Consistency and Realism Losses

The system is trained using a bidirectional image-to-image translator ($T$) and a set of specialized losses designed to enforce geometric grounding:

  • Unprojection Consistency Loss ($\ell_{UC}$): This loss ensures that the appearance of a surface point is consistent across different views. It "unprojects" pixels from the generated fake photos back into the 3D texture space and penalizes variance in color for the same UV coordinate across a batch.
  • Texture Realism Loss ($\ell_{TR}$): This constraint forces the learned neural texture to match the final translated output, preventing the translator from overriding the geometric consistency provided by the neural textures.
Figure 4: Detailed architecture of TRITON, highlighting the bidirectional translation and the key consistency losses.

Experimental Evaluation

TRITON was evaluated across several datasets, including rigid objects (AlphabetCube-3, RobotFive), robotic arms (RobotPose-xArm), and deforming surfaces (AmericanFlag).

Key Results

  • Visual Fidelity: On the AlphabetCube-3 benchmark, TRITON achieved an LPIPS score of 0.286, significantly outperforming CycleGAN (0.450), indicating much higher perceptual similarity to real-world ground truth.
  • Downstream Performance: In robotic reaching tasks, policies trained on TRITON-translated images exhibited a mean distance error of 11.59, compared to 17.97 for CUT and 19.26 for CycleGAN. This demonstrates that surface consistency is crucial for robust policy transfer.
  • Stateless Consistency: Unlike video-to-video methods that require temporal memory or optical flow, TRITON achieves consistency statelessly. Because the texture is "baked" into the 3D geometry during training, consistency is a natural property of the model.

Conclusion and Future Work

TRITON provides a robust solution for surface-consistent Sim2Real translation by leveraging implicit neural textures and geometric constraints. While highly effective, the approach currently requires known 3D geometry and UV mappings, and it does not explicitly model complex light transport or transparency. Future research will explore extending these principles to scenes with unknown geometry and more complex material properties.

Summary updated

Abstract

Unpaired image translation algorithms can be used for sim2real tasks, but many fail to generate temporally consistent results. We present a new approach that combines differentiable rendering with image translation to achieve temporal consistency over indefinite timescales, using surface consistency losses and \emph{neural neural textures}. We call this algorithm TRITON (Texture Recovering Image Translation Network): an unsupervised, end-to-end, stateless sim2real algorithm that leverages the underlying 3D geometry of input scenes by generating realistic-looking learnable neural textures. By settling on a particular texture for the objects in a scene, we ensure consistency between frames statelessly. Unlike previous algorithms, TRITON is not limited to camera movements -- it can handle the movement of objects as well, making it useful for downstream tasks such as robotic manipulation.
BibTeX
@misc{burgert2022neural,
  author = {Ryan Burgert and Jinghuan Shang and Xiang Li and Michael S. Ryoo},
  title = {{Neural Neural Textures Make Sim2Real Consistent}},
  year = {2022},
  eprint = {2206.13500},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2206.13500},
}
Figure