Abstract
Training robotic policies in simulation offers safety and scalability, yet the "reality gap" remains a significant barrier to deployment. While unsupervised image-to-image translation can narrow this gap, standard methods often lack temporal and surface consistency, leading to visual artifacts like texture "crawling" as objects move. We introduce TRITON (Texture Recovering Image Translation Network), a framework that ensures visual consistency by representing object textures as implicit neural functions. By incorporating novel unprojection consistency and texture realism losses, TRITON maintains a stable appearance for objects across different poses and viewpoints, significantly improving the reliability of sim-to-real transfer.
Introduction: The Challenge of Surface Consistency
The transition from simulation to reality (Sim2Real) is often hindered by visual discrepancies. Unsupervised image-to-image translation techniques, such as CycleGAN, attempt to bridge this gap by mapping simulated frames to the real-world domain. However, these methods typically process frames independently or with limited temporal context, failing to ensure that a specific point on an object's surface retains the same appearance throughout a sequence. This lack of surface consistency results in visual artifacts that can confuse downstream robotic controllers.
The TRITON Framework
TRITON addresses these inconsistencies by grounding the image translation process in the underlying 3D geometry of the scene.
1. Neural Neural Textures
A core innovation of TRITON is the use of Neural Neural Textures. Unlike traditional neural textures that rely on discrete raster grids, TRITON employs a continuous, implicit representation. A Multi-Layer Perceptron (MLP) with Fourier features takes 2D UV coordinates as input and outputs RGB values. This continuous representation is more stable during optimization and naturally produces smoother, less noisy textures than grid-based approaches.
2. UVL Scene Representation
Instead of raw simulated images, TRITON operates on "UVL" scenes. These scenes consist of channels representing texture coordinates ($u, v$) and object labels ($l$). This representation provides the network with explicit geometric information, allowing it to map textures directly onto object surfaces.
3. Consistency and Realism Losses
The system is trained using a bidirectional image-to-image translator ($T$) and a set of specialized losses designed to enforce geometric grounding:
- Unprojection Consistency Loss ($\ell_{UC}$): This loss ensures that the appearance of a surface point is consistent across different views. It "unprojects" pixels from the generated fake photos back into the 3D texture space and penalizes variance in color for the same UV coordinate across a batch.
- Texture Realism Loss ($\ell_{TR}$): This constraint forces the learned neural texture to match the final translated output, preventing the translator from overriding the geometric consistency provided by the neural textures.
Experimental Evaluation
TRITON was evaluated across several datasets, including rigid objects (AlphabetCube-3, RobotFive), robotic arms (RobotPose-xArm), and deforming surfaces (AmericanFlag).
Key Results
- Visual Fidelity: On the AlphabetCube-3 benchmark, TRITON achieved an LPIPS score of 0.286, significantly outperforming CycleGAN (0.450), indicating much higher perceptual similarity to real-world ground truth.
- Downstream Performance: In robotic reaching tasks, policies trained on TRITON-translated images exhibited a mean distance error of 11.59, compared to 17.97 for CUT and 19.26 for CycleGAN. This demonstrates that surface consistency is crucial for robust policy transfer.
- Stateless Consistency: Unlike video-to-video methods that require temporal memory or optical flow, TRITON achieves consistency statelessly. Because the texture is "baked" into the 3D geometry during training, consistency is a natural property of the model.
Conclusion and Future Work
TRITON provides a robust solution for surface-consistent Sim2Real translation by leveraging implicit neural textures and geometric constraints. While highly effective, the approach currently requires known 3D geometry and UV mappings, and it does not explicitly model complex light transport or transparency. Future research will explore extending these principles to scenes with unknown geometry and more complex material properties.
Summary updated