Geometric Representations for Sim-to-Real Transfer

50 concepts41 papers1 proposal

The High-Entropy Trap of RGB Images

When training machine learning models in simulation to deploy them in the real world—a paradigm known as sim-to-real transfer—the primary obstacle is the "reality gap." If you train a robot or a computer vision model on raw RGB images from a simulator, the model invariably overfits to the specific lighting configurations, surface textures, and shadow patterns of that virtual environment. When deployed in the physical world, even minor variations in ambient light or material reflection cause catastrophic failure. RGB data is high-entropy and highly domain-sensitive, making direct transfer notoriously difficult.

However, there is a fundamental property of the physical world that simulators can replicate with near-perfect fidelity: geometry. A 3D CAD model or a simulated room has the exact same structural dimensions, spatial layouts, and surface boundaries as its real-world counterpart. By shifting the operational modality from visual appearance (RGB) to geometric structure (such as depth maps, surface normals, and point clouds), we can build a robust, domain-invariant bridge between simulation and reality.

Modality Abstraction: Operating Entirely in the Geometric Domain

The most direct way to leverage this insight is to completely strip away RGB data and force models to perceive the world through depth information alone. Because geometry remains consistent across domains, a policy trained purely on simulated depth maps can achieve zero-shot transfer to physical hardware.

Saenko et al. demonstrate this effect in [Adapting control policies from simulation to reality using a pairwise loss], showing that utilizing depth maps instead of RGB images minimizes sensitivity to lighting variations and surface textures, thereby simplifying the domain adaptation task. In complex robotic manipulation tasks, such as multi-fingered grasping, this geometric focus is crucial. Wu et al. show in [Pixel-Attentive Policy Gradient for Multi-Fingered Grasping in Cluttered Scenes] that relying solely on depth-based geometric information allows reinforcement learning policies to transfer with minimal performance loss, achieving high success rates on novel real-world objects.

This principle extends to contact-rich tasks and highly deformable objects like cloth, where visual textures change unpredictably during manipulation. Liang demonstrates in [Learning Preconditions of Hybrid Force-Velocity Controllers for Contact-Rich Manipulation] that training models on geometric point cloud data rather than RGB textures enhances zero-shot transfer from simulation to real-world environments. For cloth manipulation, Mo et al. in [Foldsformer: Learning Sequential Multi-Step Cloth Manipulation With Space-Time Attention] and Weng et al. in [FabricFlowNet: Bimanual Cloth Manipulation with a Flow-based Policy] both show that depth-only visual processing significantly narrows the sim-to-real gap by focusing on invariant geometric structures rather than surface appearance. Similarly, Liu et al. utilize depth-based heightmaps in [Hybrid Robotic Grasping With a Soft Multimodal Gripper and a Deep Multistage Learning Scheme] to ensure their manipulation policies remain resilient to appearance differences between synthetic and real-world objects.

Even in dynamic navigation tasks, geometry provides a stable abstraction. Savva et al. observe in [Habitat: A Platform for Embodied AI Research] that depth sensors provide a consistent structural signal that prevents embodied agents from overfitting to the high-entropy visual features found in color textures. This allows models to generalize across completely different architectural environments. In autonomous flight, the authors of [Learning High-Speed Flight in the Wild] use depth-based representations as a primary input to bypass traditional mapping and planning stages, enabling zero-shot transfer of high-speed quadrotor trajectories through unseen environments.

Cross-Modality Matching: Bridging RGB Images to 3D Geometry

While depth-only processing is powerful, many real-world applications are constrained to use standard RGB cameras. In these scenarios, geometry can still act as an intermediate bridge. Instead of directly aligning real RGB images with synthetic RGB images, models can be trained to match real RGB features against synthetic geometric representations.

For 3D pose estimation and object retrieval, Corona et al. show in [Pose Estimation for Objects with Rotational Symmetry] that matching real-world images against rendered depth maps rather than RGB textures enhances robustness to domain shifts, forcing the network to prioritize geometric features over surface appearance. Grabner et al. employ a similar strategy in [3D Pose Estimation and 3D Model Retrieval for Objects in the Wild], comparing real RGB images against rendered depth maps to ignore discrepancies in lighting and sensor noise. To eliminate manual annotation entirely, Rad et al. use unlabeled real RGB-D pairs alongside synthetic depth maps as a geometric bridge in [Domain Transfer for 3D Pose Estimation from Color Images without Manual Annotations], transferring pose knowledge seamlessly from the synthetic domain to the real world.

Alternatively, models can use depth or surface orientation as an auxiliary prediction target to ground their visual representations. In semantic segmentation, Vu et al. in [DADA: Depth-Aware Domain Adaptation in Semantic Segmentation] and Zhou et al. in [Geometry-Aware Network for Domain Adaptive Semantic Segmentation] demonstrate that forcing a network to predict depth in the target domain helps disentangle the underlying geometric structure of a scene from its superficial texture and lighting conditions.

Wang et al. extend this in [Cross-Modality Domain Adaptation for Freespace Detection: A Simple yet Effective Baseline] by using geometric surface normals as a primary feature, providing a stable anchor because properties like flatness remain perfectly consistent between simulated and real-world driving environments. For 3D reconstruction, Ye demonstrates in [Diffusion-Guided Reconstruction of Everyday Hand-Object Interaction Clips] that using geometric representations like surface normals and depth maps as priors effectively bridges the domain gap between synthetic training data and real-world video.

Mitigating the Residual Gap: Sensor Noise and Artifacts

Although geometry is highly invariant, a secondary domain gap exists: the discrepancy between the perfect depth maps generated by a simulator and the noisy, imperfect depth maps produced by physical sensors (such as LiDAR or structured-light cameras). Real depth sensors suffer from missing pixels, edge distortions, and hardware-specific noise patterns.

To close this residual gap, researchers employ two main strategies: geometric abstraction and high-fidelity sensor simulation.

The first strategy involves smoothing out the noise to maintain a unified representation. For example, the authors of [Learning a State Representation and Navigation in Cluttered and Dynamic Environments] apply morphological operations like dilation and hole filling to depth images, intentionally removing high-frequency noise and fine-grained details to make the model robust to physical sensor discrepancies. Similarly, Rizzoli et al. apply frequency-based style transfer to depth maps during pre-training in [Source-Free Domain Adaptation for RGB-D Semantic Segmentation with Vision Transformers], encouraging the model to ignore hardware-specific artifacts and focus purely on underlying structure.

The second strategy is to bring the simulation closer to reality by modeling the physics of the sensor itself. Bai et al. develop a pipeline in [Close the Sim2real Gap via Physically-based Structured Light Synthetic Data Simulation] that simulates the actual physics of structured-light cameras, rendering gray-code patterns and reconstructing depth maps with realistic sensor noise. By focusing on the geometric noise characteristics of the sensor rather than trying to fix the RGB textures, they achieve a minimal performance gap when transferring models to real-world industrial environments.

Key Foundational Work

3D Pose Estimation and 3D Model Retrieval for Objects in the Wild (2018) — contributed 2 concept elements to this direction.
Adapting control policies from simulation to reality using a pairwise loss (2018) — contributed 2 concept elements to this direction.
Object segmentation in depth maps with one user click and a synthetically trained fully convolutional network (2018) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_depth_based_sim_to_real_transfer_9e04b6d1,
  title = {depth-based sim-to-real transfer},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/depth-based-sim-to-real-transfer/rdr_9e04b6d117e34519aa00833f947aeb36},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_depth_based_sim_to_real_transfer_9e04b6d1_v1,
  title = {depth-based sim-to-real transfer},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/depth-based-sim-to-real-transfer/rdr_9e04b6d117e34519aa00833f947aeb36/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake