Geometric Abstraction for Sim-to-Real Policy Transfer

51 concepts40 papers1 proposal

Deep reinforcement learning (RL) and imitation learning excel in simulated environments where a robot can experience millions of trials without physical wear and tear. However, deploying these learned policies onto physical hardware often fails due to the reality gap—the discrepancy between simulated environments and the messy, unpredictable real world.

Traditionally, researchers have attempted to bridge this gap using photorealistic rendering or extensive domain randomization, forcing the simulator to generate endless variations of lighting, textures, and colors. The hope is that the policy will become invariant to visual noise. However, this approach is computationally expensive and struggles to account for every real-world edge case.

A powerful alternative paradigm has emerged: geometric abstraction. Instead of trying to make the simulation look like reality, this approach makes both simulation and reality look like abstract geometry. Simulators are exceptionally precise at computing 3D spatial structures, and modern physical sensors (such as depth cameras and LiDAR) can capture these exact same structures in the real world. By stripping away non-essential visual features—like color, texture, and lighting—and reducing inputs to pure geometric representations, the domain gap is drastically narrowed. The policy learns to reason about where things are and how they are shaped, rather than what they look like.


Point Clouds as a Universal Geometric Bridge

In robotic manipulation, 3D point clouds serve as a highly effective intermediate representation. Because point clouds directly encode spatial geometry, they are fundamentally immune to visual domain shifts like lighting or texture changes.

For closed-loop 6D robotic grasping, Wang et al. demonstrated that utilizing segmented 3D point clouds as the primary input representation facilitates robust generalization to unseen objects and dynamic scenarios, such as human-to-robot handovers. By focusing the policy on spatial relationships and object shapes, the model bypasses the visual discrepancies that typically impede image-based methods. Similarly, Qin et al. showed that relying on 3D point clouds rather than RGB images enables multi-fingered robotic hands to master dexterous manipulation tasks—like opening doors and grasping novel objects—generalizing seamlessly across different lighting conditions and object textures.

This geometric consistency is powerful enough to enable zero-shot transfer, where a policy is deployed directly to the real world without any real-world fine-tuning. For example, Varley et al. developed a transformer-based architecture for a two-armed robot that uses 3D point clouds to predict grasp poses, maintaining high performance levels on physical hardware without requiring a single frame of physical training data.

[Raw Visual Scene] ----(Depth Sensor / Segmentation)----> [3D Point Cloud] ----> [Policy Network] ----> [Robot Actions]
                                                                 ^
                                                    Same representation in Sim & Real

Geometric abstraction also simplifies the process of learning from human demonstrations. Wang et al. utilized 3D point clouds to remove human-specific visual artifacts from motion capture data, aligning the demonstrations into a robot-centric frame. This allowed physical robots to learn complex, bimanual dexterous tasks directly from human movements.


Low-Dimensional Projections for Navigation and Flight

While dense 3D point clouds are ideal for manipulation, high-speed navigation and agile flight require rapid inference where processing dense 3D data can introduce computational overhead. In these domains, researchers compress environmental geometry into sparse, low-dimensional projections or local maps.

For mapless mobile robot navigation, Paolo et al. reduced high-resolution sensor inputs into sparse, low-dimensional vectors. This abstraction forces the agent to learn navigation policies based on core spatial geometry, filtering out environment-specific noise and facilitating direct deployment on physical hardware. Similarly, Chen et al. processed raw 3D sensor data into simplified 2D local costmaps before feeding it into a neural network, allowing a recurrent network to "remember" obstacles that pass out of its immediate sight while minimizing the reality gap.

In autonomous flight and marine navigation, this intermediate representation acts as a critical filter against simulation artifacts. Kaufmann et al. achieved zero-shot transfer of high-speed acrobatic drone maneuvers by replacing raw visual inputs with abstracted geometric feature tracks. For unmanned surface vehicles navigating lake shores, Richard et al. demonstrated that transforming raw sensor data into a simplified geometric projection or local map filters out sensor noise and provides a consistent input space across simulated and physical environments.


Distilling Tasks into Topological and Keypoint Logic

Beyond raw coordinate maps, geometric abstraction can take the form of high-level topological constraints, keypoint distributions, or task-relevant features. By distilling the environment into its logical geometric components, the control policy is completely decoupled from the rendering engine.

In visual odometry, Messikommer et al. trained reinforcement learning agents on abstract geometric representations—such as keypoint distributions and map statistics—rather than raw pixel data. This abstraction ensures that the learned control logic remains invariant to visual domain shifts between synthetic training environments and real-world sensor data.

Task Domain Raw Input Abstracted Geometric Representation Benefit
6D Grasping RGB-D Video Segmented 3D Point Clouds Invariance to lighting, color, and background textures
Drone Racing Camera Pixels Task-Relevant Gate Edges / Feature Tracks Minimizes computational overhead; focuses on critical features
Needle Threading Tactile Images Center of Mass Coordinates of Segmented Masks Removes visual discrepancies between simulated & real sensors
Visual Odometry Video Stream Keypoint Distributions & Map Statistics Decouples control logic from specific sensor hardware

For highly precise, contact-rich tasks like robotic needle-threading, Yu et al. abstracted raw vision-based tactile sensor data into geometric coordinates, such as the center of mass of segmented masks. This intermediate representation removes visual discrepancies between simulated and real-world sensors, allowing policies trained in physics engines to generalize to physical hardware.

This principle scales even to training-free paradigms. Yin et al. decomposed natural language instructions into a graph of spatial constraints, proving that systems based on abstract geometric logic rather than learned visual features demonstrate high cross-domain generalization when transitioning from simulators to physical robot platforms.


Bypassing the Contact Physics Gap for Deformable Objects

Simulating the exact contact physics of deformable objects—such as cloth, ropes, or cables—is notoriously difficult because their high degrees of freedom create a massive search space. However, geometric abstraction allows models to reason about the state and forces of these objects without needing a perfect physical simulation model.

Held et al. trained models to estimate contact forces by observing deformable object interactions using voxelized geometric point clouds and relative distances. By prioritizing spatial geometry over visual appearance, the system successfully generalized from synthetic physics simulations to real-world robotic hardware. To align simulated deformable behaviors with reality, the authors of DiffCloud utilized point clouds as a robust intermediate representation, providing a common format for both raw depth sensor data and simulated meshes, thereby avoiding complex vertex-to-pixel correspondences.

For manipulating cables and ropes, Pecyna et al. trained control policies on distilled geometric features, such as relative angles and positions. This abstraction decouples the control logic from specific sensor hardware, making the system more adaptable to different environments and facilitating a seamless transfer from simulation to reality.

Ultimately, geometric abstraction shifts the burden of sim-to-real transfer away from high-fidelity rendering engines and onto structural filtering. By forcing the robot to view the world through the invariant lens of geometry, the boundaries between simulation and reality begin to dissolve.

Key Foundational Work

Deep Reinforcement Learning for Audio-Visual Gaze Control (2018) — contributed 2 concept elements to this direction.
Learning 6-DoF Grasping and Pick-Place Using Attention Focus (2018) — contributed 1 concept elements to this direction.
Virtual-to-real Deep Reinforcement Learning: Continuous Control of Mobile Robots for Mapless Navigation (2017) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_geometric_sim_to_real_transfer_09684a80,
  title = {geometric sim-to-real transfer},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/geometric-sim-to-real-transfer/rdr_09684a8054154a049f1fa72d64de3399},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-21},
}

This exact version (version 1, frozen text):

@misc{lacuna_geometric_sim_to_real_transfer_09684a80_v1,
  title = {geometric sim-to-real transfer},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/geometric-sim-to-real-transfer/rdr_09684a8054154a049f1fa72d64de3399/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-21},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake