Aligning Simulation and Reality through Action Transformation

21 concepts11 papers1 proposal

Reinforcement learning (RL) agents thrive in simulated environments where data collection is cheap, fast, and entirely safe. However, deploying a policy trained in simulation onto physical hardware or real-world systems often fails due to the reality gap—the inevitable discrepancy between simulated physics and real-world dynamics.

Traditionally, researchers attempt to bridge this gap using System Identification (SysID), which involves tuning internal simulator parameters like friction coefficients, masses, or gear latencies to match reality. But SysID has severe limitations: many high-performance simulators are closed-source black boxes, exact physical constants are notoriously difficult to measure, and simplified physics engines often cannot model complex real-world phenomena regardless of how they are tuned.

Instead of modifying the simulator's internal physics code, what if we modify the actions we feed into it? This is the core intuition behind Grounded Action Transformation (GAT). By learning a functional mapping that warps control signals between domains, we can force a simulator to behave like the real world without altering a single line of its physics engine.

The Core Intuition: Changing Inputs, Not Physics

To understand why action transformation works, imagine driving a real car that has a loose steering wheel; you must turn the wheel 15 degrees to get a 10-degree turn at the wheels. If your simulator is perfectly calibrated to a tight steering wheel (where a 10-degree turn of the wheel equals a 10-degree turn at the wheels), the simulation will not match reality.

SysID would require you to open the simulator's source code and modify the steering joint geometry or gear ratios. Action transformation takes a simpler approach: it intercepts your steering command. If the policy wants to simulate what happens in the real world when turning the wheel 15 degrees, the action transformer intercepts that command and inputs a 10-degree turn to the simulator. Because a 10-degree input in the simulator produces the exact same physical trajectory as a 15-degree input in the real world, the simulator now perfectly mirrors reality.

As first proposed by Hanna et al. in Grounded Action Transformation for Robot Learning in Simulation and later detailed in Grounded action transformation for sim-to-real reinforcement learning, this behavioral alignment allows an RL agent to train in an environment that feels exactly like the real world.

The Dual-Model Mechanism

To automatically calculate this action mapping, GAT frameworks combine two distinct models: a real-world forward model and a simulator inverse model.

[Current State s] + [Intended Action a]
        │
        ▼
┌──────────────────────────────┐
│  Real-World Forward Model    │ ──► Predicts Real Next State s'
└──────────────────────────────┘
        │
        ▼
┌──────────────────────────────┐
│   Simulator Inverse Model    │ ──► Outputs Transformed Action a_sim
└──────────────────────────────┘
        │
        ▼
[Execute a_sim in Simulator] ──► Simulator transitions to s'
  1. The Forward Dynamics Model: Trained on a small dataset of real-world transitions, this model predicts what actually happens in reality. Given a current state $s$ and an intended action $a$, it outputs the expected next state $s'$ in the real world: $$\text{Forward}_{\text{real}}(s, a) \to s'$$

  2. The Inverse Simulator Model: This model understands the internal quirks of the simulator. It answers the question: What action must I input to the simulator to force a transition from state $s$ to state $s'$? $$\text{Inverse}_{\text{sim}}(s, s') \to a_{\text{sim}}$$

By chaining these two models together, we construct the action transformer. When the RL agent selects an action $a$ in state $s$, the system calculates: $$a_{\text{sim}} = \text{Inverse}_{\text{sim}}(s, \text{Forward}_{\text{real}}(s, a))$$

When $a_{\text{sim}}$ is executed inside the simulator, the resulting state transition matches the real-world transition $s'$. The simulator has been "grounded."

Overcoming Noise and Compounding Errors

While the basic dual-model mechanism is elegant, early implementations suffered from two primary challenges: environmental noise and compounding prediction errors.

Modeling Stochasticity

Real-world environments are rarely deterministic; friction changes, surfaces are uneven, and actuators slip. If a deterministic action transformer is used, the simulator remains idealized. To address this, Desai et al. introduced Stochastic Grounded Action Transformation (SGAT) in Stochastic Grounded Action Transformation for Robot Learning in Simulation. By forcing the forward model to output a probability distribution over next states, the inverse model learns to inject the exact same stochasticity into the simulator, teaching the RL agent to be robust to real-world noise.

Mitigating Compounding Errors via RL

Because the forward and inverse models are typically trained separately via supervised learning, small errors accumulate over long trajectories. If the forward model predicts a state that is physically impossible in the simulator, the inverse model fails.

To bypass this bottleneck, Karnan et al. proposed Reinforced Grounded Action Transformation (RGAT) in Reinforced Grounded Action Transformation for Sim-to-Real Transfer. Instead of chaining separate models, the authors framed the action transformer itself as an RL agent. The transformer receives a reward based on how closely the simulator's actual next state matches the observed real-world transition. Training the transformer end-to-end drastically reduces compounding errors and accommodates complex neural network policies.

Broadening the Paradigm

The concept of aligning domains by translating actions has proven highly versatile, expanding far beyond classical rigid-body robotics.

  • Soft Robotics: Sedal demonstrated in Soft Robots Learn to Crawl: Jointly Optimizing Design and Control with Sim-to-Real Transfer that a linear action mapping can align the complex, non-linear bending responses of soft actuators between simulation and physical prototypes, enabling zero-shot transfer.
  • Multi-Agent Traffic Control: In macro-level systems like traffic signal coordination, local changes cascade. Mei et al. introduced uncertainty-aware grounding (UGAT) to determine exactly when to trust action transformations under volatile real-world traffic conditions. To capture spatial dependencies, Turnau et al. developed JL-GAT, which incorporates joint state and action information from neighboring intersections into the decentralized grounding process.
  • Integrating Language Models: Recent iterations leverage Large Language Models (LLMs) to ground actions using qualitative human knowledge. For instance, Da et al. developed PromptGAT, using LLMs to reason about how weather conditions alter vehicle dynamics. Similarly, Wu et al. introduced HLMTrans to bridge domain gaps in spatial crowdsourcing by using LLMs to model human preferences and real-world traffic dynamics.
  • Cross-Task Meta-Learning: Action translation is not limited to sim-to-real transfer. Guo et al. developed MCAT, an action translator for Meta-RL. By mapping actions between different tasks based on functional state-transition equivalence, the framework allows an agent to instantly reuse high-level strategies in environments with entirely different physical dynamics.
  • Joint Visual and Dynamics Alignment: In Policy Transfer across Visual and Dynamics Domain Gaps via Iterative Grounding, Zhang et al. combined an action transformation layer with unsupervised visual domain adaptation (IDAPT), proving that an agent can overcome physical and visual domain gaps simultaneously.

By shifting the problem of domain alignment from modifying the environment to translating the control interface, action transformation provides a modular, data-efficient, and black-box-friendly solution to the reality gap.

Key Foundational Work

Reinforced Grounded Action Transformation for Sim-to-Real Transfer (2020) — contributed 4 concept elements to this direction.
Grounded Action Transformation for Robot Learning in Simulation (2017) — contributed 3 concept elements to this direction.
Stochastic Grounded Action Transformation for Robot Learning in Simulation (2020) — contributed 2 concept elements to this direction.
History & citationVersion 2 · 2026-10-04

Current · Permanent link

  1. Version 2 (latest) · 2026-10-04 · 11 papers
  2. Version 1 · 2026-10-04 · 17 papers
Cite this direction

Latest version (a living page):

@misc{lacuna_action_transformation_3ba604c5,
  title = {action transformation},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/action-transformation/rdr_3ba604c52b314255806f5e0ba18042a4},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 2, frozen text):

@misc{lacuna_action_transformation_3ba604c5_v2,
  title = {action transformation},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/action-transformation/rdr_3ba604c52b314255806f5e0ba18042a4/v/2},
  note = {Version 2, frozen 2026-10-04. Accessed 2026-11-19},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake