Mutual Alignment Transfer Learning
Abstract
Training robots for operation in the real world is a complex, time-consuming, and potentially expensive task. Despite the significant success of reinforcement learning (RL) in games and simulations, research in real robot applications has not matched this progress. While sample complexity can be reduced by training policies in simulation, these often perform sub-optimally on real platforms due to the "reality gap"—imperfect calibration of model dynamics. We present Mutual Alignment Transfer Learning (MATL), an approach supplemental to fine-tuning that harnesses parallel access to a simulator during training. MATL uses auxiliary rewards to guide real-world exploration based on simulation proficiency and vice versa. We demonstrate empirically that reciprocal alignment provides significant benefits, as the simulator adjusts its behavior to optimize for states commonly visited by the real-world agent.
Introduction
Reinforcement Learning has achieved remarkable success in environments where data is essentially infinite. However, applying RL to physical robots is notoriously difficult due to the high cost of real-world data and the prevalence of sparse rewards. While simulators provide a safe, fast training ground, discrepancies in friction, joint dampening, or contact dynamics mean that a policy perfected in simulation often fails on a real platform. Simply fine-tuning a pre-trained simulation policy often fails because the agent has "overfit" to the simulator's specific quirks, leading to poor exploration and local optima in the real world.
Methods: The MATL Framework
The core of MATL is an adversarial framework that treats transfer learning as a two-way street. Unlike standard one-way transfer (sim → real), MATL ensures that both environments align their state distributions.
As illustrated in Figure 1, the system utilizes two agents—one in simulation (source) and one on the robot (target)—both trained using Trust Region Policy Optimization (TRPO). A neural network discriminator ($D$) is trained to distinguish whether a sequence of states originated from the simulator or the real robot. The agents then receive auxiliary rewards ($r_{aux}$) for visiting states that "confuse" the discriminator.
This reciprocal alignment is the "secret sauce": by forcing the simulator to align with the robot, the simulated agent is pushed to find high-reward solutions that are actually reachable within the robot's real-world dynamics, rather than exploiting simulation-only physics exploits.
Experimental Evaluation
We evaluated MATL on several standard RL benchmarks, including Cartpole, Cartpole Swingup, and Reacher2D. A significant test involved "Cross-Engine Transfer," where the source and target environments used entirely different physics engines (MuJoCo vs. DART). This represented a high degree of model discrepancy designed to test the robustness of the alignment.
Key Results
Empirical results show that MATL, particularly when combined with fine-tuning (MATLf), reaches optimal performance significantly faster than independent training or simple fine-tuning.
As shown in Figure 6, MATLf demonstrates superior sample efficiency. Key findings include:
- Sparse Reward Navigation: In environments with sparse feedback, the adversarial auxiliary reward acts as a "dense" signal, rewarding the robot for matching the distribution of a successful simulator agent even before the task goal is reached.
- Zero Environment Reward Success: Remarkably, MATL can guide a robot to solve a task even when the robot receives no task-specific reward at all, relying entirely on alignment with the simulator.
- Bridging Physics Gaps: In MuJoCo-to-DART experiments, where standard fine-tuning often performed worse than training from scratch, MATL successfully bridged the gap and outperformed all baselines.
Discussion and Limitations
While MATL provides a robust mechanism for bridging the reality gap, it shares the instabilities common to GAN-based approaches. The weight of the alignment reward ($\lambda$) requires careful tuning; excessive weighting can lead the agent to prioritize "looking like the simulator" over actually solving the task. Furthermore, future work is needed to integrate explicit safety constraints to ensure that the exploration encouraged by the alignment reward does not lead to dangerous hardware configurations.
Summary updated