Mutual Alignment Transfer Learning

Markus Wulfmeier, Ingmar Posner, Pieter Abbeel

Mutual Alignment Transfer Learning

Abstract

Training robots for operation in the real world is a complex, time-consuming, and potentially expensive task. Despite the significant success of reinforcement learning (RL) in games and simulations, research in real robot applications has not matched this progress. While sample complexity can be reduced by training policies in simulation, these often perform sub-optimally on real platforms due to the "reality gap"—imperfect calibration of model dynamics. We present Mutual Alignment Transfer Learning (MATL), an approach supplemental to fine-tuning that harnesses parallel access to a simulator during training. MATL uses auxiliary rewards to guide real-world exploration based on simulation proficiency and vice versa. We demonstrate empirically that reciprocal alignment provides significant benefits, as the simulator adjusts its behavior to optimize for states commonly visited by the real-world agent.

Introduction

Reinforcement Learning has achieved remarkable success in environments where data is essentially infinite. However, applying RL to physical robots is notoriously difficult due to the high cost of real-world data and the prevalence of sparse rewards. While simulators provide a safe, fast training ground, discrepancies in friction, joint dampening, or contact dynamics mean that a policy perfected in simulation often fails on a real platform. Simply fine-tuning a pre-trained simulation policy often fails because the agent has "overfit" to the simulator's specific quirks, leading to poor exploration and local optima in the real world.

Methods: The MATL Framework

The core of MATL is an adversarial framework that treats transfer learning as a two-way street. Unlike standard one-way transfer (sim → real), MATL ensures that both environments align their state distributions.

Figure 1: The MATL architecture featuring reciprocal alignment between simulation and real-world agents via an adversarial discriminator.

As illustrated in Figure 1, the system utilizes two agents—one in simulation (source) and one on the robot (target)—both trained using Trust Region Policy Optimization (TRPO). A neural network discriminator ($D$) is trained to distinguish whether a sequence of states originated from the simulator or the real robot. The agents then receive auxiliary rewards ($r_{aux}$) for visiting states that "confuse" the discriminator.

This reciprocal alignment is the "secret sauce": by forcing the simulator to align with the robot, the simulated agent is pushed to find high-reward solutions that are actually reachable within the robot's real-world dynamics, rather than exploiting simulation-only physics exploits.

Experimental Evaluation

We evaluated MATL on several standard RL benchmarks, including Cartpole, Cartpole Swingup, and Reacher2D. A significant test involved "Cross-Engine Transfer," where the source and target environments used entirely different physics engines (MuJoCo vs. DART). This represented a high degree of model discrepancy designed to test the robustness of the alignment.

Key Results

Empirical results show that MATL, particularly when combined with fine-tuning (MATLf), reaches optimal performance significantly faster than independent training or simple fine-tuning.

Figure 6: Performance comparisons across Cartpole, Cartpole Swingup, and Reacher2D tasks, showing MATL's superior convergence and stability compared to baselines.

As shown in Figure 6, MATLf demonstrates superior sample efficiency. Key findings include:

  • Sparse Reward Navigation: In environments with sparse feedback, the adversarial auxiliary reward acts as a "dense" signal, rewarding the robot for matching the distribution of a successful simulator agent even before the task goal is reached.
  • Zero Environment Reward Success: Remarkably, MATL can guide a robot to solve a task even when the robot receives no task-specific reward at all, relying entirely on alignment with the simulator.
  • Bridging Physics Gaps: In MuJoCo-to-DART experiments, where standard fine-tuning often performed worse than training from scratch, MATL successfully bridged the gap and outperformed all baselines.

Discussion and Limitations

While MATL provides a robust mechanism for bridging the reality gap, it shares the instabilities common to GAN-based approaches. The weight of the alignment reward ($\lambda$) requires careful tuning; excessive weighting can lead the agent to prioritize "looking like the simulator" over actually solving the task. Furthermore, future work is needed to integrate explicit safety constraints to ensure that the exploration encouraged by the alignment reward does not lead to dangerous hardware configurations.

Summary updated

Abstract

Training robots for operation in the real world is a complex, time consuming and potentially expensive task. Despite significant success of reinforcement learning in games and simulations, research in real robot applications has not been able to match similar progress. While sample complexity can be reduced by training policies in simulation, these can perform sub-optimally on the real platform given imperfect calibration of model dynamics. We present an approach - supplemental to fine tuning on the real robot - to further benefit from parallel access to a simulator during training. The developed approach harnesses auxiliary rewards to guide the exploration for the real world agent based on the proficiency of the agent in simulation and vice versa. In this context, we demonstrate empirically that the reciprocal alignment for both agents provides further benefit as the agent in simulation can adjust to optimize its behaviour for states commonly visited by the real-world agent.
BibTeX
@inproceedings{wulfmeier2017mutual,
  author = {Markus Wulfmeier and Ingmar Posner and Pieter Abbeel},
  title = {{Mutual Alignment Transfer Learning}},
  booktitle = {Conference on Robot Learning},
  year = {2017},
  url = {https://openreview.net/forum?id=S8lBoIzezZ},
}
Figure