MULTIPOLAR: Multi-Source Policy Aggregation for Transfer Reinforcement Learning between Diverse Environmental Dynamics

Mohammadamin Barekatain, Ryo Yonetani, Masashi Hamaya

Abstract

Transfer reinforcement learning (RL) aims at improving the learning efficiency of an agent by exploiting knowledge from other source agents trained on relevant tasks. However, it remains challenging to transfer knowledge between different environmental dynamics without having access to the source environments. In this work, we explore a new challenge in transfer RL, where only a set of source policies collected under diverse unknown dynamics is available for learning a target task efficiently. To address this problem, the proposed approach, MULTI-source POLicy AggRegation (MULTIPOLAR), comprises two key techniques. We learn to aggregate the actions provided by the source policies adaptively to maximize the target task performance. Meanwhile, we learn an auxiliary network that predicts residuals around the aggregated actions, which ensures the target policy's expressiveness even when some of the source policies perform poorly. We demonstrated the effectiveness of MULTIPOLAR through an extensive experimental evaluation across six simulated environments ranging from classic control problems to challenging robotics simulations, under both continuous and discrete action spaces.

Introduction

In reinforcement learning (RL), training an agent from scratch is notoriously data-hungry. Transfer RL seeks to mitigate this by leveraging knowledge from "source" agents already trained on similar tasks. However, a significant hurdle arises when environmental dynamics change—for instance, transferring a walking policy from a robot with long legs to one with shorter legs. Most existing methods assume identical physics or require access to source transition distributions, which is often impractical. MULTIPOLAR addresses this by treating a library of existing policies as a "council of mentors" to accelerate learning in a target environment with unknown dynamics.

Methodology: The MULTIPOLAR Framework

The MULTIPOLAR architecture consists of two primary components designed to balance the use of prior knowledge with the need for environment-specific adaptation.

The MULTIPOLAR architecture: Source policies are aggregated via state-independent weights, while an auxiliary network provides state-dependent residuals.

1. Adaptive Aggregation ($F_{agg}$)

Instead of selecting a single source policy, MULTIPOLAR aggregates deterministic actions from $K$ source policies. It learns a weight matrix ($\theta_{agg}$) to scale and combine these actions. These weights are state-independent, based on the observation that physical differences (e.g., mass or link length) typically affect optimal actions consistently across the state space. This approach reduces parameter overhead compared to state-dependent attention mechanisms.

2. Auxiliary Residual Network ($F_{aux}$)

To account for discrepancies between source and target dynamics, a separate neural network learns to predict a residual correction. The final action is the sum of the weighted mentor advice and the target agent’s own correction. This ensures that even if source policies are suboptimal or "wrong" for the new environment, the target agent can learn to ignore them and fine-tune its behavior.

Experimental Setup

The framework was evaluated across six OpenAI Gym and Roboschool environments, including CartPole, Acrobot, LunarLander, Hopper, Ant, and InvertedPendulumSwingUp. To simulate diverse dynamics, the researchers created 100 unique instances of each environment by randomly sampling physical parameters such as mass, friction, gravity, and link lengths.

Key Results and Insights

MULTIPOLAR demonstrates significant improvements in sample efficiency and final performance across various tasks.

  • Performance Gains: In the Roboschool Hopper environment, MULTIPOLAR achieved a mean episodic reward approximately 3x higher than learning from scratch (MLP) and 2x higher than state-of-the-art transfer methods like A2T and RPL after 2 million timesteps.
  • Robustness to Negative Transfer: The residual network acts as a safety valve. If a source policy is detrimental, the system learns to suppress its influence, preventing the "negative transfer" that often plagues other methods.
  • Scalability: Performance scales positively with the number of source policies ($K$), indicating that a larger "council of mentors" generally leads to better outcomes, even if the mentors are diverse.
Learning curves in the RoboschoolHopper environment, showing MULTIPOLAR (red) significantly outperforming baselines.

Conclusion

MULTIPOLAR provides a robust framework for transfer RL that is agnostic to the internal physics of source environments. By combining global policy aggregation with local residual corrections, it achieves high sample efficiency in environments with diverse dynamics. Future work may explore scenarios where both dynamics and reward functions vary, as well as applications in combinatorial optimization.

Summary updated

Abstract

Transfer reinforcement learning (RL) aims at improving the learning efficiency of an agent by exploiting knowledge from other source agents trained on relevant tasks. However, it remains challenging to transfer knowledge between different environmental dynamics without having access to the source environments. In this work, we explore a new challenge in transfer RL, where only a set of source policies collected under diverse unknown dynamics is available for learning a target task efficiently. To address this problem, the proposed approach, MULTI-source POLicy AggRegation (MULTIPOLAR), comprises two key techniques. We learn to aggregate the actions provided by the source policies adaptively to maximize the target task performance. Meanwhile, we learn an auxiliary network that predicts residuals around the aggregated actions, which ensures the target policy's expressiveness even when some of the source policies perform poorly. We demonstrated the effectiveness of MULTIPOLAR through an extensive experimental evaluation across six simulated environments ranging from classic control problems to challenging robotics simulations, under both continuous and discrete action spaces. The demo videos and code are available on the project webpage: https://omron-sinicx.github.io/multipolar/.
BibTeX
@inproceedings{barekatain2020multipolar,
  author = {Mohammadamin Barekatain and Ryo Yonetani and Masashi Hamaya},
  title = {{MULTIPOLAR: Multi-Source Policy Aggregation for Transfer Reinforcement Learning between Diverse Environmental Dynamics}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2020},
  url = {https://openreview.net/forum?id=o8oB5FRbblD},
}
Figure