MULTIPOLAR: Multi-Source Policy Aggregation for Transfer Reinforcement Learning between Diverse Environmental Dynamics

Mohammadamin Barekatain, Ryo Yonetani, Masashi Hamaya

Abstract

Transfer reinforcement learning (RL) aims at improving the learning efficiency of an agent by exploiting knowledge from other source agents trained on relevant tasks. However, it remains challenging to transfer knowledge between different environmental dynamics without having access to the source environments. In this work, we explore a new challenge in transfer RL, where only a set of source policies collected under diverse unknown dynamics is available for learning a target task efficiently. To address this problem, the proposed approach, MULTI-source POLicy AggRegation (MULTIPOLAR), comprises two key techniques. We learn to aggregate the actions provided by the source policies adaptively to maximize the target task performance. Meanwhile, we learn an auxiliary network that predicts residuals around the aggregated actions, which ensures the target policy's expressiveness even when some of the source policies perform poorly. We demonstrated the effectiveness of MULTIPOLAR through an extensive experimental evaluation across six simulated environments ranging from classic control problems to challenging robotics simulations, under both continuous and discrete action spaces.

Introduction

In traditional Reinforcement Learning (RL), an agent typically learns from scratch through trial and error. Transfer RL aims to accelerate this process by leveraging knowledge from "source" agents. However, a significant gap exists in current methodologies: most assume that source and target environments share identical physical dynamics or that the learner has full access to source training data.

In real-world scenarios, such as a fleet of industrial robots, physical parameters like joint friction, link lengths, or payload weights often vary between units. Introducing a new "target" robot requires it to learn from existing "source" policies without necessarily having access to their proprietary data or assuming identical hardware. The core challenge addressed by this research is: how can an agent effectively combine multiple, potentially suboptimal policies when the underlying physics of the environment has changed?

The MULTIPOLAR Framework

The authors propose MULTIPOLAR (MULTI-source POLicy AggRegation), a framework that constructs a target policy by integrating two parallel components:

  1. Adaptive Aggregation ($F_{agg}$): Rather than selecting a single best source policy, MULTIPOLAR aggregates actions from all $K$ available source policies. These actions are multiplied by a learnable weight matrix ($\theta_{agg}$). Unlike interpolation-based attention mechanisms, MULTIPOLAR can scale actions independently, allowing the agent to "mute" irrelevant sources or amplify useful ones across the state space.
  2. Auxiliary Residual Network ($F_{aux}$): To prevent the target agent from being limited by potentially poor source policies, an auxiliary network learns to predict a "residual" correction. The final action is the sum of the aggregated source actions and this residual.
The MULTIPOLAR framework enables knowledge transfer across diverse physical dynamics by aggregating source policies and learning residuals.

This architecture is end-to-end differentiable, allowing the agent to learn source weighting and residual corrections simultaneously using standard RL algorithms like PPO or SAC. It is designed to be robust across both continuous and discrete action spaces.

Experimental Evaluation

The effectiveness of MULTIPOLAR was evaluated across six diverse OpenAI Gym environments, modified to introduce "diverse dynamics" by randomly sampling physical parameters such as mass, friction, and gravity.

Benchmark environments used in the study, including complex robotics simulations like Ant and Hopper alongside classic control tasks.

The benchmark setup involved generating 100 unique instances for each environment (e.g., CartPole, Acrobot, Hopper, Ant, LunarLander). Source agents were trained on a subset of these instances, while the target agent was tested on unseen dynamics, ensuring no direct overlap in physical parameters during training.

Results and Key Insights

The experimental results highlighted several advantages of the MULTIPOLAR approach:

  • Superior Efficiency: In the Hopper environment, MULTIPOLAR achieved a mean reward three times higher than learning from scratch and nearly double that of state-of-the-art baselines like A2T (Attend, Adapt, and Transfer) within 2 million samples.
  • Mitigating Negative Transfer: A common risk in transfer learning is "negative transfer," where irrelevant old knowledge hinders new learning. MULTIPOLAR's weighting mechanism successfully mitigates this; the network learns to assign near-zero weights to detrimental source policies.
  • Guided Exploration: The source policies act as a powerful inductive bias. Even if a source robot has different physical properties, its knowledge of basic task mechanics provides a "baseline" behavior. This prevents immediate failure during early training, allowing the residual network to safely discover optimal refinements.
  • Scalability: Performance improved monotonically as the number of source policies increased (tested up to $K=16$), demonstrating the framework's ability to effectively sift through large libraries of prior knowledge.

Conclusion and Future Work

MULTIPOLAR demonstrates that source policies can serve as effective priors for learning in environments with unknown and diverse dynamics. While the current work assumes a consistent reward function across tasks, future research could explore transferring knowledge between different goals. Additionally, while the framework scales well to 16 sources, identifying the saturation point where overhead exceeds benefit remains an open question. The ultimate goal is the application of these techniques to sim-to-real transfer in physical robotic systems.

Summary updated

Abstract

Transfer reinforcement learning (RL) aims at improving the learning efficiency of an agent by exploiting knowledge from other source agents trained on relevant tasks. However, it remains challenging to transfer knowledge between different environmental dynamics without having access to the source environments. In this work, we explore a new challenge in transfer RL, where only a set of source policies collected under diverse unknown dynamics is available for learning a target task efficiently. To address this problem, the proposed approach, MULTI-source POLicy AggRegation (MULTIPOLAR), comprises two key techniques. We learn to aggregate the actions provided by the source policies adaptively to maximize the target task performance. Meanwhile, we learn an auxiliary network that predicts residuals around the aggregated actions, which ensures the target policy's expressiveness even when some of the source policies perform poorly. We demonstrated the effectiveness of MULTIPOLAR through an extensive experimental evaluation across six simulated environments ranging from classic control problems to challenging robotics simulations, under both continuous and discrete action spaces. The demo videos and code are available on the project webpage: https://omron-sinicx.github.io/multipolar/.
BibTeX
@inproceedings{barekatain2020multipolar,
  author = {Mohammadamin Barekatain and Ryo Yonetani and Masashi Hamaya},
  title = {{MULTIPOLAR: Multi-Source Policy Aggregation for Transfer Reinforcement Learning between Diverse Environmental Dynamics}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2020},
  url = {https://openreview.net/forum?id=o8oB5FRbblD},
}
Figure