Curriculum-based Asymmetric Multi-task Reinforcement Learning

Hanchi Huang, Deheng Ye, Li Shen, Wei Liu

Abstract

Multi-task reinforcement learning (RL) often suffers from negative transfer, where interference between tasks degrades overall performance. We present Curriculum-based Asymmetric Multi-task Reinforcement Learning (CAMRL), a framework that addresses this by dynamically switching between parallel training and a directed asymmetric transfer mode. CAMRL employs a learned transfer matrix and differentiable ranking functions to manage knowledge sharing between tasks. Evaluated on benchmarks like Meta-world and RLBench, CAMRL achieves state-of-the-art performance, significantly outperforming existing multi-task baselines and effectively mitigating negative transfer.

Introduction

Multi-task learning (MTL) aims to leverage shared knowledge across related tasks to improve sample efficiency and final performance. In reinforcement learning, however, this is complicated by the non-stationary nature of the environment and the agent's policy. A primary obstacle is negative transfer, where gradients or data from one task conflict with another. While Asymmetric Multi-task Learning (AMTL) allows for directed transfer (e.g., from a mastered task to a struggling one), determining the optimal transfer order in a non-stationary RL setting is challenging. CAMRL addresses this by treating the curriculum as a dynamic process rather than a fixed sequence.

The CAMRL Framework

CAMRL introduces a flexible training regime that alternates between two modes based on a composite indicator, $I_{mul}$. This indicator accounts for training time, performance variance, and the performance gap between tasks to decide when transfer is most beneficial.

Overview of the CAMRL framework showing dynamic mode switching between parallel single-task learning and curriculum-based AMTL.

Dynamic Mode Switching

Instead of a static curriculum, CAMRL evaluates the state of training at the start of each epoch. If tasks are progressing evenly, it continues in parallel mode. If a significant performance gap emerges or a task hits a bottleneck, it switches to AMTL mode to allow "teacher" tasks to assist "student" tasks through weight transfer.

Asymmetric Transfer Matrix

The core of the transfer mechanism is a learned matrix $B$, where each element $B_{st}$ represents the weight of knowledge transferred from source task $s$ to target task $t$. This matrix is optimized using the Frank-Wolfe algorithm to ensure sparse and effective directed regularization, avoiding the "one-size-fits-all" approach of traditional distillation.

Incorporating Prior Knowledge

To guide the learning of the transfer matrix, CAMRL uses differentiable ranking functions. These functions allow the integration of prior knowledge—such as task difficulty or known similarities—into the optimization process. By using tanh-based "soft" ranking, the model can enforce expectations (e.g., easier tasks should generally donate knowledge to harder ones) while remaining fully end-to-end differentiable within the gradient-based optimization.

Experimental Evaluation

CAMRL was tested on several challenging benchmarks, including Meta-world (MT10 & MT50), RLBench, and classic Atari games. In the MT50 benchmark, CAMRL outperformed the second-best algorithm, Soft Module, by over 20% in 32 out of 50 tasks. In RLBench, it ranked first in 9 out of 10 vision-guided manipulation tasks.

Learned Transfer Matrix B for the MT10 benchmark, illustrating the asymmetric relationships between tasks.

Notably, the framework showed almost no signs of negative transfer, a rare feat in large-scale multi-task RL. The learned transfer matrix (as seen in Figure 4) demonstrates how the model successfully identifies specific task-to-task relationships to facilitate learning.

Conclusion

CAMRL represents a significant step forward in multi-task RL by combining the strengths of curriculum learning and asymmetric transfer. Its ability to dynamically adapt the transfer graph and incorporate prior knowledge makes it robust to the diversity and non-stationarity of complex RL environments. Future work will explore non-linear transfer functions to further enhance the flexibility of knowledge sharing between tasks.

Summary updated

BibTeX
@article{huang2023curriculum,
  author = {Hanchi Huang and Deheng Ye and Li Shen and Wei Liu},
  title = {{Curriculum-based Asymmetric Multi-task Reinforcement Learning}},
  journal = {IEEE Trans. Pattern Anal. Mach. Intell.},
  year = {2023},
  doi = {10.1109/tpami.2022.3223872},
}
Figure