Curriculum-Based Asymmetric Multi-Task Reinforcement Learning

Hanchi Huang, Deheng Ye, Li Shen, Wei Liu

TL;DR

CAMRL is the first reinforcement learning (RL) algorithm to combine curriculum learning with asymmetric multi-task transfer. By dynamically switching between parallel training and a directed "teacher-student" transfer mode, it effectively mitigates negative transfer and achieves state-of-the-art results across diverse robotic and gaming benchmarks.

What Problem Does This Paper Address?

Multi-task learning (MTL) aims to train an agent on several related tasks simultaneously, hoping that knowledge from one task helps another. However, RL faces a major hurdle: negative transfer. This occurs when the gradients or data from one task interfere with and degrade the performance of another.

Existing solutions often try to find "common traits" across all tasks, but this fails when tasks are highly diverse. Asymmetric Multi-task Learning (AMTL) offers a better path by allowing directed transfer—for instance, letting a "mastered" task help a "struggling" one, but not vice versa. However, applying AMTL to RL is difficult because RL is non-stationary (the agent's behavior and data distribution change constantly), making it hard to determine a fixed training order or a stable "regularization graph" between tasks.

Methods

The authors introduce CAMRL, which manages multi-task training through several novel mechanisms:

  1. Dynamic Mode Switching: Instead of using a fixed curriculum, CAMRL uses a composite indicator ($I_{mul}$) to decide at the start of each epoch whether to train tasks in parallel (single-task mode) or enter the AMTL mode. This indicator balances training time, overall performance, and the performance gap between tasks.
  2. Asymmetric Transfer Matrix: The core of the algorithm is a transfer matrix ($B$) that defines how much knowledge (network weights) is shared between every pair of tasks. This matrix is learned and updated continuously.
  3. Differentiable Ranking Functions: To incorporate prior knowledge (like task difficulty or similarity), the authors designed three novel differentiable ranking functions. These allow the model to "softly" enforce expectations—for example, that easier tasks should donate more knowledge to harder ones—within the gradient-based optimization process.
  4. Optimization Strategy: The algorithm uses alternating optimization and the Frank-Wolfe algorithm to update the transfer matrix and network weights.
  5. Automatic Hyper-parameter Tuning: To avoid the "hyper-parameter hell" common in multi-task loss functions, CAMRL uses an uncertainty-based adjustment mechanism to weigh different loss terms automatically.

Datasets

The authors evaluated CAMRL on an extensive suite of benchmarks:

  • Gym-minigrid: Navigation tasks with sparse rewards.
  • Meta-world (MT10 & MT50): 10 and 50 distinct robotic manipulation tasks (e.g., opening doors, pushing buttons).
  • Atari Video Games: Classic arcade environments.
  • PyBullet (Ravens): Vision-based tabletop manipulation.
  • RLBench: A large-scale benchmark for vision-guided manipulation.

Key Insights and Ideas

  • Curricula aren't always needed: The paper finds that performing curriculum-based transfer all the time is inefficient. By switching modes, CAMRL focuses on transfer only when there is a clear performance gap or a "bottleneck" task that needs help.
  • Directed Regularization: By treating the transfer matrix as a sparse, directed graph, the model avoids the "one-size-fits-all" distillation approach, allowing for more nuanced relationships between tasks.
  • Absorbing Prior Knowledge: The use of differentiable tanh-based ranking functions is a clever way to turn "common sense" or "expert rankings" into a signal that a neural network can actually learn from during backpropagation.
  • Adapting to New Tasks: Because the transfer matrix is modular, CAMRL can adapt to a brand-new task by simply expanding the matrix, without needing to retrain the original tasks from scratch.

Key Results

CAMRL consistently outperformed both single-task baselines and state-of-the-art multi-task algorithms (like Distral, Gradient Surgery, and Soft Module):

  • Meta-world: In the massive MT50 benchmark, CAMRL was the top performer, beating the second-best algorithm (Soft Module) by at least 20% in 32 out of 50 tasks.
  • Robotics: In RLBench, CAMRL ranked first in 9 out of 10 tasks, showing significant robustness in high-dimensional, vision-based environments.
  • Negative Transfer: The results showed almost no signs of negative transfer, a rare feat in large-scale multi-task RL.

Limitations and Future Work

  • Computational Overhead: While the authors mitigated this by randomly sampling task pairs for mutual evaluation, calculating the transfer matrix still adds complexity compared to simple parallel training.
  • Linear Transfer Assumption: The current model assumes a linear relationship for weight transfer ($w_t \approx \sum B_{st}w_s$). The authors suggest that exploring non-linear transfer functions could further improve performance.
  • Non-stationarity: While mode-switching helps, the rapid changes in RL agents still pose a challenge for maintaining a stable transfer graph. Future work could involve "sliding windows" or exponential discounting to better handle the evolving nature of the tasks.

Summary updated

Abstract

We introduce CAMRL, the first curriculum-based asymmetric multi-task learning (AMTL) algorithm for dealing with multiple reinforcement learning (RL) tasks altogether. To mitigate the negative influence of customizing the one-off training order in curriculum-based AMTL, CAMRL switches its training mode between parallel single-task RL and asymmetric multi-task RL (MTRL), according to an indicator regarding the training time, the overall performance, and the performance gap among tasks. To leverage the multi-sourced prior knowledge flexibly and to reduce negative transfer in AMTL, we customize a composite loss with multiple differentiable ranking functions and optimize the loss through alternating optimization and the Frank-Wolfe algorithm. The uncertainty-based automatic adjustment of hyper-parameters is also applied to eliminate the need of laborious hyper-parameter analysis during optimization. By optimizing the composite loss, CAMRL predicts the next training task and continuously revisits the transfer matrix and network weights. We have conducted experiments on a wide range of benchmarks in multi-task RL, covering Gym-minigrid, Meta-world, Atari video games, vision-based PyBullet tasks, and RLBench, to show the improvements of CAMRL over the corresponding single-task RL algorithm and state-of-the-art MTRL algorithms. The code is available at: <uri xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">https://github.com/huanghanchi/CAMRL</uri> .
BibTeX
@article{huang2023curriculum,
  author = {Hanchi Huang and Deheng Ye and Li Shen and Wei Liu},
  title = {{Curriculum-Based Asymmetric Multi-Task Reinforcement Learning}},
  journal = {IEEE Trans. Pattern Anal. Mach. Intell.},
  year = {2023},
  doi = {10.1109/tpami.2022.3223872},
}
Figure