TL;DR
CAMRL is the first reinforcement learning (RL) algorithm to combine curriculum learning with asymmetric multi-task transfer. By dynamically switching between parallel training and a directed "teacher-student" transfer mode, it effectively mitigates negative transfer and achieves state-of-the-art results across diverse robotic and gaming benchmarks.
What Problem Does This Paper Address?
Multi-task learning (MTL) aims to train an agent on several related tasks simultaneously, hoping that knowledge from one task helps another. However, RL faces a major hurdle: negative transfer. This occurs when the gradients or data from one task interfere with and degrade the performance of another.
Existing solutions often try to find "common traits" across all tasks, but this fails when tasks are highly diverse. Asymmetric Multi-task Learning (AMTL) offers a better path by allowing directed transfer—for instance, letting a "mastered" task help a "struggling" one, but not vice versa. However, applying AMTL to RL is difficult because RL is non-stationary (the agent's behavior and data distribution change constantly), making it hard to determine a fixed training order or a stable "regularization graph" between tasks.
Methods
The authors introduce CAMRL, which manages multi-task training through several novel mechanisms:
- Dynamic Mode Switching: Instead of using a fixed curriculum, CAMRL uses a composite indicator ($I_{mul}$) to decide at the start of each epoch whether to train tasks in parallel (single-task mode) or enter the AMTL mode. This indicator balances training time, overall performance, and the performance gap between tasks.
- Asymmetric Transfer Matrix: The core of the algorithm is a transfer matrix ($B$) that defines how much knowledge (network weights) is shared between every pair of tasks. This matrix is learned and updated continuously.
- Differentiable Ranking Functions: To incorporate prior knowledge (like task difficulty or similarity), the authors designed three novel differentiable ranking functions. These allow the model to "softly" enforce expectations—for example, that easier tasks should donate more knowledge to harder ones—within the gradient-based optimization process.
- Optimization Strategy: The algorithm uses alternating optimization and the Frank-Wolfe algorithm to update the transfer matrix and network weights.
- Automatic Hyper-parameter Tuning: To avoid the "hyper-parameter hell" common in multi-task loss functions, CAMRL uses an uncertainty-based adjustment mechanism to weigh different loss terms automatically.
Datasets
The authors evaluated CAMRL on an extensive suite of benchmarks:
- Gym-minigrid: Navigation tasks with sparse rewards.
- Meta-world (MT10 & MT50): 10 and 50 distinct robotic manipulation tasks (e.g., opening doors, pushing buttons).
- Atari Video Games: Classic arcade environments.
- PyBullet (Ravens): Vision-based tabletop manipulation.
- RLBench: A large-scale benchmark for vision-guided manipulation.
Key Insights and Ideas
- Curricula aren't always needed: The paper finds that performing curriculum-based transfer all the time is inefficient. By switching modes, CAMRL focuses on transfer only when there is a clear performance gap or a "bottleneck" task that needs help.
- Directed Regularization: By treating the transfer matrix as a sparse, directed graph, the model avoids the "one-size-fits-all" distillation approach, allowing for more nuanced relationships between tasks.
- Absorbing Prior Knowledge: The use of differentiable tanh-based ranking functions is a clever way to turn "common sense" or "expert rankings" into a signal that a neural network can actually learn from during backpropagation.
- Adapting to New Tasks: Because the transfer matrix is modular, CAMRL can adapt to a brand-new task by simply expanding the matrix, without needing to retrain the original tasks from scratch.
Key Results
CAMRL consistently outperformed both single-task baselines and state-of-the-art multi-task algorithms (like Distral, Gradient Surgery, and Soft Module):
- Meta-world: In the massive MT50 benchmark, CAMRL was the top performer, beating the second-best algorithm (Soft Module) by at least 20% in 32 out of 50 tasks.
- Robotics: In RLBench, CAMRL ranked first in 9 out of 10 tasks, showing significant robustness in high-dimensional, vision-based environments.
- Negative Transfer: The results showed almost no signs of negative transfer, a rare feat in large-scale multi-task RL.
Limitations and Future Work
- Computational Overhead: While the authors mitigated this by randomly sampling task pairs for mutual evaluation, calculating the transfer matrix still adds complexity compared to simple parallel training.
- Linear Transfer Assumption: The current model assumes a linear relationship for weight transfer ($w_t \approx \sum B_{st}w_s$). The authors suggest that exploring non-linear transfer functions could further improve performance.
- Non-stationarity: While mode-switching helps, the rapid changes in RL agents still pose a challenge for maintaining a stable transfer graph. Future work could involve "sliding windows" or exponential discounting to better handle the evolving nature of the tasks.
Summary updated