TL;DR
This paper introduces SoCo-DT, a novel approach for multi-task reinforcement learning (MTRL) that tackles the challenge of conflicting gradients between tasks. By employing a "soft" masking strategy that considers parameter importance and dynamically adapting sparsity based on task-specific conflict levels, SoCo-DT significantly improves generalization and learning efficiency in offline MTRL settings.
What Problem Does This Paper Address?
Multi-task reinforcement learning (MTRL) aims to train a single policy that can perform well across multiple tasks, leveraging shared knowledge to improve efficiency and generalization. However, a major hurdle is "gradient conflict," where updates for one task can negatively impact performance on another. Existing methods often use binary masks to block conflicting parameters, but this is too coarse. It can lead to over-suppression of important parameters, hindering knowledge sharing, and doesn't account for the fact that conflict levels vary greatly between tasks and even change during training. This inflexibility limits the model's ability to generalize and learn efficiently, especially in complex, heterogeneous environments.
Methods
SoCo-DT introduces two key innovations to address the limitations of existing masking strategies:
- Importance-Aware Soft Masking Mechanism: Instead of binary masks (either block or allow), SoCo-DT uses "soft" masks. These masks assign a continuous value (between 0 and 1) to each parameter, reflecting its importance and conflict level. This is achieved by leveraging Fisher information (a measure of parameter importance) to dynamically adjust mask values. This allows important parameters that are also conflicting to be retained with higher weights, while less critical conflicting parameters are gradually suppressed. This principled approach balances conflict mitigation with knowledge preservation.
- Task-Aware Mask Update with Adaptive Sparsity (TAMU): This strategy dynamically adjusts the masking process to be task-specific and evolve over time.
- Conflict and Harmony Scores: It first computes fine-grained "conflict" and "harmony" scores for each parameter. The conflict score incorporates gradient alignment and parameter importance, while the harmony score uses a magnitude-sensitive mechanism to downweight parameters with extreme deviations, preventing one task from dominating.
- Adaptive Sparsity: TAMU uses the Interquartile Range (IQR) of these scores to determine task-specific thresholds for masking. This means each task gets its own sparsity level, adapting to its unique conflict profile.
- Asymmetric Cosine Annealing: To further refine the sparsity evolution, an asymmetric cosine annealing schedule is used to continuously update the thresholds during training. This allows for more flexible adaptation as training progresses, balancing stability and performance.
The overall framework integrates these components with a prompt-enhanced Decision Transformer architecture.
Datasets
The experiments were conducted on the Meta-World benchmark, a popular multi-task robotic manipulation environment. The paper uses two types of datasets:
- Near-optimal dataset: Composed of high-quality, early-stage trajectories from a SAC-Replay policy.
- Sub-optimal dataset: Contains early-stage trajectories with only 50% expert demonstrations retained, making it a more challenging scenario.
The experiments cover configurations with 5, 30, and 50 tasks (MT5, MT30, MT50).
Key Insights and Ideas
- Binary masks are too blunt: The paper's empirical study highlights that hard, binary masks often over-suppress parameters that are crucial for individual tasks, even if they are involved in conflicts. This is a significant bottleneck for knowledge sharing.
- Parameter importance matters: Simply masking conflicting parameters isn't enough; understanding how important a conflicting parameter is to a specific task is key to making informed decisions about masking.
- Conflict is dynamic and task-specific: The degree of conflict isn't static. It varies significantly across tasks and changes throughout the training process. A one-size-fits-all sparsity strategy fails to capture this nuance.
- Soft masking is a more principled approach: By using continuous mask values weighted by parameter importance, SoCo-DT achieves a better balance between mitigating conflicts and preserving valuable knowledge.
- Adaptive sparsity is crucial for stability: Dynamically adjusting masking thresholds based on task-specific conflict distributions (using IQR) and annealing schedules leads to more stable and effective training.
Key Results
SoCo-DT demonstrates significant improvements over state-of-the-art methods on the Meta-World benchmark:
- On the MT50 benchmark, SoCo-DT achieved 7.6% higher performance on near-optimal datasets and 10.5% higher performance on sub-optimal datasets compared to the best baseline (HarmoDT-F).
- The performance gains were particularly pronounced on sub-optimal datasets, indicating SoCo-DT's enhanced stability and robustness when reward signals are ambiguous.
- Ablation studies confirmed the effectiveness of each proposed component: the soft mask mechanism, adaptive sparsity, and magnitude-sensitive harmony scoring.
Limitations and Future Work
- Limitations: While the paper shows strong results, the authors acknowledge that the current approach is primarily focused on mitigating gradient conflicts. Future work could explore other sources of inter-task interference. The computational cost of calculating Fisher information might also be a consideration for extremely large models or task sets.
- Future Work: The authors suggest exploring more sophisticated methods for identifying and resolving inter-task conflicts. They also propose investigating the application of SoCo-DT to other domains beyond robotic manipulation and exploring its potential in online MTRL settings.
Summary updated