Soft Conflict-Resolution Decision Transformer for Offline Multi-Task Reinforcement Learning

Shudong Wang, Xinfei Wang, Chenhao Zhang, Shanchen Pang, Haiyuan Gui, Wenhao Ji, Xiaojian Liao

TL;DR

This paper introduces SoCo-DT, a novel approach for multi-task reinforcement learning (MTRL) that tackles the challenge of conflicting gradients between tasks. By employing a "soft" masking strategy that considers parameter importance and dynamically adapting sparsity based on task-specific conflict levels, SoCo-DT significantly improves generalization and learning efficiency in offline MTRL settings.

What Problem Does This Paper Address?

Multi-task reinforcement learning (MTRL) aims to train a single policy that can perform well across multiple tasks, leveraging shared knowledge to improve efficiency and generalization. However, a major hurdle is "gradient conflict," where updates for one task can negatively impact performance on another. Existing methods often use binary masks to block conflicting parameters, but this is too coarse. It can lead to over-suppression of important parameters, hindering knowledge sharing, and doesn't account for the fact that conflict levels vary greatly between tasks and even change during training. This inflexibility limits the model's ability to generalize and learn efficiently, especially in complex, heterogeneous environments.

Methods

SoCo-DT introduces two key innovations to address the limitations of existing masking strategies:

  1. Importance-Aware Soft Masking Mechanism: Instead of binary masks (either block or allow), SoCo-DT uses "soft" masks. These masks assign a continuous value (between 0 and 1) to each parameter, reflecting its importance and conflict level. This is achieved by leveraging Fisher information (a measure of parameter importance) to dynamically adjust mask values. This allows important parameters that are also conflicting to be retained with higher weights, while less critical conflicting parameters are gradually suppressed. This principled approach balances conflict mitigation with knowledge preservation.
  2. Task-Aware Mask Update with Adaptive Sparsity (TAMU): This strategy dynamically adjusts the masking process to be task-specific and evolve over time.
  3. Conflict and Harmony Scores: It first computes fine-grained "conflict" and "harmony" scores for each parameter. The conflict score incorporates gradient alignment and parameter importance, while the harmony score uses a magnitude-sensitive mechanism to downweight parameters with extreme deviations, preventing one task from dominating.
  4. Adaptive Sparsity: TAMU uses the Interquartile Range (IQR) of these scores to determine task-specific thresholds for masking. This means each task gets its own sparsity level, adapting to its unique conflict profile.
  5. Asymmetric Cosine Annealing: To further refine the sparsity evolution, an asymmetric cosine annealing schedule is used to continuously update the thresholds during training. This allows for more flexible adaptation as training progresses, balancing stability and performance.

The overall framework integrates these components with a prompt-enhanced Decision Transformer architecture.

Datasets

The experiments were conducted on the Meta-World benchmark, a popular multi-task robotic manipulation environment. The paper uses two types of datasets:

  • Near-optimal dataset: Composed of high-quality, early-stage trajectories from a SAC-Replay policy.
  • Sub-optimal dataset: Contains early-stage trajectories with only 50% expert demonstrations retained, making it a more challenging scenario.

The experiments cover configurations with 5, 30, and 50 tasks (MT5, MT30, MT50).

Key Insights and Ideas

  • Binary masks are too blunt: The paper's empirical study highlights that hard, binary masks often over-suppress parameters that are crucial for individual tasks, even if they are involved in conflicts. This is a significant bottleneck for knowledge sharing.
  • Parameter importance matters: Simply masking conflicting parameters isn't enough; understanding how important a conflicting parameter is to a specific task is key to making informed decisions about masking.
  • Conflict is dynamic and task-specific: The degree of conflict isn't static. It varies significantly across tasks and changes throughout the training process. A one-size-fits-all sparsity strategy fails to capture this nuance.
  • Soft masking is a more principled approach: By using continuous mask values weighted by parameter importance, SoCo-DT achieves a better balance between mitigating conflicts and preserving valuable knowledge.
  • Adaptive sparsity is crucial for stability: Dynamically adjusting masking thresholds based on task-specific conflict distributions (using IQR) and annealing schedules leads to more stable and effective training.

Key Results

SoCo-DT demonstrates significant improvements over state-of-the-art methods on the Meta-World benchmark:

  • On the MT50 benchmark, SoCo-DT achieved 7.6% higher performance on near-optimal datasets and 10.5% higher performance on sub-optimal datasets compared to the best baseline (HarmoDT-F).
  • The performance gains were particularly pronounced on sub-optimal datasets, indicating SoCo-DT's enhanced stability and robustness when reward signals are ambiguous.
  • Ablation studies confirmed the effectiveness of each proposed component: the soft mask mechanism, adaptive sparsity, and magnitude-sensitive harmony scoring.

Limitations and Future Work

  • Limitations: While the paper shows strong results, the authors acknowledge that the current approach is primarily focused on mitigating gradient conflicts. Future work could explore other sources of inter-task interference. The computational cost of calculating Fisher information might also be a consideration for extremely large models or task sets.
  • Future Work: The authors suggest exploring more sophisticated methods for identifying and resolving inter-task conflicts. They also propose investigating the application of SoCo-DT to other domains beyond robotic manipulation and exploring its potential in online MTRL settings.

Summary updated

Abstract

Multi-task reinforcement learning (MTRL) seeks to learn a unified policy for diverse tasks, but often suffers from gradient conflicts across tasks. Existing masking-based methods attempt to mitigate such conflicts by assigning task-specific parameter masks. However, our empirical study shows that coarse-grained binary masks have the problem of over-suppressing key conflicting parameters, hindering knowledge sharing across tasks. Moreover, different tasks exhibit varying conflict levels, yet existing methods use a one-size-fits-all fixed sparsity strategy to keep training stability and performance, which proves inadequate. These limitations hinder the model’s generalization and learning efficiency. To address these issues, we propose SoCo-DT, a Soft Conflict-resolution method based by parameter importance. By leveraging Fisher information, mask values are dynamically adjusted to retain important parameters while suppressing conflicting ones. In addition, we introduce a dynamic sparsity adjustment strategy based on the Interquartile Range (IQR), which constructs task-specific thresholding schemes using the distribution of conflict and harmony scores during training. To enable adaptive sparsity evolution throughout training, we further incorporate an asymmetric cosine annealing schedule to continuously update the threshold. Experimental results on the Meta-World benchmark show that SoCo-DT outperforms the state-of-the-art method by 7.6% on MT50 and by 10.5% on the suboptimal dataset, demonstrating its effectiveness in mitigating gradient conflicts and improving overall multi-task performance.
BibTeX
@inproceedings{wang2026soft,
  author = {Shudong Wang and Xinfei Wang and Chenhao Zhang and Shanchen Pang and Haiyuan Gui and Wenhao Ji and Xiaojian Liao},
  title = {{Soft Conflict-Resolution Decision Transformer for Offline Multi-Task Reinforcement Learning}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2026},
  url = {https://openreview.net/forum?id=YOVnCXmYwc},
}
Figure