Pre-training as Batch Meta Reinforcement Learning with tiMe

Quan Vuong, Shuang Liu, Minghua Liu, Kamil Ciosek, Hao Su, Henrik Iskov ChristensenarXiv 2019 · OpenReview

Abstract

We tackle the Multi-task Batch Reinforcement Learning problem. Given multiple datasets collected from different tasks, we train a multi-task policy to perform well in unseen tasks sampled from the same distribution. The task identities of the unseen tasks are not provided. To perform well, the policy must infer the task identity from collected transitions by modelling its dependency on states, actions and rewards. Because the different datasets may have state-action distributions with large divergence, the task inference module can learn to ignore the rewards and spuriously correlate only state-action pairs to the task identity, leading to poor test time performance. To robustify task inference, we propose a novel application of the triplet loss. To mine hard negative examples, we relabel the transitions from the training tasks by approximating their reward functions. When we allow further training on the unseen tasks, using the trained policy as an initialization leads to significantly faster convergence compared to randomly initialized policies (up to 80% improvement and across 5 different Mujoco task distributions). We name our method MBML (Multi-task Batch RL with Metric Learning).

Introduction: The Challenge of Dataset Bias

Batch Reinforcement Learning (RL) focuses on learning effective policies from fixed, offline datasets. In multi-task settings, the agent must infer the current task's identity from a "context set" of recent transitions. However, a significant hurdle arises: Dataset Bias.

In many offline datasets, state-action visitation frequencies vary drastically between tasks. For instance, if data for "Task 1" is concentrated in one region of the state space and "Task 2" in another, a task inference module may learn to identify tasks based solely on location, ignoring the reward signal. This spurious correlation fails when the agent encounters a new task in a previously visited region that requires different behavior.

A toy example illustrating how non-overlapping state-action distributions lead to failed task inference.

Methodology: MBML

To address this, we propose MBML (Multi-task Batch RL with Metric Learning). The approach involves a two-phase process:

  1. Single-Task Pre-training: We train independent policies for each task using Batch Constrained Q-Learning (BCQ).
  2. Multi-Task Distillation with Metric Learning: These policies are distilled into a single multi-task policy. To prevent spurious correlations, we introduce a triplet loss combined with reward relabeling.

The core innovation lies in the mining of hard negative examples. We train an ensemble of reward approximators for each task. During training, we take a transition from Task $j$ and relabel its reward using Task $i$'s reward function. The triplet loss then forces the task inference module to produce similar embeddings for (Task $i$ data) and (Task $j$ data relabeled as Task $i$), while distinguishing them from (Task $j$ data with its original reward). This forces the model to attend to the reward signal rather than just state-action pairs.

The MBML action selection architecture, which integrates inferred task identity (z) into the policy.

Experimental Results

MBML was evaluated on several continuous control benchmarks, including MuJoCo environments (AntDir, AntGoal, HalfCheetahVel, WalkerParam) and a modified HumanoidDir-M.

Key findings include:

  • Zero-Shot Performance: MBML consistently outperformed baselines like Contextual BCQ and batch-adapted PEARL, achieving higher average returns on unseen tasks without additional training.
  • Accelerated Fine-tuning: Policies initialized with MBML converged up to 80% faster than randomly initialized policies when allowed to interact with new tasks.
  • Robustness: Ablation studies confirmed that both the triplet loss and reward relabeling are critical for robust task inference.

Conclusion and Limitations

MBML provides a robust framework for multi-task batch RL by explicitly combating dataset bias through metric learning. While effective, the method's reliance on reward approximators suggests potential bottlenecks if these models fail to generalize. Future work will explore extending this approach to high-dimensional visual inputs and tasks with sparse reward structures.

Summary updated

Abstract

We tackle the Multi-task Batch Reinforcement Learning problem. Given multiple datasets collected from different tasks, we train a multi-task policy to perform well in unseen tasks sampled from the same distribution. The task identities of the unseen tasks are not provided. To perform well, the policy must infer the task identity from collected transitions by modelling its dependency on states, actions and rewards. Because the different datasets may have state-action distributions with large divergence, the task inference module can learn to ignore the rewards and spuriously correlate $\textit{only}$ state-action pairs to the task identity, leading to poor test time performance. To robustify task inference, we propose a novel application of the triplet loss. To mine hard negative examples, we relabel the transitions from the training tasks by approximating their reward functions. When we allow further training on the unseen tasks, using the trained policy as an initialization leads to significantly faster convergence compared to randomly initialized policies (up to $80\%$ improvement and across 5 different Mujoco task distributions). We name our method $\textbf{MBML}$ ($\textbf{M}\text{ulti-task}$ $\textbf{B}\text{atch}$ RL with $\textbf{M}\text{etric}$ $\textbf{L}\text{earning}$).
BibTeX
@misc{vuong2019pre,
  author = {Quan Vuong and Shuang Liu and Minghua Liu and Kamil Ciosek and Hao Su and Henrik Iskov Christensen},
  title = {{Pre-training as Batch Meta Reinforcement Learning with tiMe}},
  year = {2019},
  eprint = {1909.11373},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1909.11373},
}
Figure