TL;DR
MBML tackles a critical "cheating" problem in multi-task offline reinforcement learning: task inference modules often ignore rewards and rely on biased state-action distributions to identify tasks. By combining triplet loss with a novel reward-relabeling strategy, MBML forces models to learn robust task representations, leading to superior generalization and up to an 80% speedup when fine-tuning on new tasks.
What Problem Does This Paper Address?
Reinforcement Learning (RL) is notoriously sample-inefficient, often requiring millions of interactions in simulation. Batch RL (or Offline RL) attempts to solve this by learning from pre-collected datasets. When we move to Multi-task Batch RL, the goal is to train a single policy on multiple datasets from different tasks so it can generalize to a new, unseen task without being told the task's identity.
To succeed, the agent must perform task inference: it looks at a few recent transitions (the "context set") to figure out what the current goal is. However, the authors identify a major pitfall: spurious correlation. In offline datasets, different tasks often have very different state-action distributions (e.g., in Task A the robot only moves left, in Task B it only moves right). A neural network will "cheat" by identifying the task based solely on where the robot is, completely ignoring the rewards.
At test time, if a new task requires the robot to move in a way that overlaps with training distributions, the inference module fails because it never learned that rewards are the true task indicators.
Methods
The authors propose MBML (Multi-task Batch RL with Metric Learning), which uses a two-phase distillation process:
- Single-task Training: They first train independent "expert" policies for each task in the batch using Batch Constrained Q-learning (BCQ).
- Multi-task Distillation: They distill these experts into one global policy. To solve the "cheating" problem, they introduce a task inference module ($q_\phi$) trained with a Triplet Loss.
The Secret Sauce: Reward Relabeling To make the triplet loss effective, they need "hard negatives." They train a reward regressor for each task. They then take transitions from Task A and "relabel" them using the reward function of Task B.
- Positive Pair: Transitions from Task A vs. Transitions from Task B relabeled with Task A’s rewards. (Same rewards, different states).
- Negative Pair: Transitions from Task B relabeled with Task A’s rewards vs. Transitions from Task B with its original rewards. (Same states, different rewards).
By forcing the model to distinguish between these pairs, the agent is forced to look at the rewards to identify the task, as the state-action pairs are identical in the negative case.
Datasets
The researchers evaluated MBML on several challenging continuous control benchmarks:
- MuJoCo Environments: AntDir (moving in specific directions), AntGoal (navigating to locations), HalfCheetahVel (reaching target velocities), and WalkerParam (varying physical properties like friction).
- HumanoidDir-M: A modified version of the Humanoid direction task designed to be more difficult for task inference.
- D4RL: The UmazeGoal-M task from the standard offline RL benchmark suite.
Key Insights and Ideas
- Causal Confusion: The paper highlights that offline RL is particularly susceptible to "causal confusion," where the model mistakes a correlation (location) for a cause (the task goal).
- Metric Learning as a Robustifier: While triplet loss is common in computer vision, applying it here to "anchor" task embeddings to rewards rather than state-visitation patterns is a clever bridge between metric learning and RL.
- Initialization is Everything: One of the most powerful realizations is that even if a multi-task policy isn't perfect out of the box, it serves as a massive "head start" for online learning, providing a much better starting point than random weights.
Key Results
- Zero-shot Performance: MBML consistently outperformed baselines like PEARL (adapted for batch) and Contextual BCQ across all five MuJoCo task distributions.
- Sample Efficiency: When allowed to fine-tune on unseen tasks, policies initialized with MBML reached optimal performance significantly faster than those starting from scratch. In the HumanoidDir-M task, MBML achieved a 76% improvement in sample efficiency, requiring only 85k interactions compared to the baseline's 350k.
- Ablation Success: The authors showed that "No relabeling" or "No triplet loss" led to significantly lower returns, proving that the combination of metric learning and reward-based hard negative mining is the core driver of their success.
Limitations and Future Work
- Reward Generalization: The method assumes that a reward function learned on one task's data can generalize well enough to predict rewards for state-action pairs from a different task.
- State Complexity: The experiments focused on proprioceptive states (joint angles, velocities). The authors acknowledge that high-dimensional visual inputs (pixels) present a different set of challenges for task inference.
- Dense Rewards: The current framework relies on relatively dense reward signals; adapting this to sparse or binary rewards (where "relabeling" provides less signal) remains an open question.
- Architecture: Future work could explore more complex architectures like Graph Neural Networks (GNNs) to improve the task inference module's reasoning capabilities.
Summary updated