On Context Distribution Shift in Task Representation Learning for Offline Meta RL

Chenyang Zhao, Zihao Zhou, Bin LiuarXiv 2023 · DOI · arXiv

Abstract

Offline Meta-Reinforcement Learning (OMRL) faces a significant challenge known as "context distribution shift," where context encoders trained on high-quality offline data struggle with low-quality trajectories at test time. This paper proposes a robust training strategy utilizing supervised contrastive learning with hard positive and hard negative sampling. This approach enhances the agent's ability to adapt to new tasks even when provided with imperfect or sub-optimal context information.

Introduction

Meta-Reinforcement Learning (Meta-RL) aims to develop agents capable of rapid adaptation to novel tasks by leveraging a small set of "context" transitions (state, action, reward, next state). In the offline setting (OMRL), agents must learn this capability from static, pre-collected datasets without further environment interaction.

A major bottleneck identified in current OMRL methods is the distribution shift between training and testing contexts. While training datasets often consist of high-performing trajectories, real-world test-time contexts may be generated by sub-optimal or random policies. Standard context encoders, having only seen "expert-like" data, often fail to generalize to these "messy" contexts, leading to a collapse in task identification and overall performance when the agent tries to adapt.

Methods

The proposed framework adopts a decoupled training strategy. First, a robust task context encoder is trained to map transitions to a latent task representation. Subsequently, this encoder is frozen, and a contextual policy is trained using Implicit Q-Learning (IQL).

Overview of the meta-training and meta-testing architecture, highlighting the decoupled training of the encoder and contextual policy.

The core innovation is the Supervised Contrastive Learning (SCL) objective for the encoder, which incorporates two specialized sampling strategies:

  1. Hard Negative Sampling: This strategy identifies "negative" samples (trajectories from different tasks) that are currently close together in the embedding space and applies a higher penalty to force them apart.
  2. Hard Positive Sampling: This is crucial for robustness. It identifies "positive" samples (trajectories from the same task) that are far apart in the embedding space—for example, an expert trajectory versus a low-quality trajectory of the same task. By weighing these "hard" pairs more heavily, the encoder learns to map them to the same task representation regardless of their quality.

By combining these strategies into a single objective ($\mathcal{L}^{HP+HG}$), the encoder learns a representation that is invariant to the quality of the behavior policy that generated the context.

Key Insights: The Impact of Context Quality

A central finding of this work is that context quality significantly impacts the separability of task representations. Standard encoders often fail to cluster low-quality contexts correctly, causing the agent to misidentify the current task.

Visualization of trajectories (top) and t-SNE task embeddings (bottom) across low, medium, and high quality contexts. The proposed method aims to maintain cluster separability regardless of context quality.

By focusing on hard sampling, the model improves task separability. The goal is to ensure that latent space boundaries are defined by the underlying task mechanics rather than the performance level of the agent that generated the context. Analysis of the resulting embeddings shows improved alignment (tighter clustering of similar tasks) while maintaining broad coverage of the representation space (uniformity).

Experimental Results

The method was evaluated across five MuJoCo-based continuous control environments: PointRobotGoal, AntDir, CheetahVel, WalkerParams, and HopperParams. To simulate distribution shift, testing buffers were categorized by return quality (e.g., bottom 10th percentile vs. top 10th percentile).

The proposed method significantly outperformed baselines like FOCAL, CORRO, and Offline PEARL, particularly when faced with low-quality contexts:

  • PointRobotGoal: The proposed method achieved a return of -44.9, while the next best baseline (CORRO) trailed at -52.7.
  • AntDir: The proposed method reached 352.1, significantly higher than CORRO's 313.0.

While baselines often performed well on high-quality data, their performance dropped sharply on low-quality data. In contrast, the proposed hard-sampling approach remained stable, demonstrating a more fundamental understanding of task identity.

Conclusion and Future Work

This research highlights the importance of addressing context distribution shift in OMRL. While the proposed method shows clear advantages in shift scenarios, its gains are more modest when training and testing distributions are perfectly matched. Future work could explore scaling these robust representations to more complex task spaces involving discrete actions or high-dimensional observations, as well as investigating the interaction between the context encoder and different offline RL backbones.

Summary updated

Abstract

Offline Meta Reinforcement Learning (OMRL) aims to learn transferable knowledge from offline datasets to enhance the learning process for new target tasks. Context-based Reinforcement Learning (RL) adopts a context encoder to expediently adapt the agent to new tasks by inferring the task representation, and then adjusting the policy based on this inferred representation. In this work, we focus on context-based OMRL, specifically on the challenge of learning task representation for OMRL. We conduct experiments that demonstrate that the context encoder trained on offline datasets might encounter distribution shift between the contexts used for training and testing. To overcome this problem, we present a hard-sampling-based strategy to train a robust task context encoder. Our experimental findings on diverse continuous control tasks reveal that utilizing our approach yields more robust task representations and better testing performance in terms of accumulated returns compared to baseline methods. Our code is available at https://github.com/ZJLAB-AMMI/HS-OMRL.
BibTeX
@misc{zhao2023context,
  author = {Chenyang Zhao and Zihao Zhou and Bin Liu},
  title = {{On Context Distribution Shift in Task Representation Learning for Offline Meta RL}},
  year = {2023},
  eprint = {2304.00354},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2304.00354},
}
Figure