OCEAN: Online Task Inference for Compositional Tasks with Context Adaptation

Hongyu Ren, Yuke Zhu, Jure Leskovec, Animashree Anandkumar, Animesh Garg

Abstract

Meta-reinforcement learning (meta-RL) often struggles with compositional tasks that require sequential sub-task execution. We present OCEAN (Online ContExt AdaptatioN), a framework that decouples task inference into global variables for overall goals and local variables for mid-episode sub-task adaptation. By utilizing flexible latent distributions and a recurrent architecture, OCEAN enables agents to navigate complex, multi-stage environments without manual transition labels.

Introduction

Standard meta-RL algorithms typically excel in scenarios with fixed goals but falter when tasks are compositional—requiring a sequence of distinct actions like reaching, grasping, and pulling. Existing models often rely on a single, static latent variable inferred at the start of an episode. This "one-size-fits-all" approach cannot account for mid-episode transitions or the hierarchical nature of complex tasks. Furthermore, simple Gaussian priors are often insufficient for modeling discrete task factors or multi-modal goal distributions.

Methods: The OCEAN Architecture

OCEAN addresses these limitations by introducing a joint latent space that separates task inference into two components:

  1. Local Context Encoder ($q^L_\phi$): A Variational Recurrent Neural Network (VRNN) that performs online sub-task inference. It processes the history of state-action-reward transitions within the current episode to update the agent's belief about the active sub-task.
  2. Global Context Encoder ($q^G_\phi$): This module identifies the overarching task structure by aggregating contexts from previous episodes, such as the set of potential goal locations.
The OCEAN architecture decoupling task inference into global ($z^{Global}$) and local ($z^{Local}_t$) latent variables.

The framework supports diverse latent distributions (Categorical, Dirichlet, Logit-normal), allowing for the incorporation of domain-specific inductive biases. Integrated with Soft Actor-Critic (SAC), OCEAN maintains high sample efficiency and operates in an unsupervised manner, discovering sub-task structures automatically.

Experimental Setup

The framework was evaluated on several continuous control benchmarks modified for multi-stage complexity:

  • 2D Point Robot Navigation: Moving to goals on a semi-circle.
  • MuJoCo Environments: Including Cheetah-Fwd-Back (baseline), Cheetah-Multi-Vel (shifting goal velocities), Cheetah/Humanoid-Multi-Direc (changing directions mid-episode), and Humanoid-Multi-Goal (sequential random goals).

Key Results and Insights

OCEAN demonstrates significant improvements in both sample efficiency and final returns, particularly in high-dimensional environments like Humanoid-Multi-Direc.

Performance comparison of OCEAN against baselines (PEARL, ProMP, RL2, E-MAML) across various compositional tasks.

Dynamic Inference and Disentanglement

The primary advantage of OCEAN is its ability to adapt task identity dynamically. By allowing the latent variable to evolve, the agent can recognize transitions between sub-tasks. Disentangling global "task identity" from local "current step" prevents the confusion inherent in single-variable models.

The Role of Priors and Global Context

Ablation studies confirmed that the choice of latent distribution (e.g., Dirichlet for multi-modal goals) acts as a powerful inductive bias. Additionally, the global context variable is crucial for reducing variance and accelerating learning by narrowing the search space at the start of new episodes.

Limitations and Future Work

While effective, OCEAN introduces computational overhead due to stochastic sampling at every timestep. Future work could explore amortizing these updates or adding auxiliary tasks to predict the timing of sub-task transitions. Additionally, improving "cold start" performance by more effectively leveraging historical context remains an open challenge.

Summary updated

Abstract

Real-world tasks often exhibit a compositional structure that contains a sequence of simpler sub-tasks. For instance, opening a door requires reaching, grasping, rotating, and pulling the door knob...
BibTeX
@inproceedings{ren2020ocean,
  author = {Hongyu Ren and Yuke Zhu and Jure Leskovec and Animashree Anandkumar and Animesh Garg},
  title = {{OCEAN: Online Task Inference for Compositional Tasks with Context Adaptation}},
  booktitle = {Conference on Uncertainty in Artificial Intelligence},
  year = {2020},
}
Figure