Single Episode Policy Transfer in Reinforcement Learning

Jiachen Yang, Brenden K. Petersen, Hongyuan Zha, Daniel Faissol

Abstract

Transfer and adaptation to new unknown environmental dynamics is a key challenge for reinforcement learning (RL). An even greater challenge is performing near-optimally in a single attempt at test time, possibly without access to dense rewards, which is not addressed by current methods that require multiple experience rollouts for adaptation. To achieve single episode transfer in a family of environments with related dynamics, we propose a general algorithm that optimizes a probe and an inference model to rapidly estimate underlying latent variables of test dynamics, which are then immediately used as input to a universal control policy. This modular approach enables integration of state-of-the-art algorithms for variational inference or RL. Moreover, our approach does not require access to rewards at test time, allowing it to perform in settings where existing adaptive approaches cannot. In diverse experimental domains with a single episode test constraint, we report our method significantly outperforms existing adaptive approaches and shows favorable performance against baselines for robust transfer.

Introduction

Standard Reinforcement Learning (RL) agents are often brittle; a robot trained to walk on high-friction carpet may fail immediately when encountering ice. While transfer learning and meta-learning aim to address this, most existing techniques require multiple episodes of trial-and-error to adapt or rely on continuous reward feedback during the test phase.

In many real-world scenarios, such as precision medicine or autonomous flight, we cannot afford multiple "trial" episodes. A treatment plan for a new patient must be effective from the first dose. This paper introduces Single Episode Policy Transfer (SEPT), a framework designed to achieve near-optimal performance in a single attempt by rapidly identifying environmental characteristics without requiring test-time rewards.

Methodology: The SEPT Framework

SEPT decomposes the transfer problem into three modular components that work in sequence to adapt to a new environment instance:

  1. Probe Policy ($\pi_\phi$): At the beginning of an episode, the agent executes a short "probing" sequence (e.g., the first 5% of the episode). This policy is specifically trained to interact with the environment in a way that maximizes the visibility of its unique latent dynamics, such as gravity, friction, or biological parameters.
  2. Inference Model (VAE): The trajectory $\tau_p$ generated by the probe is processed by a Variational Auto-Encoder (VAE). The VAE compresses this trajectory into a latent variable $\hat{z}$, which serves as a signature for the current environment's dynamics.
  3. Universal Control Policy ($\pi_\theta$): This master policy receives both the current state $s$ and the inferred latent signature $\hat{z}$. Having been trained across a wide distribution of environment variations, $\pi_\theta$ can immediately adjust its behavior to the specific dynamics identified by the probe.
The SEPT architecture: A probe policy generates a short trajectory, which a VAE uses to infer latent dynamics for the control policy.

Training and Objectives

A key innovation in SEPT is how the probe policy is trained. Rather than using standard task rewards, the probe is trained to maximize the VAE's objective function—the variational lower bound (L.B.). By treating the L.B. as a reward, the probe learns to be as informative as possible for the inference model. This "machine teaching" approach ensures that the short probing phase provides the most useful data for identifying the environment.

Algorithm 1: The training phase for Single Episode Policy Transfer (SEPT).

Experimental Evaluation

The researchers evaluated SEPT across three distinct domains within the Hidden-Parameter Markov Decision Process (HiP-MDP) framework:

  • 2D Navigation: Agents must reach a goal despite discontinuous dynamics, such as sudden reversals in action effects.
  • Acrobot: A classic control task where link masses and lengths vary between instances.
  • HIV Treatment: A high-dimensional simulation of patient responses to drug therapies. This represents a high-stakes application where adaptation must be both rapid and accurate.

Key Results

  • Efficiency and Speed: In 2D Navigation, SEPT reached the goal in approximately 20 steps, significantly faster than baselines like MAML or robust policies. Furthermore, at test time, SEPT is four orders of magnitude faster than model-based Bayesian methods (BNN) because it relies on a single neural network forward pass rather than expensive "fictional rollouts."
  • Reward-Free Adaptation: Unlike meta-RL methods that require reward signals to calculate adaptation gradients, SEPT adapts using only state transitions ($s, a, s'$). This allows it to function in "reward-sparse" test environments.
  • Clinical Relevance: On the HIV treatment task, SEPT achieved higher cumulative rewards than all competing methods, demonstrating its potential for complex, real-world medical applications.

Discussion and Limitations

SEPT demonstrates that active probing combined with latent inference is a powerful paradigm for single-episode transfer. However, the method currently assumes stationarity—that environmental dynamics remain constant throughout the episode. If dynamics shift mid-episode, the initial probe may become obsolete. Additionally, the "cost of probing" must be considered; in safety-critical environments where any mistake is fatal, a dedicated probing phase might be too risky. Future research may focus on integrating probing more seamlessly into the primary control task to mitigate these risks.

Summary updated

Abstract

Transfer and adaptation to new unknown environmental dynamics is a key challenge for reinforcement learning (RL). An even greater challenge is performing near-optimally in a single attempt at test time, possibly without access to dense rewards, which is not addressed by current methods that require multiple experience rollouts for adaptation. To achieve single episode transfer in a family of environments with related dynamics, we propose a general algorithm that optimizes a probe and an inference model to rapidly estimate underlying latent variables of test dynamics, which are then immediately used as input to a universal control policy. This modular approach enables integration of state-of-the-art algorithms for variational inference or RL. Moreover, our approach does not require access to rewards at test time, allowing it to perform in settings where existing adaptive approaches cannot. In diverse experimental domains with a single episode test constraint, we report our method significantly outperforms existing adaptive approaches and shows favorable performance against baselines for robust transfer.
BibTeX
@inproceedings{yang2020single,
  author = {Jiachen Yang and Brenden K. Petersen and Hongyuan Zha and Daniel Faissol},
  title = {{Single Episode Policy Transfer in Reinforcement Learning}},
  booktitle = {International Conference on Learning Representations},
  year = {2020},
  url = {https://openreview.net/forum?id=UFLrD8O3a9o},
}
Figure