Introduction
In traditional Reinforcement Learning (RL), agents typically require thousands of interactions to master a task. While transfer learning aims to accelerate this process by applying knowledge from one task to another, most existing methods still require multiple "rollouts" (episodes) to adapt to new dynamics. In many real-world applications, such as precision medicine or robotics, multiple attempts are not feasible; a treatment strategy or a robot's movement must be effective immediately. This paper introduces SEPT (Single Episode Policy Transfer), a framework designed for "one-shot" adaptation where an agent must perform near-optimally in its very first encounter with a new environment instance, often without immediate reward feedback.
The SEPT Framework
SEPT addresses the challenge of single-episode transfer by decoupling environment identification from task execution. The framework consists of three modular components:
- Optimized Probing ($\pi_\phi$): At the start of a test episode, the agent executes a specialized "probe policy" for a short duration (e.g., the first 5% of the episode). This policy is specifically trained to behave in ways that reveal the environment's unique characteristics, such as friction or drug resistance.
- Variational Inference (VAE): The trajectory from the probe phase is processed by a Variational Auto-Encoder (VAE). A bidirectional LSTM encoder compresses this trajectory into a latent vector $\hat{z}$, representing the "hidden personality" of the environment.
- Universal Control Policy ($\pi_\theta$): For the remainder of the episode, the agent switches to a control policy conditioned on both the current state $s$ and the inferred latent vector $\hat{z}$.
Unlike meta-learning methods like MAML, SEPT is "reward-free" at test time; it adapts based purely on observed state transitions during the probe. The probe policy itself is trained using the VAE's variational lower bound as a reward signal, essentially teaching the agent how to perform "active sensing" to best interrogate its environment.
Experimental Evaluation
The researchers evaluated SEPT across three benchmark domains with distinct hidden dynamics:
- 2D Navigation: Features discontinuous dynamics where actions might be flipped and wind directions vary.
- Acrobot: A classic control problem where the masses and lengths of pendulum links vary continuously.
- HIV Treatment: A high-dimensional medical simulation where a patient's internal response to drugs is governed by 12 hidden variables, representing a complex precision medicine challenge.
Key Results and Insights
SEPT demonstrated significant improvements in both performance and efficiency over standard baselines. In the 2D Navigation task, SEPT solved the environment in roughly 20 steps, compared to 50 steps for standard baselines. Furthermore, SEPT was found to be four orders of magnitude faster than model-based baselines at test time because adaptation requires only a single forward pass through a neural network rather than an expensive optimization process.
In the HIV domain, SEPT significantly outperformed other adaptive and robust baselines. By accurately identifying patient-specific dynamics early in the episode, SEPT achieved much higher cumulative rewards (see Figure 2). Ablation studies confirmed that the optimized probe is crucial; using a standard control policy to gather data for the VAE (SEPT-NP) resulted in significantly worse performance in complex environments.
Discussion and Limitations
While SEPT provides a powerful blueprint for one-shot RL, it currently assumes that environmental dynamics remain constant throughout a single episode. The authors proposed DynaSEPT to handle shifting dynamics, though it remains more complex. Additionally, the method relies on the assumption that critical environmental differences are identifiable early in an episode. If a dynamic only manifests late, a short initial probe may fail to capture it. Despite these constraints, SEPT offers a robust solution for high-stakes scenarios where traditional trial-and-error is not an option.
Summary updated