Zero-Shot Off-Policy Learning

Arip Asadulaev, Maksim Bobrin, Salem Lahlou, Dmitry V. Dylov, Fakhri Karray, Martin Takáč

Introduction

The current trajectory of artificial intelligence is increasingly leaning toward Behavioral Foundation Models (BFMs)—generalist reinforcement learning (RL) agents trained on massive, reward-free datasets. These models are designed to be "prompted" to perform specific tasks at test time. However, they face a critical bottleneck: distributional shift.

When a BFM is prompted with a novel task, the resulting policy may attempt to visit states or execute actions rarely seen in the training data. Standard value-estimation methods are notoriously overly optimistic about these out-of-distribution (OOD) regions, leading agents to make confident but catastrophic errors. While traditional offline RL techniques (such as the DICE family) mitigate this by calculating density ratios to constrain the policy within the data's support, they require task-specific optimization that is too slow for true zero-shot adaptation.

Zero-Shot Off-Policy Learning (ZOL) introduces a novel framework that allows RL agents to adapt to new tasks instantly, without any additional training or environment interaction. By discovering a theoretical bridge between successor measures and stationary density ratios, ZOL enables agents to correct for distributional shifts on the fly, significantly enhancing the robustness and reliability of BFMs.

Methodology: Bridging Successor Measures and Density Ratios

The core innovation of ZOL rests on a key theoretical insight: Successor Measures (SM), which describe the expected future state visitation of an agent, are mathematically linked to stationary density ratios ($w_\pi$). Traditionally, successor measures are used to predict future rewards; ZOL repurposes them to calculate how much a candidate policy deviates from the training distribution.

ZOL leverages the Forward-Backward (FB) representation framework, which decomposes the successor density into a "forward" embedding (representing the policy's intent) and a "backward" embedding (representing the state-action basis of the dataset). Because of the bilinear structure of FB, ZOL produces a low-rank estimator for density ratios, making the correction computationally cheap enough to run on the fly during the prompt-adaptation phase.

The ZOL Algorithm

The ZOL framework operates in three primary phases:

  1. Pretraining: Standard FB embeddings are trained on a broad, reward-free dataset.
  2. Zero-Shot Inference: Upon receiving a new task, ZOL uses the pretrained FB embeddings to estimate the stationary density ratio $w_\pi \approx (1 - \gamma) W_\pi^\top B(s, a)$.
  3. Latent Optimization: Rather than simply selecting a task latent $z$ that appears optimal, ZOL optimizes $z$ at test time using a regularized objective. This optimization is entirely training-free and interaction-free, relying on:
  4. Reward Centering: Prevents the agent from remaining idle in environments with constant reward offsets.
  5. $\chi^2$ Penalty: Constrains importance weights from exploding, keeping the policy firmly grounded in the supported data distribution.
  6. Trust-Region Constraint: Ensures the search remains near the initial task specification.

This approach addresses the "Inference Gap." The authors argue that zero-shot failures are often not due to a lack of knowledge within the model, but rather a failure of the inference procedure itself. Standard methods often select a task latent based on where data is densest, missing the actual task solution located in a sparser, yet supported, region.

Experiments and Results

ZOL was rigorously evaluated across a diverse suite of challenging benchmarks, including ExORL (16 continuous control tasks using reward-free data), OGBench (long-horizon reasoning and temporal stitching), SMPL Humanoid (a 358-dimensional state space tracking complex human MoCap data), and a custom 2D Toy "Donut" Domain to visualize inference recovery in low-data regions.

Key Findings: * Superior Zero-Shot Performance: On the ExORL benchmark, ZOL achieved a 10% average improvement over vanilla FB and other state-of-the-art BFMs like TD-JEPA and HILP. * Outperforming On-Policy Methods: Remarkably, in the long-horizon OGBench tasks, ZOL (a strictly offline, zero-shot method) outperformed online adaptation methods like LoLA and ReLA, which are permitted environment interaction. ZOL achieved an average success rate of 0.47 compared to LoLA's 0.42. * High-Dimensional Scaling: ZOL demonstrated dramatic gains in the complex SMPL Humanoid environment. For instance, on the RAISEARMS-H-L task, performance surged from a return of 42.55 (vanilla FB) to 241.40 (ZOL). * Robustness: Hyperparameter ablations confirmed ZOL's stability, maintaining high performance across various optimization steps and regularization coefficients.

Limitations and Future Directions

While ZOL presents a significant leap in zero-shot adaptation, it is not without limitations. Its effectiveness is inherently bound by a representational bottleneck; if the pretraining phase fails to capture the environment's underlying dynamics, ZOL cannot invent new behaviors. Furthermore, because ZOL explicitly penalizes out-of-distribution regions, it faces a conservatism trade-off, potentially hindering performance on tasks that require exploring entirely novel parts of the state space.

Future work aims to extend the ZOL framework to stochastic and partially observable domains (POMDPs). Additionally, incorporating uncertainty-aware occupancy correction could help the model better navigate noisy or exceptionally sparse data distributions.

Summary updated

Abstract

Off-policy learning methods seek to derive an optimal policy directly from a fixed dataset of prior interactions. This objective presents significant challenges, primarily due to the inherent distributional shift and value function overestimation bias. These issues become even more noticeable in zero-shot reinforcement learning, where an agent trained on reward-free data must adapt to new tasks at test time without additional training. In this work, we address the off-policy problem in a zero-shot setting by discovering a theoretical connection of successor measures to stationary density ratios. Using this insight, our algorithm can infer optimal importance sampling ratios, effectively performing a stationary distribution correction with an optimal policy for any task on the fly. We benchmark our method in motion tracking tasks on SMPL Humanoid, continuous control on ExoRL, and for the long-horizon OGBench tasks. Our technique seamlessly integrates into forward-backward representation frameworks and enables fast-adaptation to new tasks in a training-free regime. More broadly, this work bridges off-policy learning and zero-shot adaptation, offering benefits to both research areas.
BibTeX
@inproceedings{asadulaev2026zero,
  author = {Arip Asadulaev and Maksim Bobrin and Salem Lahlou and Dmitry V. Dylov and Fakhri Karray and Martin Takáč},
  title = {{Zero-Shot Off-Policy Learning}},
  booktitle = {International Conference on Machine Learning},
  year = {2026},
  url = {https://openreview.net/forum?id=HFpAMjHiZm},
}
Figure