Abstract
Off-policy learning methods seek to derive an optimal policy directly from a fixed dataset of prior interactions. This objective presents significant challenges, primarily due to the inherent distributional shift and value function overestimation bias. These issues become even more noticeable in zero-shot reinforcement learning, where an agent trained on reward-free data must adapt to new tasks at test time without additional training. In this work, we address the off-policy problem in a zero-shot setting by discovering a theoretical connection of successor measures to stationary density ratios. Using this insight, our algorithm can infer optimal importance sampling ratios, effectively performing a stationary distribution correction with an optimal policy for any task on the fly. We benchmark our method in motion tracking tasks on SMPL Humanoid, continuous control on ExoRL, and for the long-horizon OGBench tasks. Our technique seamlessly integrates into forward-backward representation frameworks and enables fast-adaptation to new tasks in a training-free regime. More broadly, this work bridges off-policy learning and zero-shot adaptation, offering benefits to both research areas.
Introduction: The Challenge of Zero-Shot Adaptation
The rise of Behavioral Foundation Models (BFMs) promises a "GPT moment" for robotics: pretrain an agent on massive amounts of unlabeled, reward-free interaction data, and then "prompt" it to perform any task at test time. However, these models face a fundamental hurdle: distributional shift.
When a BFM is prompted for a new task (e.g., "run backward"), the resulting policy might attempt to visit states or take actions that were rare in the original training dataset. Because the model hasn't seen these regions, its value estimates become unreliable, often leading to "OOD (Out-of-Distribution) greedification"—the agent thinks a certain path is optimal simply because it is overestimating the value of unknown states. Traditional offline RL fixes this with expensive retraining, but in a zero-shot setting, we need a way to correct this bias on the fly, without new data or gradient updates.
Methods: Zero-Shot Off-Policy Learning (ZOL)
This work introduces Zero-Shot Off-Policy Learning (ZOL), which leverages a novel theoretical link between successor measures and density ratios to steer agents toward high-performing, data-supported behaviors without requiring additional training.
The authors build upon the Forward-Backward (FB) representation framework, where a model learns "Forward" embeddings ($F$) predicting future state visitations and "Backward" embeddings ($B$) representing state-action features. The key innovation is proving that the stationary density ratio—a measure of how much a target policy deviates from training data—is implicitly contained within these FB embeddings.
The ZOL algorithm follows three steps:
- Pretraining: Train standard FB embeddings on an unlabeled dataset.
- Latent Inference: Infer an initial task latent $z$ for a new reward function or goal.
- Zero-Shot Optimization: Perform a "latent search" to optimize $z$, maximizing task return while using the discovered density ratio to reweight supervision and regularize the search (using a Chi-square penalty).
This optimization happens entirely in the latent space, requiring zero environment interaction and zero weight finetuning.
Experimental Results
ZOL was evaluated across several challenging benchmarks, including ExORL (continuous control), OGBench (long-horizon tasks), and SMPL Humanoid (high-dimensional motion tracking).
Key findings include:
- ExORL: ZOL consistently outperformed the baseline FB model and even beat "online" adaptation methods like LoLA. For instance, on the Quadruped "Stand" task, ZOL achieved a return of 904 compared to the baseline's 876.
- OGBench: ZOL reached an average success rate of 0.47, surpassing both the base FB model (0.45) and ReLA (0.39), demonstrating its effectiveness in long-horizon reasoning.
- Humanoid Motion Tracking: In the high-dimensional HumEnv, ZOL showed dramatic improvements in tracking accuracy across diverse motion styles.
Discussion and Future Work
The most significant insight is that successor measures are essentially scaled versions of stationary distribution corrections. This turns a complex distribution-matching problem into a simple inner product of pretrained embeddings. The paper argues that BFM performance gaps often stem from inference procedures failing to find existing behaviors within the model's capacity.
While powerful, ZOL is limited by the quality of the underlying FB embeddings; it cannot "invent" dynamics not captured during pretraining. Future work may extend this to stochastic environments and incorporate uncertainty-aware occupancy corrections to better handle regions where the model's knowledge is sparse.
Summary updated