Off-Policy Evaluation via Off-Policy Classification

Alexander Irpan, Kanishka Rao, Konstantinos Bousmalis, Chris Harris, Julian Ibarz, Sergey Levine

In the world of Reinforcement Learning (RL), training a policy in simulation is the easy part. The real headache begins when you try to figure out which of your 100 trained models will actually work on a physical robot. Usually, you’d just run them and see, but in the real world, every "test run" is expensive, slow, and risks breaking a million-dollar arm. This is the challenge of Off-Policy Evaluation (OPE): estimating how well a new policy will perform using only historical data.

A new paper from researchers at Google Brain, DeepMind, and UC Berkeley proposes a clever shift in perspective. Instead of trying to predict exactly how much reward a policy will get, why not just treat evaluation as a classification problem?

TL;DR

This paper introduces Off-Policy Classification (OPC), a method for evaluating RL policies by framing the problem as a "Positive-Unlabeled" classification task. By using the Q-function as a classifier to distinguish between successful and failing actions, the authors provide a metric that reliably ranks models—even when transferring from simulation to the real world—without needing complex environment models or unstable importance sampling.

What Problem Does This Paper Address?

To pick the best model, we need a metric that correlates with real-world performance. Traditionally, researchers use two main tools for OPE:

  1. Importance Sampling (IS): This corrects for the fact that your data was collected by an old policy. However, it explodes in variance with high-dimensional data (like images) and is often mathematically "ill-conditioned" for the deterministic policies common in deep RL.
  2. Model-Based Methods: You build a "world model" to simulate the new policy. But building a model that can accurately predict future video frames of a robot grasping a cluttered bin is an open research problem in itself.

The authors identify a specific, high-value niche: MDPs with continuous action spaces and sparse binary rewards (e.g., a robot either successfully picks up an object or it doesn't). In these settings, standard metrics like Temporal Difference (TD) error often fail to tell you which model is actually better.

Methods

The core innovation is framing OPE as Positive-Unlabeled (PU) learning.

Q-value classification separating positive and negative reward trajectories

In a binary reward environment, we know that any action in a successful trajectory was "feasible." However, if a trajectory failed, we don't know if it was because of one bad action or just bad luck—so we treat those as "unlabeled."

The authors propose two metrics:

  • OPC (Off-Policy Classification): It uses the Q-function as a decision function. It calculates how much higher the Q-values are for "feasible" actions compared to the general distribution of actions in the dataset.
  • SoftOPC: A variation that uses a soft loss function, which the authors found to be slightly more stable in practice.

The beauty of this approach is that it doesn't care about the absolute value of the reward. It only cares that the Q-function can correctly "classify" or rank good actions above bad ones. If a Q-function is good at this classification, the policy derived from it (by taking the argmax action) is likely to succeed.

Datasets

The researchers validated their approach across three distinct domains:

  1. Binary Tree: A "didactic toy" environment to prove the mathematical intuition.
  2. Pong (Atari): Modified to have binary rewards (the episode ends after the first point is scored).
  3. Robotic Grasping (Sim & Real):
Simulated and real-world robotic grasping environments

This is the "heavy lifter." They used a massive dataset of 900,000 simulated episodes and a validation set of 4,000 real-world robotic trials. The goal was to see if the metric could predict which simulation-trained model would perform best on a physical robot arm.

Key Insights and Ideas

  • Ranking > Estimation: For model selection, you don't need to know if a policy will get a 0.85 or 0.87 success rate. You just need to know that Model A is better than Model B. Classification is a much easier task than regression for neural networks in high-dimensional spaces.
  • Q-Functions are Classifiers: In value-based RL, the Q-function’s primary job is to provide a boundary between "good" and "bad" actions. Treating it as a classifier aligns perfectly with its functional role.
  • Independence from the "Behavior Policy": Unlike Importance Sampling, OPC doesn't require you to know the exact probabilities of the policy that originally collected the data. This makes it much more flexible for using "found" historical data.

Key Results

The authors compared OPC/SoftOPC against standard baselines like TD Error and the "Sum of Advantages." * Sim-to-Real Success: In the robotic grasping task, SoftOPC achieved a Spearman rank correlation ($\rho$) of 0.94 and an $R^2$ of 0.91 when predicting real-world performance. * Robustness: While baselines like the "Sum of Advantages" worked well in simulation, they collapsed when applied to real-world data. OPC remained consistent across the "reality gap." * Early Stopping: The authors showed that SoftOPC can track a policy's performance during training, meaning it could be used to decide exactly when to stop training a model before it starts overfitting to the simulator.

Limitations and Future Work

The most significant limitation is the reliance on binary rewards. While many robotics tasks (grasping, door opening, button pressing) fit this mold, many others (like racing or complex navigation) use "dense" continuous rewards.

The authors mention that extending this classification-based logic to non-binary rewards is a promising direction for future work. Additionally, the method assumes the validation dataset is "broad" enough to cover the state space the new policy might visit—a common challenge in all of off-policy evaluation.

Summary updated

Abstract

In this work, we consider the problem of model selection for deep reinforcement learning (RL) in real-world environments. Typically, the performance of deep RL algorithms is evaluated via on-policy interactions with the target environment. However, comparing models in a real-world environment for the purposes of early stopping or hyperparameter tuning is costly and often practically infeasible. This leads us to examine off-policy policy evaluation (OPE) in such settings. We focus on OPE of value-based methods, which are of particular interest in deep RL with applications like robotics, where off-policy algorithms based on Q-function estimation can often attain better sample complexity than direct policy optimization. Furthermore, existing OPE metrics either rely on a model of the environment, or the use of importance sampling (IS) to correct for the data being off-policy. However, for high-dimensional observations, such as images, models of the environment can be difficult to fit and value-based methods can make IS hard to use or even ill-conditioned, especially when dealing with continuous action spaces. In this paper, we focus on the specific case of MDPs with continuous action spaces and sparse binary rewards, which is representative of many important real-world applications. We propose an alternative metric that relies on neither models nor IS, by framing OPE as a positive-unlabeled (PU) classification problem. We experimentally show that this metric outperforms baselines on a number of tasks. Most importantly, it can reliably predict the relative performance of different policies in a number of generalization scenarios, including the transfer to the real-world of policies trained in simulation for an image-based robotic manipulation task.
BibTeX
@inproceedings{irpan2019off,
  author = {Alexander Irpan and Kanishka Rao and Konstantinos Bousmalis and Chris Harris and Julian Ibarz and Sergey Levine},
  title = {{Off-Policy Evaluation via Off-Policy Classification}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2019},
  url = {https://openreview.net/forum?id=vBsYYVhK-6},
}
Figure