Abstract
In this work, we consider the problem of model selection for deep reinforcement learning (RL) in real-world environments. Typically, the performance of deep RL algorithms is evaluated via on-policy interactions with the target environment. However, comparing models in a real-world environment for the purposes of early stopping or hyperparameter tuning is costly and often practically infeasible. This leads us to examine off-policy policy evaluation (OPE) in such settings. We focus on OPE for value-based methods, which are of particular interest in deep RL, with applications like robotics, where off-policy algorithms based on Q-function estimation can often attain better sample complexity than direct policy optimization. Existing OPE metrics either rely on a model of the environment, or the use of importance sampling (IS) to correct for the data being off-policy. However, for high-dimensional observations, such as images, models of the environment can be difficult to fit and value-based methods can make IS hard to use or even ill-conditioned, especially when dealing with continuous action spaces. In this paper, we focus on the specific case of MDPs with continuous action spaces and sparse binary rewards, which is representative of many important real-world applications. We propose an alternative metric that relies on neither models nor IS, by framing OPE as a positive-unlabeled (PU) classification problem with the Q-function as the decision function. We experimentally show that this metric outperforms baselines on a number of tasks. Most importantly, it can reliably predict the relative performance of different policies in a number of generalization scenarios, including the transfer to the real-world of policies trained in simulation for an image-based robotic manipulation task.
Introduction
Evaluating reinforcement learning (RL) policies in real-world settings, such as robotics, presents significant challenges. While on-policy evaluation is straightforward in simulation, it is often prohibitively expensive or dangerous in physical environments. Consequently, off-policy evaluation (OPE) is essential for model selection, hyperparameter tuning, and early stopping.
Traditional OPE methods typically rely on Importance Sampling (IS) or dynamics-based world models. However, IS suffers from high variance in high-dimensional spaces and requires exact action probabilities, while modeling complex visual environments remains an open research problem. This paper introduces Off-Policy Classification (OPC), a novel approach that treats OPE as a classification problem rather than a regression task.
Methodology: The Classification Framing
The core innovation is shifting from predicting a numerical "return" to a classification task. In environments with sparse binary rewards (success or failure), a Q-function's primary role is to distinguish between actions that lead to success and those that lead to failure.
We frame this as a Positive-Unlabeled (PU) learning problem. In a dataset, trajectories ending in success provide "positive" labels. However, for failed trajectories, we do not know exactly when the mistake occurred; these are treated as "unlabeled." By using PU learning techniques, we can estimate the classification error of a policy's Q-function without explicit negative labels. We propose two variants:
- OPC: Uses a hard threshold to classify actions as good or bad.
- SoftOPC: A more stable version that uses raw Q-values directly, avoiding the need for a fixed threshold.
Experimental Setup and Datasets
The authors evaluated OPC across a spectrum of difficulty, including Atari games and robotic manipulation. A critical test was the "reality gap"—the performance difference between policies trained in simulation and their deployment on physical hardware.
The real-world experiments involved seven KUKA robot arms performing grasping tasks on diverse physical objects. This dataset comprised 4,000 real-world episodes, providing a rigorous testbed for the proposed metrics.
Key Results
Experimental results demonstrate that SoftOPC significantly outperforms standard baselines like Temporal-Difference (TD) error and the Sum of Advantages.
- Real-World Correlation: In the physical robotic grasping task, SoftOPC achieved a Spearman rank correlation ($\xi$) of 0.94 and a coefficient of determination ($R^2$) of 0.91. In contrast, the "Sum of Advantages" baseline dropped to a correlation of only 0.50 in the real world.
- Generalization and Early Stopping: OPC metrics were significantly better at identifying when a model was overfitting to training objects. Furthermore, SoftOPC proved capable of tracking policy improvement during training, making it a viable tool for early stopping.
Conclusion
By reframing off-policy evaluation as a classification task, we provide a robust and scalable metric for deep RL in complex, real-world environments. SoftOPC offers a practical solution for model selection in robotics, where safety and cost constraints make traditional evaluation methods infeasible. Future work will explore extending this framework to non-binary, continuous reward structures.
Summary updated