Abstract
In this work, we argue for the importance of an online evaluation budget for a reliable comparison of deep offline RL algorithms. First, we delineate that the online evaluation budget is problem-dependent, where some problems allow for less but others for more. And second, we demonstrate that the preference between algorithms is budget-dependent across a diverse range of decision-making domains such as Robotics, Finance, and Energy Management. Following the points above, we suggest reporting the performance of deep offline RL algorithms under varying online evaluation budgets. To facilitate this, we propose to use a reporting tool from the NLP field, Expected Validation Performance. This technique makes it possible to reliably estimate expected maximum performance under different budgets while not requiring any additional computation beyond hyperparameter search. By employing this tool, we also show that Behavioral Cloning is often more favorable to offline RL algorithms when working within a limited budget.
Introduction
Offline Reinforcement Learning (RL) aims to learn effective policies from static datasets without environment interaction. However, a "secret" cost often plagues research: the extensive online testing required to select optimal hyperparameters. In practice, researchers often train numerous models and select the best-performing one based on simulator results—a luxury not available in real-world domains like healthcare or finance.
This paper addresses the lack of standardized reporting for algorithms under realistic deployment constraints. In many applications, a practitioner may only have the budget to test a handful of policies. If an algorithm achieves high peak performance but is extremely sensitive to hyperparameters, it may be a poor choice compared to a more stable, albeit lower-peaking, alternative.
The Expected Online Performance (EOP) Framework
To bridge the gap between research and practice, the authors introduce Expected Online Performance (EOP), adapted from the "Expected Validation Performance" metric in Natural Language Processing.
Instead of a single "best" score, EOP provides a curve showing the expected maximum performance as a function of the online budget ($B$)—the number of policies allowed for real-world testing. Using order statistics and Empirical Cumulative Distribution Functions (ECDF), EOP estimates the average performance of the best policy if $B$ policies were randomly sampled from a pool of $N$ trained models. This allows researchers to quantify tunability and robustness without additional computation beyond standard hyperparameter sweeps.
Experimental Evaluation
The study evaluates EOP across diverse domains, including standard robotics benchmarks (D4RL, NeoRL), energy management (CityLearn), and stock trading (FinRL).
Budget-Dependent Rankings
A key finding is that algorithm rankings are not static; they shift significantly based on the available budget. An algorithm that dominates at a high budget ($B=30$) might be the worst performer when only one or two evaluations are permitted.
The Robustness of Behavioral Cloning
Surprisingly, simple Behavioral Cloning (BC) often outperforms state-of-the-art offline RL algorithms like CQL or PLAS when the budget is small ($B < 5$). While advanced algorithms have higher potential "peaks," they are often "needles in a haystack." BC provides a much higher "floor," making it a safer and more reliable choice for practitioners with limited testing opportunities. For instance, in the Hopper-v3 environment, BC maintained superior expected performance until the budget reached at least four online evaluations.
Offline Policy Selection (OPS)
The authors also investigated heuristics for picking policies without any online testing. They found that the "Action Difference" metric—measuring a policy's deviation from the original data—served as an effective conservative regularizer, performing well across various tasks when the budget was tight.
Conclusion
This work serves as a critical reminder that offline RL performance must be contextualized by the cost of evaluation. By adopting EOP curves, the community can provide more transparent and actionable insights for real-world deployment. Future work may extend this framework to "Deployment-Constrained RL," where data from initial evaluations is used to iteratively refine training.
Summary updated