Showing Your Offline Reinforcement Learning Work: Online Evaluation Budget Matters

Vladislav Kurenkov, Sergey Kolesnikov

Offline Reinforcement Learning (ORL) holds immense promise for domains where interacting with the real-world environment is dangerous, expensive, or unethical—such as healthcare, robotics, and industrial control. By learning from a fixed, pre-recorded dataset, ORL aims to deliver effective policies without the need for trial-and-error in the wild.

However, a critical flaw exists in how these algorithms are currently evaluated and compared: the "best" algorithm often depends entirely on your online evaluation budget.

The Infinite Budget Paradox

Current research typically reports the performance of the single "best" set of hyperparameters found through extensive online testing. This creates a paradox: if a practitioner actually had the budget to test dozens of policies online to find the optimal one, they likely wouldn't need a strictly "offline" approach in the first place.

In reality, a researcher or engineer might only have the budget to test one or two policies in the real world. This gap between idealized "infinite budget" research reports and the "limited budget" reality of deploying offline RL makes it difficult to know which algorithms actually work best in practice.

A Budget-Aware Evaluation Pipeline

To address this, researchers propose a more realistic evaluation pipeline that explicitly accounts for the online evaluation budget ($B$)—the number of policies you can afford to test in the real environment.

Standard vs. Budget-Aware Evaluation Pipeline

To make this practical, the authors adapt a technique from Natural Language Processing called Expected Validation Performance, renaming it Expected Online Performance (EOP) for the RL context.

EOP uses statistical estimators to calculate the expected maximum performance you would achieve if you randomly sampled $B$ policies from your total pool of $N$ hyperparameter trials. This allows researchers to generate a curve showing how an algorithm performs as the budget increases, requiring only a single round of hyperparameter tuning.

Key Insights: Rankings are Budget-Dependent

Evaluating algorithms across diverse domains—including NeoRL (robotics), CityLearn (smart grids), FinRL (finance), and the Industrial Benchmark—reveals that algorithm rankings are highly sensitive to the evaluation budget.

An algorithm that appears to be a "winner" when you can test 30 policies might actually be a poor choice if you only have a budget of 1.

Expected Online Performance (EOP) across Robotic Tasks

As seen in the EOP graphs above, performance curves for different algorithms frequently cross as the budget increases:

  • The "Simplicity" Advantage: Behavioral Cloning (BC)—often treated as a lowly baseline—is frequently the best choice for practitioners with very tight evaluation budgets. It is more stable and less sensitive to hyperparameter choices than complex "conservative" offline RL algorithms.
  • Robotics: In the Walker2d task, CQL is clearly preferred over PLAS, but only if you have the budget to evaluate at least 10 policies online. Below that threshold, the choice is much less clear.
  • Finance & Energy: In the FinRL and CityLearn domains, BC remains highly competitive or even superior to state-of-the-art offline RL methods under restricted budgets.

Rethinking Algorithm Utility

These findings suggest that hyperparameter tuning is a core part of the algorithm itself. An algorithm that requires 100 trials to find a good policy is fundamentally less "deployment-efficient" than one that works well in 5. Furthermore, even the Offline Policy Selection (OPS) methods used to choose which policy to test are sensitive to the budget, with no single selection method proving to be a universal winner.

Limitations and Future Directions

While EOP provides a much-needed reality check for ORL evaluation, it currently focuses on maximum performance. In high-stakes fields like autonomous driving, a practitioner might care more about the minimum performance (avoiding a catastrophic failure) than the expected maximum. Future work could extend this framework to account for risk sensitivity, non-uniform policy selection, and "deployment-constrained RL," where data from the $B$ evaluated policies is used to iteratively improve the model.

By adopting budget-aware metrics like EOP, the RL community can bridge the gap between theoretical algorithm design and practical, real-world deployment.

Summary updated

Abstract

In this work, we argue for the importance of an online evaluation budget for a reliable comparison of deep offline RL algorithms. First, we delineate that the online evaluation budget is problem-de...
BibTeX
@inproceedings{kurenkov2022showing,
  author = {Vladislav Kurenkov and Sergey Kolesnikov},
  title = {{Showing Your Offline Reinforcement Learning Work: Online Evaluation Budget Matters}},
  booktitle = {International Conference on Machine Learning},
  year = {2022},
  url = {https://openreview.net/forum?id=L0BRtIzDVP-},
}
Figure