Measuring Progress in Deep Reinforcement Learning Sample Efficiency

Florian E. Dorner

Introduction

Deep Reinforcement Learning (DRL) has achieved remarkable milestones, from mastering complex board games like Go to achieving superhuman performance in video games. However, these successes often come at the cost of extreme "sample inefficiency," requiring millions or even billions of interactions with an environment to learn tasks that humans master in minutes. While research often focuses on achieving new state-of-the-art (SOTA) scores, this paper by Florian E. Dorner investigates whether we are actually making algorithmic progress in learning more from less data.

Methodology: The Constant Performance Metric

To quantify progress, the author shifts the focus from peak performance to the cost of achieving a specific milestone. By adopting a "constant performance" metric, the study tracks how many samples (frames) subsequent algorithms require to reach the performance level of the original 2015 DQN agent.

The methodology involves:

  • Sample Efficiency ($1/S$): Where $S$ is the number of samples needed to reach a fixed score threshold.
  • Data Extraction: Manual extraction of training curves from dozens of published DRL papers to build a longitudinal dataset.
  • Trend Analysis: Fitting exponential curves to SOTA data points over time to determine "doubling times"—the period required for algorithms to become twice as efficient.

Key Insights: The SOTA Paradox

A significant observation in DRL research is the "SOTA Paradox." While algorithms are becoming more efficient at reaching baseline performance, the most advanced agents often consume significantly more total data to push the boundaries of what is possible.

Figure 1: (a) Total frames used to train various ALE agents over time. (b) Median human-normalized scores achieved by these agents. While scores are rising, the total data consumption for top-tier agents has also increased.

As shown in Figure 1, there is a clear upward trend in both the scores achieved and the total frames used by flagship agents like MuZero and Agent57. This suggests that while we are raising the floor of efficiency, we are also pushing the ceiling of performance through increased scale.

Key Results: Exponential Efficiency Gains

Despite the increase in total data usage for peak performance, the underlying algorithmic efficiency is improving at an exponential rate. Across various benchmarks, the time it takes for an algorithm to require only half as much data to reach the same score is remarkably short, often outstripping Moore’s Law.

Figure 5: (a) Exponential decrease in frames required to reach DQN-level performance on Atari (ALE). (b) The sample/performance Pareto front shifting over time, showing simultaneous improvements in efficiency and peak performance.

The study identifies specific doubling times for sample efficiency:

  • Atari (ALE): Doubling every 10 to 18 months.
  • State-based Continuous Control: Doubling every 5 to 24 months.
  • Pixel-based Continuous Control: Doubling every 4 to 9 months.

These gains are driven by fundamental algorithmic shifts, such as the move to off-policy learning, the integration of auxiliary objectives, and more robust hyperparameter optimization.

Limitations and Future Work

The analysis acknowledges several limitations. The reliance on manually extracted data from published plots introduces noise. Furthermore, the metric only accounts for the samples used by the final trained agent, ignoring the substantial "R&D tax" of samples consumed during hyperparameter tuning and algorithmic development. Future work should aim for standardized reporting of training data to facilitate more precise meta-analyses and explore the efficiency of sim-to-real transfer and offline reinforcement learning.

Conclusion

Algorithmic progress in DRL is real and exponential. We are not merely "brute-forcing" better results with more compute; we are developing fundamentally more efficient ways to learn from environmental interactions. This rapid improvement suggests that DRL may soon become viable for data-constrained real-world applications like robotics and autonomous systems.

Summary updated

Abstract

Sampled environment transitions are a critical input to deep reinforcement learning (DRL) algorithms. Current DRL benchmarks often allow for the cheap and easy generation of large amounts of samples such that perceived progress in DRL does not necessarily correspond to improved sample efficiency. As simulating real world processes is often prohibitively hard and collecting real world experience is costly, sample efficiency is an important indicator for economically relevant applications of DRL. We investigate progress in sample efficiency on Atari games and continuous control tasks by comparing the amount of samples that a variety of algorithms need to reach a given performance level according to training curves in the corresponding publications. We find exponential progress in sample efficiency with estimated doubling times of around 10 to 18 months on Atari, 5 to 24 months on state-based continuous control and of around 4 to 9 months on pixel-based continuous control depending on the specific task and performance level.
BibTeX
@misc{dorner2021measuring,
  author = {Florian E. Dorner},
  title = {{Measuring Progress in Deep Reinforcement Learning Sample Efficiency}},
  year = {2021},
  url = {https://openreview.net/forum?id=zlj3SgVSMP},
}
Figure