TL;DR
Reinforcement learning (RL) research often uses simulators as proxies for real-world deployment. This paper argues that researchers must clearly distinguish between two distinct goals: "solving a simulator" (achieving peak performance within the simulator itself) and "using a simulator as a proxy" (learning for eventual deployment). Failing to make this distinction leads to misleading conclusions and suboptimal algorithm development.
What Problem Does This Paper Address?
Reinforcement learning (RL) research heavily relies on simulators to develop and test algorithms. While simulators offer a safe and efficient environment for experimentation, they also present a temptation for researchers to optimize solely for simulator performance rather than for generalizability to real-world deployment. This can lead to the development of algorithms that exploit simulator-specific features (like parallel environments or the ability to reset to any state) which are not available or practical in real-world scenarios. The paper highlights that this lack of clear distinction between "solving a simulator" and "using a simulator as a proxy for learning in deployment" can lead to confusion, misinterpretation of results, and ultimately hinder progress in developing RL agents that truly work in the real world.
Methods
The paper doesn't introduce new algorithms but rather proposes a conceptual framework and a call to action for the RL community. It meticulously outlines the differences between two simulator use cases:
- Solving a Simulator: The primary goal is to achieve the highest possible performance within the simulator. This allows for the exploitation of simulator-specific features like running multiple parallel environments simultaneously, resetting the agent to specific states, and simulating multiple outcomes from a single state-action pair.
- Using a Simulator as a Proxy for Learning in Deployment: The simulator is used to mimic the constraints and dynamics of a real-world deployment environment. This means adhering to a single environment interaction, sequential observations, and avoiding simulator-exclusive actions.
The authors analyze how these differing goals impact algorithm design, interaction constraints, and evaluation metrics. They also provide experimental examples to illustrate the issues arising from conflating these two use cases.
Datasets
The paper references several standard RL environments and simulators used for experiments, including:
- Classic Control environments: (e.g., MountainCar-v0)
- MuJoCo: A physics engine for model-based control.
- Arcade Learning Environment (ALE): Used for Atari games.
- MinAtar: A simplified Atari benchmark.
- VizDoom: A simulator for Doom.
No new datasets or benchmarks were introduced in this paper.
Key Insights and Ideas
- Simulator-exclusive actions are a double-edged sword: Features like parallel environments and state resetting drastically speed up learning within a simulator but are often unavailable in real-world deployment. Algorithms designed to exploit these features are not suitable proxies for deployment learning.
- Evaluation metrics matter: "Online return" (rewards received during actual learning) is more representative of deployment performance than "evaluation rollouts" (performance of a frozen policy), which can be artificially inflated by simulator-specific advantages.
- Hyperparameter tuning can be misleading: Standard hyperparameter tuning practices, which involve extensive resets and discarding of trials, are a form of "solving a simulator" and can lead to suboptimal algorithm choices when the goal is proxy learning.
- The "arms race" in benchmarks can obscure true progress: The drive to achieve high scores on benchmarks can incentivize solving simulators rather than developing generalizable agents.
Key Results
The paper presents qualitative and experimental results to support its claims:
- Figure 1: Demonstrates a learning curve that appears successful when solving a simulator (best policy evaluation is high) but indicates poor performance when used as a proxy (online evaluation collapses).
- Figure 2: Shows that PQN, an algorithm that leverages parallel environments, is significantly slower and less performant when restricted to a single environment, highlighting the issue of using simulator-solving algorithms as proxies.
- Figure 3: Illustrates that PPO with a naive resetting mechanism significantly outperforms standard PPO on MountainCar-v0, demonstrating the performance gains missed when not exploiting simulator-specific features for solving a simulator.
- Figure 4: Contrasts online evaluation with deterministic evaluation rollouts for SAC and DQN, showing that rollouts can appear smoother and higher, potentially masking issues with exploration during actual learning.
Limitations and Future Work
- Scope: The paper focuses on two primary simulator use cases ("solving a simulator" and "using as a proxy for deployment") and acknowledges that other use cases exist (e.g., sim2real).
- Call to Action: The authors call for researchers to clearly state their problem setting and adhere to the constraints implied by their chosen use case. They also suggest that top venues should be more receptive to papers applying RL to real-world systems.
- Future Work: The paper invites other communities using research simulators to discuss similar distinctions and encourages the development of novel empirical approaches that allow for parallel environments without compromising empirical conclusions for proxy learning. They also suggest explicitly labeling different versions of algorithms (e.g., PPO-v1 vs. PPO-MultiEnv) to clarify their intended use.
Summary updated