TL;DR
This paper introduces the first comprehensive empirical benchmark for low-switching-cost Reinforcement Learning (RL), a setting where the policy used to interact with the environment cannot be updated frequently due to real-world costs or risks. The authors demonstrate that while theoretical approaches often rely on complex "information gain" metrics, a simpler feature-based similarity criterion provides the best balance between high rewards and minimal policy updates across medical, gaming, and robotic tasks.
What Problem Does This Paper Address?
In standard RL, we typically assume that the agent can update its policy after every single interaction with the environment. However, in many high-stakes applications, this is impossible:
- Medicine: Changing a treatment protocol requires rigorous expert review and approval.
- Robotics: Frequent policy shifts can be risky or cause mechanical wear during an operation.
- Hardware: Evaluating a new policy on specialized hardware like FPGAs is time-consuming and expensive.
This creates a need for "low-switching-cost RL," where the goal is to achieve optimal performance while minimizing the number of times the "deployed" policy is changed. While there is a wealth of theoretical literature providing regret bounds for this setting in simple (tabular) environments, there has been a lack of systematic empirical study on how these methods perform in complex, high-dimensional deep RL scenarios.
Methods
The authors formalize the problem by maintaining two policies: an online policy ($\pi_{onl}$) that is updated constantly via standard RL algorithms, and a deployed policy ($\pi_{dep}$) that actually interacts with the environment. The core challenge is designing a switching criterion ($\mathcal{J}$) to decide when to sync $\pi_{dep}$ with $\pi_{onl}$.
They evaluate four categories of criteria:
- Non-adaptive: Switching every $n$ steps (e.g.,
FIX_1000). - Policy-based: Switching when the action distributions of the two policies diverge significantly (using KL-divergence or action mismatch ratios).
- Feature-based (Novel): Switching based on the cosine similarity of the feature representations (the final hidden layer of the Q-network) between the two policies.
- Information-gain (Theoretical): Inspired by theoretical proofs, these switch when the agent encounters "sufficiently new" information, measured by state-action visitation counts or doubling the eigenvalues of the feature covariance matrix.
Datasets
The authors tested these criteria across three distinct domains to ensure generality:
- GYMIC: A medical simulation environment for sepsis treatment based on real-world patient data (MIMIC-III).
- Atari 2600: A suite of 56 games (using Rainbow DQN) to test discrete control from pixel inputs.
- MuJoCo: 6 continuous control locomotion tasks (using Soft Actor-Critic).
They also introduced a new evaluation metric: Reward-threshold Switching Improvement (RSI). RSI quantifies the logarithmic improvement in switching cost compared to a standard "always-switch" baseline, provided the agent maintains at least 80% of the baseline's reward.
Key Insights and Ideas
- The Theory-Practice Gap: Theoretical methods based on "information gain" (like visitation counts) performed poorly. In high-dimensional spaces, these methods either switch too often (becoming standard RL) or fail to capture the nuances of deep neural network training.
- Feature Stability: A major "aha moment" was discovering that neural network features change much more smoothly than the final action distribution. By triggering updates based on feature similarity, the agent avoids "jittery" policy switches caused by minor fluctuations in the output layer.
- The Power of Simplicity: The non-adaptive
FIX_1000baseline was surprisingly competitive. This suggests that in many deep RL tasks, the exact timing of a switch is less important than simply ensuring enough data is collected between updates.
Key Results
- Feature-based Superiority: The feature-based criterion consistently achieved the highest RSI scores across all domains (9.63 in GYMIC, 3.61 in Atari, and 8.20 in MuJoCo).
- Cost Reduction: In MuJoCo tasks, the feature-based method reduced the number of policy switches by orders of magnitude while maintaining nearly identical sample efficiency and final reward compared to unconstrained SAC.
- Atari Performance: In the 56 Atari games, the feature-based approach was the only adaptive method to consistently outperform the strong
FIX_1000baseline.
Limitations and Future Work
The authors note that while the feature-based method works well, it lacks the formal provable guarantees found in the (less effective) theoretical methods. They suggest that future research should focus on bridging this gap—developing criteria that are both practically effective in large state spaces and theoretically sound.
Additionally, they observed that even the best adaptive methods don't "massively" outperform the simple non-adaptive baseline, indicating there is still significant room to design more "intelligent" switching triggers that can better sense when a policy update is truly necessary for progress.
Summary updated