Reducing the Cost of Cycle-Time Tuning for Real-World Policy Optimization

Homayoon Farrahi, A. Rupam Mahmood

TL;DR

This paper tackles the challenge of hyperparameter tuning in reinforcement learning (RL) for real-world robotics, specifically when the action cycle time (how often an action is taken) is varied. They show that standard RL algorithms like PPO and SAC, when using default hyperparameters, often fail to learn or perform poorly with different cycle times. The authors propose novel "cycle-time-aware" hyperparameter adaptation methods that significantly reduce tuning costs and improve robustness.

What Problem Does This Paper Address?

In reinforcement learning, especially for continuous control tasks like robotics, actions are typically taken at discrete, fixed intervals. This interval is called the "action-cycle time" ($dt$). While simulated environments often have a default, well-chosen cycle time, real-world robotics applications frequently require adjusting this parameter to match the robot's capabilities or the task's demands. A major hurdle is that changing the cycle time often necessitates re-tuning the algorithm's hyperparameters (e.g., learning rates, batch sizes, discount factors). This re-tuning process is extremely time-consuming and costly, especially for physical robots. The paper addresses this by investigating how standard RL algorithms perform with varying cycle times using default hyperparameters and proposes methods to make these algorithms more robust to cycle time changes without extensive re-tuning.

Methods

The core of the paper's contribution lies in proposing and evaluating "cycle-time-aware" (or $\delta t$-aware) hyperparameter adaptation strategies.

  • Baseline Investigation: The authors first analyze the performance of Proximal Policy Optimization (PPO) and Soft Actor-Critic (SAC) using their widely accepted baseline hyperparameters across a range of action cycle times.
  • Hyperparameter Scaling: For PPO, they propose scaling the batch size ($b$) and mini-batch size ($m$) inversely proportional to the cycle time. This aims to keep the "batch time" (the real-time duration to collect a batch) consistent. They also explore scaling the discount factor ($\gamma$) and trace-decay parameter ($\lambda$) by $\delta t / \delta t_0$ (where $\delta t_0$ is the initial cycle time), but find this can be detrimental for smaller cycle times.
  • Adaptive Discount Factor for SAC: For SAC, they propose two approaches for adapting the discount factor $\gamma$:
  • Tuning $\gamma$ at an initial cycle time ($\delta t_0$) and then scaling it for other cycle times ($\gamma_{\delta t,1}$).
  • Tuning $\gamma$ at an initial cycle time and keeping it constant for other cycle times ($\gamma_{\delta t,2}$).
  • Experimental Setup: To ensure consistency across different cycle times, they fix the underlying physics engine's time step ($\delta t_{env}$) and simulate larger cycle times as integer multiples of this fixed step.

Datasets

The experiments were conducted on:

  • Simulated Environments:
    • Reacher Task (using PyBullet's ReacherBulletEnv-v0)
    • Double Pendulum Task (using PyBullet's InvertedDoublePendulumBulletEnv-v0)
  • Real-World Robotic Task:
    • Real-Robot Reacher Task (using the UR-Reacher-2 robot)

The paper mentions that their implementation of the tasks and experiments are publicly available.

Key Insights and Ideas

  • Default Hyperparameters are Fragile: The most striking finding is that standard, widely-used baseline hyperparameters for PPO and SAC are highly sensitive to the action cycle time. PPO, in particular, can completely fail to learn when the cycle time deviates from the default.
  • Tuning is Essential, but Costly: While tuning hyperparameters for each specific cycle time yields much better performance than baselines, this is prohibitively expensive for real-world applications.
  • Batch Time Consistency is Key: Scaling batch and mini-batch sizes inversely with cycle time to maintain a consistent "batch time" appears to be a crucial insight for PPO, leading to more stable learning across different cycle times.
  • Discount Factor Adaptation is Nuanced: For SAC, simply scaling the discount factor $\gamma$ isn't always optimal. Tuning $\gamma$ at an initial cycle time is essential, and the best adaptation strategy depends on the specific task and cycle time range.
  • Real-World Validation: The proposed methods demonstrate effectiveness not only in simulation but also on a real robotic arm, highlighting their practical relevance.

Key Results

  • PPO Failure: With baseline hyperparameters, PPO failed to learn at a cycle time of 4ms on the Reacher task.
  • Performance Improvement: The proposed $\delta t$-aware hyperparameter strategies for both PPO and SAC performed at least as well as, and often significantly better than, their respective baseline hyperparameters across various cycle times.
  • No Learning Failure: The proposed methods did not result in learning failure for any tested cycle time.
  • Reduced Tuning Burden: The $\delta t$-aware approaches require an initial tuning on a new task at a specific cycle time, but then allow for seamless transfer to different cycle times without further extensive tuning. This is a substantial reduction in cost compared to re-tuning for every cycle time.
  • Real-Robot Success: On the Real-Robot Reacher task, the $\delta t$-aware PPO hyperparameters recovered performance lost by baselines at smaller cycle times, and the $\delta t$-aware SAC hyperparameters significantly improved performance compared to baselines at a longer cycle time (120ms).

Limitations and Future Work

  • Initial Tuning Still Required: The proposed methods still require an initial tuning phase on a new task at a chosen cycle time. The authors acknowledge this is a "negligible" improvement compared to extensive tuning for each cycle time, but it's not zero-cost.
  • Task Dependence: While validated on several tasks, the effectiveness of specific scaling factors or tuning strategies might vary across different robotic tasks and environments. The paper notes that the baseline $\gamma$ for SAC might not be universally effective across different environments.
  • Future Work: The authors suggest further studies on the baseline $\gamma$ for SAC across different environments and cycle times. They also implicitly point towards exploring more sophisticated adaptive strategies for hyperparameters beyond simple scaling.

Summary updated

Abstract

Continuous-time reinforcement learning tasks commonly use discrete steps of fixed cycle times for actions. As practitioners need to choose the action-cycle time for a given task, a significant concern is whether the hyper-parameters of the learning algorithm need to be re-tuned for each choice of the cycle time, which is prohibitive for real-world robotics. In this work, we investigate the widely-used baseline hyper-parameter values of two policy gradient algorithms—PPO and SAC—across different cycle times. Using a benchmark task where the baseline hyper-parameters of both algorithms were shown to work well, we reveal that when a cycle time different than the task default is chosen, PPO with baseline hyper-parameters fails to learn. Moreover, both PPO and SAC with their baseline hyper-parameters perform substantially worse than their tuned values for each cycle time. We propose novel approaches for setting these hyper-parameters based on the cycle time. In our experiments on simulated and real-world robotic tasks, the proposed approaches performed at least as well as the baseline hyper-parameters, with significantly better performance for most choices of the cycle time, and did not result in learning failure for any cycle time. Hyper-parameter tuning still remains a significant barrier for real-world robotics, as our approaches require some initial tuning on a new task, even though it is negligible compared to an extensive tuning for each cycle time. Our approach requires no additional tuning after the cycle time is changed for a given task and is a step toward avoiding extensive and costly hyper-parameter tuning for real-world policy optimization.
BibTeX
@inproceedings{farrahi2023reducing,
  author = {Homayoon Farrahi and A. Rupam Mahmood},
  title = {{Reducing the Cost of Cycle-Time Tuning for Real-World Policy Optimization}},
  booktitle = {IJCNN},
  year = {2023},
  url = {https://openreview.net/forum?id=KQWKjG_N4R},
}
Figure