Average-Reward Maximum Entropy Reinforcement Learning for Underactuated Double Pendulum Tasks

Jean Seong Bjorn Choe, BumKyu Choi, Jong-Kook KimarXiv 2024 · arXiv · OpenReview

TL;DR

This paper introduces AR-EAPO, a novel reinforcement learning algorithm that combines average-reward and maximum entropy principles to tackle the challenging swing-up and stabilization tasks of underactuated double pendulums. The approach achieves superior performance and robustness compared to existing methods, crucially without requiring extensive reward function engineering.

What Problem Does This Paper Address?

The paper addresses the complex control problem of making an underactuated double pendulum (like an acrobot or pendubot) swing up from a hanging position to a stable upright stance. This task is particularly tricky for reinforcement learning (RL) agents because:

  1. Balancing Exploration and Exploitation: Penalizing fast movements during the swing-up can hinder exploration, while overly lenient penalties might lead to unstable or inefficient swings.
  2. Long-Term Stability: The agent needs to maintain stability even after finding a seemingly good, but potentially suboptimal, upright position, requiring continuous learning and adaptation. Traditional RL approaches often rely on meticulously hand-crafted reward functions, which are time-consuming to design, difficult to tune, and may not generalize well across different tasks or environments. This paper aims to reduce this reliance on reward engineering by framing the problem as a continuing task with an average-reward objective and incorporating maximum entropy principles for better exploration and robustness.

Methods

The core contribution is the Average-Reward Entropy Advantage Policy Optimization (AR-EAPO) algorithm. This is an extension of the Entropy Advantage Policy Optimisation (EAPO) framework, adapted for average-reward settings. Key aspects include:

  • Average-Reward RL: Instead of optimizing for discounted future rewards, AR-EAPO focuses on maximizing the long-term average reward. This is particularly suitable for continuous tasks where there's no natural end-point, preventing bias towards immediate rewards.
  • Maximum Entropy RL (MaxEnt RL): AR-EAPO integrates an entropy bonus into the objective function. This encourages the agent to explore a wider range of actions and behaviors, leading to more robust policies that are less sensitive to environmental perturbations.
  • Soft Advantage Estimation: The algorithm estimates a "soft" advantage function that combines reward and entropy objectives. This is then used within a Proximal Policy Optimization (PPO) framework to update the policy.
  • Novel Truncation Method: To address sample correlation issues in on-policy training with vectorized environments, a random truncation method is introduced. This randomly cuts off "episodes" with a small probability, decorrelating the environments and improving sample efficiency.

Datasets

The paper uses the Gymnasium [17] interface for the acrobot and pendubot environments. These are standard benchmarks for robotics control tasks. The experiments are conducted in simulation, and the authors do not introduce new datasets but rather evaluate their algorithm on these established simulation environments.

Key Insights and Ideas

  • Average Reward for Continuous Tasks: Framing continuous control problems like pendulum balancing as average-reward tasks is crucial for achieving long-term optimal solutions, avoiding the pitfalls of artificial discounting.
  • MaxEnt RL for Robustness: The maximum entropy principle is a powerful tool for enhancing policy robustness against environmental noise and disturbances, a critical factor in real-world robotics.
  • Reduced Reward Engineering: AR-EAPO significantly lessens the burden of designing complex reward functions, making RL more accessible and applicable to a wider range of problems.
  • Random Truncation for Sample Efficiency: A simple yet effective technique for improving on-policy training in parallel environments by decorrelating samples.

Key Results

AR-EAPO demonstrated superior performance and robustness on both the acrobot and pendubot tasks compared to established baselines like TVLQR and ILQR.

  • Performance: For the acrobot, AR-EAPO achieved a higher "RealAI Score" (0.633 vs. 0.504 for TVLQR) and significantly faster swing-up times (1.39s vs. 4.05s for TVLQR). Similar improvements were observed for the pendubot.
  • Robustness: AR-EAPO achieved notably higher overall robustness scores: 0.739 for the acrobot (vs. 0.607 for TVLQR) and 0.916 for the pendubot (vs. 0.767 for TVLQR). This indicates a much better ability to maintain performance under various simulated perturbations like model noise, velocity noise, and time delays.

Limitations and Future Work

  • Simulation Only: The current results are exclusively from simulation. The authors acknowledge that the next crucial step is to test and refine the approach in real-robot settings to validate its applicability and address real-world challenges.
  • Hyperparameter Sensitivity: While the paper mentions AR-EAPO is less sensitive to certain reward function parameters (Q and R) when the reward-to-entropy ratio is balanced, further investigation into hyperparameter tuning and sensitivity analysis could be beneficial.
  • Generalization: While promising, the paper focuses on two specific pendulum tasks. Future work could explore the algorithm's generalization capabilities to other underactuated systems or more complex robotic tasks.

Summary updated

Abstract

This report presents a solution for the swing-up and stabilisation tasks of the acrobot and the pendubot, developed for the AI Olympics competition at IROS 2024. Our approach employs the Average-Reward Entropy Advantage Policy Optimization (AR-EAPO), a model-free reinforcement learning (RL) algorithm that combines average-reward RL and maximum entropy RL. Results demonstrate that our controller achieves improved performance and robustness scores compared to established baseline methods in both the acrobot and pendubot scenarios, without the need for a heavily engineered reward function or system model. The current results are applicable exclusively to the simulation stage setup.
BibTeX
@misc{choe2024average,
  author = {Jean Seong Bjorn Choe and BumKyu Choi and Jong-Kook Kim},
  title = {{Average-Reward Maximum Entropy Reinforcement Learning for Underactuated Double Pendulum Tasks}},
  year = {2024},
  eprint = {2409.08938},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2409.08938},
}
Figure