Average-Reward Maximum Entropy Reinforcement Learning for Global Policy in Double Pendulum Tasks

Jean Seong Bjorn Choe, BumKyu Choi, Jong-Kook KimarXiv 2025 · arXiv · OpenReview

TL;DR

This paper presents an improved reinforcement learning (RL) approach, AR-EAPO, for controlling underactuated double pendulum robots (acrobot and pendubot) in challenging swing-up and stabilization tasks. By adapting the training environment and reward structure, their method demonstrates robust performance and adaptability to new competition rules and increased disturbances, outperforming previous benchmarks.

What Problem Does This Paper Address?

The paper tackles the problem of training robust and adaptable reinforcement learning controllers for complex robotic tasks, specifically the swing-up and stabilization of underactuated double pendulum systems like the acrobot and pendubot. The 3rd AI Olympics at ICRA 2025 introduced stricter guidelines, emphasizing global policy robustness against significant external disturbances and requiring reliable performance from arbitrary initial configurations. Traditional RL methods often struggle with such challenges due to the need for extensive reward engineering or explicit system modeling, and they can be sensitive to variations in initial states and environmental noise. This work aims to provide a solution that meets these new demands without relying on explicit system knowledge.

Methods

The core of the proposed solution is an adaptation of their previously developed Average-Reward Entropy Advantage Policy Optimization (AR-EAPO) algorithm. Key modifications include:

  • Modified MDP Design: Instead of altering the policy optimization itself, the authors adjusted the training environment. This involved modifying the initial state distribution to be a broad Gaussian centered on the bottom state, simulating a wider range of starting conditions. They also introduced a large random truncation probability during training, which effectively shortens the expected temporal horizon for calculating the average reward. This encourages the policy to learn faster swing-ups by prioritizing immediate costs.
  • Relaxed Quadratic Cost: The reward function was simplified to a quadratic cost that penalizes deviations from the goal state, omitting torque penalties. This allows for faster achievement of the desired upright position.
  • AR-EAPO Algorithm: This is a model-free actor-critic algorithm that optimizes an average-reward maximum entropy (MaxEnt) objective. It leverages the concept of a "bias value function" to estimate the advantage of taking a particular action compared to the average performance in a given state. The algorithm incorporates variance reduction techniques to improve learning stability.

Datasets

The paper does not explicitly mention specific datasets for training. Instead, it describes extensive simulations performed using their AR-EAPO algorithm. The evaluation is conducted on the tasks defined by the 3rd AI Olympics at ICRA 2025, which involve the acrobot and pendubot systems. These tasks are characterized by updated evaluation metrics and increased external disturbances, differing from previous competition benchmarks.

Key Insights and Ideas

  • Environment Adaptation is Key: The paper highlights that modifying the training environment (initial state distribution, truncation probability) can be more effective than solely tweaking the RL algorithm for achieving robustness and adaptability to new task requirements.
  • Short-Term Bootstrapping for Speed: By artificially shortening the expected horizon through truncation probability, the algorithm is incentivized to learn faster policies, which is crucial for swing-up tasks.
  • MaxEnt for Robustness: The use of maximum entropy in the objective encourages policies that are not overly sensitive to small perturbations, contributing to robustness against disturbances.
  • Unified Approach for Different Robots: The AR-EAPO framework, with its adaptations, proves effective for both the acrobot and pendubot, demonstrating its generality for underactuated double pendulum systems.

Key Results

The AR-EAPO approach demonstrated superior and stable performance across five different random seeds for both the acrobot and pendubot tasks.

  • Acrobot: The proposed AR-EAPO achieved an average score of 0.667, significantly outperforming other tested methods like AR-EAPO (v2) (0.367), MCPILCO (0.037), and TVLQR (0.008).
  • Pendubot: For the pendubot, AR-EAPO achieved an average score of 0.752, also showing a substantial improvement over AR-EAPO (v2) (0.647), MCPILCO (0.060), and TVLQR (0.075).
  • Qualitative Results: Time-series plots of joint angles, velocities, and torques show the controller's ability to recover and stabilize the robots even under periodic random disturbances.

Limitations and Future Work

  • Numerical Instability: The paper notes that some benchmark controllers experienced numerical instability or NaN errors during execution, indicated by "(E)" in the tables. While their method did not exhibit this issue in the reported results, it's a general challenge in RL.
  • Computational Cost: Training the best-performing policy took approximately 100 minutes on a high-end GPU setup, which might be a consideration for real-time deployment or rapid iteration.
  • Future Work: The paper implicitly suggests that further exploration of environment design and reward shaping within the AR-EAPO framework could lead to even more robust and efficient controllers for increasingly complex robotic challenges. The success in this competition also sets a strong foundation for tackling more advanced tasks.

Summary updated

Abstract

This report presents our reinforcement learning-based approach for the swing-up and stabilisation tasks of the acrobot and pendubot, tailored specifcially to the updated guidelines of the 3rd AI Olympics at ICRA 2025. Building upon our previously developed Average-Reward Entropy Advantage Policy Optimization (AR-EAPO) algorithm, we refined our solution to effectively address the new competition scenarios and evaluation metrics. Extensive simulations validate that our controller robustly manages these revised tasks, demonstrating adaptability and effectiveness within the updated framework.
BibTeX
@misc{choe2025average,
  author = {Jean Seong Bjorn Choe and BumKyu Choi and Jong-Kook Kim},
  title = {{Average-Reward Maximum Entropy Reinforcement Learning for Global Policy in Double Pendulum Tasks}},
  year = {2025},
  eprint = {2505.07516},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2505.07516},
}
Figure