State-Dependent Maximum Entropy Reinforcement Learning for Robot Long-Horizon Task Learning

Deshuai Zheng, Yan Jin, Tao Xue, Yong LiuJournal of Intelligent & Robotic Systems 2024 · DOI

Abstract

Task-oriented robot learning has shown significant potential with the development of Reinforcement Learning (RL) algorithms. However, the learning of long-horizon tasks for robots remains a formidable challenge due to the inherent complexity of tasks, typically comprising multiple diverse stages. Universal RL algorithms commonly encounter issues such as slow convergence or even failure to converge altogether when applied to such tasks. The reasons behind these challenges lie in the local optima trap or redundant exploration during the new stages or the junction of two continuous stages. To address these challenges, we propose a novel state-dependent maximum entropy (SDME) reinforcement learning algorithm. This algorithm effectively balances the trade-off between exploration and exploitation around three kinds of critical states arising from the unique nature of long-horizon tasks. We conducted experiments within an open-source simulation environment, focusing on two representative long-horizon tasks. The proposed SDME algorithm exhibits faster and more stable learning capabilities, requiring merely one-third of the number of learning samples necessary for baseline approaches. Furthermore, we assess the generalization ability of our method under randomly initialized conditions, and the results show that the success rate of the SDME algorithm is nearly twice that of the baselines. Our code will be available at https://github.com/Peter-zds/SDME .

Introduction

Training robots for "long-horizon" tasks—complex sequences like opening a drawer to put an object inside—is notoriously difficult because standard Reinforcement Learning (RL) often gets stuck at stage transitions. These tasks consist of multiple distinct stages (e.g., approach handle → grasp → pull → release).

Current Deep Reinforcement Learning (DRL) algorithms face two main hurdles in these scenarios:

  1. The Local Optima Trap: A robot might find a way to get a decent reward in an early stage and stop trying to progress because the "risk" of exploration seems too high.
  2. Redundant Exploration: Conversely, a robot might continue to move randomly even after it has successfully transitioned to a new stage, wasting time and destabilizing the learning process.

Essentially, standard RL treats every state with the same "curiosity" level. In a multi-stage task, this is inefficient. You want the robot to be highly curious when it's stuck, but focused and "boring" once it has found the path to the next milestone.

Methodology

The proposed State-Dependent Maximum Entropy (SDME) algorithm builds upon Soft Actor-Critic (SAC), which balances reward maximization with action entropy (exploration) via a temperature parameter, $\alpha$. SDME makes this balance state-dependent by categorizing robot states into four types based on reward dynamics:

  • Up-stage: The robot just hit a milestone (a "jump" in reward). SDME suppresses exploration here to lock in progress.
  • Stuck: The reward is stagnant. SDME increases exploration to find a way out of the rut.
  • Ahead: The robot is near its historical best. SDME encourages extra exploration to fine-tune the optimal path.
  • Normal: Standard linear progress where the algorithm behaves like a typical SAC agent.

To support this, the authors designed a Continuous Reward Function with Jump Properties to signal stage transitions. They also introduced Gradient Separation to prevent rare transition data from being washed out by common samples, and Soft Reward Reduction to handle physics glitches that might provide unrealistic reward signals.

Experimental Results

The researchers evaluated SDME using the Meta-world simulation environment on two complex tasks: drawer-put-in (7 stages) and box-pick-out (6 stages).

Comparison of exploration-focused (a) and exploitation-focused (b) baseline strategies.

As shown in the figures, SDME demonstrates a massive leap in efficiency and robustness. While standard exploration or exploitation-focused baselines require 7.5 to 10 million samples to converge, SDME achieves success in approximately 2.5 million samples.

Performance comparison between a non-converging baseline (c) and the proposed SDME algorithm (d).

Key performance metrics include:

  • Sample Efficiency: SDME requires only one-third of the training samples compared to state-of-the-art baselines.
  • Stability: Learning curves are significantly smoother, avoiding the long plateaus typical of other algorithms.
  • Generalization: In environments with randomized initial positions, SDME achieved an 86% success rate, nearly double the 44% of the best SAC baseline.

Conclusion

SDME addresses the critical challenge of stage transitions in long-horizon robot tasks by dynamically adjusting exploration. While currently dependent on manually designed reward functions with explicit "jumps," future work aims to automate stage identification through supervised learning or goal-image rewards. This framework provides a robust path toward making RL agents more efficient at complex, multi-step planning.

Summary updated

Abstract

Abstract Task-oriented robot learning has shown significant potential with the development of Reinforcement Learning (RL) algorithms. However, the learning of long-horizon tasks for robots remains a formidable challenge due to the inherent complexity of tasks, typically comprising multiple diverse stages. Universal RL algorithms commonly encounter issues such as slow convergence or even failure to converge altogether when applied to such tasks. The reasons behind these challenges lie in the local optima trap or redundant exploration during the new stages or the junction of two continuous stages. To address these challenges, we propose a novel state-dependent maximum entropy (SDME) reinforcement learning algorithm. This algorithm effectively balances the trade-off between exploration and exploitation around three kinds of critical states arising from the unique nature of long-horizon tasks. We conducted experiments within an open-source simulation environment, focusing on two representative long-horizon tasks. The proposed SDME algorithm exhibits faster and more stable learning capabilities, requiring merely one-third of the number of learning samples necessary for baseline approaches. Furthermore, we assess the generalization ability of our method under randomly initialized conditions, and the results show that the success rate of the SDME algorithm is nearly twice that of the baselines. Our code will be available at https://github.com/Peter-zds/SDME .
BibTeX
@article{zheng2024state,
  author = {Deshuai Zheng and Yan Jin and Tao Xue and Yong Liu},
  title = {{State-Dependent Maximum Entropy Reinforcement Learning for Robot Long-Horizon Task Learning}},
  journal = {Journal of Intelligent \& Robotic Systems},
  year = {2024},
  doi = {10.1007/s10846-024-02049-8},
}
Figure