Evolutionary Reinforcement Learning with Parameterized Action Primitives for Diverse Manipulation Tasks

Xianxu Qiu, Haiming Huang, Weiwei Chen, Qiuzhen Lin, Wei-Neng Chen, Fuchun SunAAAI 2025 · DOI

TL;DR

This paper introduces ERLAP, a novel framework that combines Evolutionary Algorithms (EA) and Hierarchical Reinforcement Learning (HRL) with parameterized action primitives. ERLAP effectively tackles robotic manipulation tasks (RMTs) by improving exploration efficiency and mitigating catastrophic forgetting, outperforming state-of-the-art methods in both dense and sparse reward scenarios.

What Problem Does This Paper Address?

Robotic manipulation tasks (RMTs) are notoriously challenging for Reinforcement Learning (RL) agents. Two primary issues hinder their performance:

  1. Inefficient Exploration: The action space in RMTs is vast, involving continuous control signals like force and torque. This makes it incredibly difficult for standard RL algorithms to explore effectively and discover optimal manipulation strategies.
  2. Catastrophic Forgetting: Even when agents learn successful policies, they tend to forget previously learned skills when encountering new tasks or data distributions. This is often exacerbated by poor sampling efficiency, where the agent doesn't gather enough diverse experiences to retain knowledge.

While Hierarchical RL (HRL) helps by breaking down tasks into sub-goals and using parameterized actions to reduce the exploration burden, it still struggles with catastrophic forgetting. Conversely, Evolutionary RL (ERL) can combat forgetting by maintaining populations of policies, but it can suffer from network collapse or a lack of semantic understanding in its evolutionary process. This paper aims to bridge this gap by synergistically combining the strengths of HRL and ERL to address both exploration and forgetting challenges in RMTs.

Methods

ERLAP is built upon two core components: an RL module and an EA module, working in tandem.

  • RL Module (Hierarchical RL with Parameterized Primitives):
    • Instead of raw actions, ERLAP utilizes a library of predefined, parameterized action primitives (e.g., "Reach," "Grasp," "Release"). These primitives act as high-level functional APIs.
  • A high-level task policy learns to select the type of primitive, while a low-level parameter network instantiates the specific parameters for that primitive (e.g., target location, gripper state). This significantly reduces the effective action space for exploration.
  • The RL module uses a state-of-the-art DRL algorithm (SAC) for learning, with specific loss functions designed for the parameterized action space.
  • EA Module (Dual-Population Evolutionary Mechanism):
    • To combat catastrophic forgetting and improve sample efficiency, ERLAP employs a dual-population EA: a "success population" (Ps) and a "development population" (Pd).
  • Primitive Sequence Evolution: Complete sequences of parameterized primitives generated by the RL module are treated as individuals in the EA. These sequences are evolved using crossover and mutation operators (replacement, deletion, addition, and Gaussian noise).
  • Population Management: The success population (Ps) stores high-quality primitive sequences. The development population (Pd) is used to explore new variations. The populations interact and update their fitness based on environmental rewards. This mechanism ensures that high-quality experiences are retained and diversified, preventing the RL agent from forgetting.

Datasets

The paper evaluates ERLAP on simulated robotic manipulation tasks within the Robosuite environment.

  • Dense Rewards: Six manipulation tasks of varying complexity are used: Lift (LIF), Pick-and-Place (PAP), Nut-Assembly (NAS), Peg-Insert (PIN), Clean-Up (CUP), and Stack (STA). These tasks provide dense reward signals, allowing for evaluation of sampling efficiency.
  • Sparse Rewards: Four more challenging tasks are used to specifically test the ability to avoid catastrophic forgetting: Lift (LIF), Pick-and-Place (PAP), Stack (STA), and Put-into-Drawer (PID). These tasks provide sparse rewards, making learning significantly harder.

No new datasets were introduced; standard benchmarks within Robosuite were utilized.

Key Insights and Ideas

  • Synergy of HRL and EA: The core insight is that combining HRL's structured exploration with EA's robust forgetting prevention creates a powerful framework for complex RMTs. Neither approach alone fully solves the problem.
  • Parameterized Primitives as Building Blocks: Using parameterized primitives as the fundamental units of action, rather than raw control signals, dramatically simplifies the learning problem and improves exploration.
  • Dual-Population EA for Experience Management: The dual-population EA is crucial for retaining and diversifying high-quality experiences. It acts as a memory bank and a creative engine, ensuring that learned skills are not lost.
  • Primitive Sequence Evolution: Evolving entire sequences of primitives, rather than just individual actions or network parameters, allows the EA to capture higher-level task structure and relationships between actions.

Key Results

  • Dense Rewards: ERLAP significantly outperforms state-of-the-art RL algorithms (TD3, SAC, MAPLE, TRAPs) in terms of sampling efficiency and final success rates across various RMTs.
  • ERLAP achieved a 100% success rate in four out of six dense reward tasks, surpassing MAPLE and TRAPs which only achieved 100% in two tasks.
  • ERLAP used fewer steps for training compared to baselines like TRAPs (18.8% reduction) and MAPLE (30.1% reduction).
  • Sparse Rewards: ERLAP effectively avoids catastrophic forgetting in challenging sparse reward tasks.
  • In the Lift task, ERLAP achieved a 100% success rate with significantly fewer steps (over 66.6% reduction compared to MAPLE).
  • For the Put-into-Drawer task, where other algorithms struggled to achieve any reward, ERLAP achieved a 77.6% success rate.
  • The experience distribution analysis (Figure 7) shows ERLAP's replay buffer is more uniform and contains a higher proportion of successful experiences, indicating better knowledge retention.

Limitations and Future Work

  • Limitations:
    • The paper relies on a predefined library of action primitives. The performance is contingent on the quality and coverage of this library.
  • While ERLAP addresses catastrophic forgetting, the inherent complexity of RMTs means that achieving perfect performance across all tasks remains a challenge.
  • Future Work:
    • Automatic Primitive Discovery: A key future direction is to develop methods for automatically discovering new primitives and adding them to the library, making the system more adaptable.
  • Leveraging Large Language Models (LLMs): The authors suggest exploring the use of LLMs to leverage knowledge graph inference for primitive sequence generation, which could lead to more sophisticated and generalizable manipulation skills.

Summary updated

BibTeX
@inproceedings{qiu2025evolutionary,
  author = {Xianxu Qiu and Haiming Huang and Weiwei Chen and Qiuzhen Lin and Wei-Neng Chen and Fuchun Sun},
  title = {{Evolutionary Reinforcement Learning with Parameterized Action Primitives for Diverse Manipulation Tasks}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2025},
  doi = {10.1609/aaai.v39i14.33606},
}
Figure